Multimodal entity connection method based on data enhancement

Abstract

The present invention discloses a multimodal entity connection method based on data enhancement, which relates to the field of data processing technology, including: using a large language model to enhance entities, summarizing entity descriptions through designing instructions, extracting key information and optimizing; using a multimodal large language model to enhance semantic information, generating enhanced information descriptions by fusing text and visual features, including entity names, types, functions and behavioral characteristics, physical or structural attributes, and visual features such as spatial structure, appearance morphology, color texture extracted from visual information; expanding and optimizing candidate entity sets through retrieval technology to achieve retrieval enhancement; using a large language model for decision-making selection, and selecting the best matching item from the candidate entity set obtained through retrieval enhancement. The present invention significantly improves the accuracy and efficiency of multimodal entity linking through multi-level data enhancement and precise entity selection process.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal entity connection method based on data enhancement. Background Art

[0002] Entity Linking is a key technology in the field of Natural Language Processing (NLP) and Information Retrieval. It is essential for understanding and organizing large amounts of unstructured data. In the real world, whether it is online news, social media posts or academic papers, people frequently mention various entities in texts, such as names, places, organizations, etc. Correctly linking these mentions with the corresponding entities in the knowledge base can bring important value. For example, through entity linking, information scattered in different documents can be integrated into a unified knowledge framework to form a richer and more structured knowledge representation; entity linking is also a key step in building and maintaining knowledge graphs, which helps to connect new information with existing knowledge structures; in applications such as automated question-answering systems, recommendation systems and virtual assistants, entity linking is the basis for intelligently understanding and responding to user queries.

[0003] The importance of entity linking is further highlighted in the context of multimodal data. With the explosive growth of digital content, entity mentions are frequently encountered not only in text, but also in images, videos, and other visual media. The importance of the Multimodal Entity Linking (MEL) task lies in the fact that multimodal data provides richer information than pure text. By combining mentions in text with visual content, the context and semantics of entities can be understood more comprehensively; visual information can help resolve ambiguities in text, for example, confirming that the mentioned entity is a specific landmark or person through images; in search engines, social media, and recommendation systems, multimodal entity linking can provide richer and more intuitive search results, thereby enhancing user experience; in multimodal data, entity linking helps discover and build cross-media knowledge connections, promoting the generation of new knowledge and the expansion of old knowledge.

[0004] Therefore, entity linking is not only a bridge connecting text and knowledge base, but also a core technology for realizing automatic information processing and intelligent application. However, multimodal entity linking faces a series of challenges. The first is the complexity of information fusion: how to effectively combine information from different modalities to achieve accurate identification and linking of entities is a technical problem. The second is the limitations of visual semantic understanding: despite great progress in computer vision technology, understanding and interpreting semantic information in visual content remains a challenge. The third is the diversity and noise of data: multimodal data often contains noise and incomplete information, which may affect the accuracy of entity linking.

[0005] The challenges faced by existing technical solutions in the multimodal entity linking task mainly include the neglect of visual information, the lack of domain knowledge, the complexity of multimodal information fusion, and over-modeling on specific datasets.

[0006] In the research field of multimodal entity linking, although existing technical solutions have made some progress, they still have some significant limitations. First, methods based on traditional text encoders, such as TF-IDF and word2vec, as well as LSTM or BERT encoders, mainly focus on the processing of text information, but fail to fully utilize the complementary role of visual information. This limits their performance in dealing with text ambiguity and noisy images.

[0007] Secondly, autoregressive entity retrieval methods, such as BLINK, although trying to enhance entity recognition by generating unique names, may rely too much on textual information and ignore the potential value of visual content. In addition, multimodal networks based on deep learning, such as deep zero-shot models, although they perform well on specific tasks, may require a large amount of annotated data and have complexities in fusing multimodal information.

[0008] Furthermore, methods based on graph matching and multimodal co-attention mechanisms, such as the bipartite graph matching problem and multimodal co-attention mechanism, attempt to solve the problem of multimodal information fusion, but may have limitations in processing complex data and generalization capabilities. Although large language models are powerful in text processing, they usually lack domain-specific knowledge, which is crucial for the success of MEL tasks. Summary of the invention

[0009] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a multimodal entity connection method based on data enhancement, which improves the accuracy and efficiency of multimodal entity linking.

[0010] To achieve the above object, the technical solution adopted by the present invention is: a multimodal entity connection method based on data enhancement, comprising the following steps:

[0011] Step 1: Use a large language model to enhance entities: Use a large language model to summarize the entity description, design instructions to clearly indicate the goal of the summary, process the entity description through the designed instructions, extract key information and generate a summary, and optimize the summary;

[0012] Step 2: Enhance semantic information using a multimodal large language model: The multimodal large language model generates an enhanced information description by fusing text and visual features. The information description is used to refine and emphasize key aspects of the information, including the name, type, function and behavior characteristics, physical or structural attributes of the entity, and the visual features of spatial structure, appearance morphology, color texture extracted from visual information. A prompt template is used to guide the multimodal large language model to generate a more accurate and relevant information description. The generated enhanced information description is used to match candidate entities to determine the best link during the entity linking process.

[0013] Step 3: Retrieval enhancement: Expand and optimize the candidate entity set through retrieval technology to improve the accuracy of entity linking;

[0014] Step 4: Decision selection uses a large language model to select the best match from the set of candidate entities obtained through retrieval enhancement.

[0015] As a further improvement of the present invention, the step 1 specifically comprises the following steps:

[0016] Step 1.1, instruction design: ,in, For a given instruction, Indicates that according to the original entity and description The process of designing instructions, Represents an entity, Represents the name of the entity, Represents a description of an entity;

[0017] Step 1.2, generate summary: ,in, Indicates the use of a large language model based on entity names and descriptions and instructions Generate summary process;

[0018] Step 1.3, Summary Evaluation: ,in, Indicates the evaluation result. Represents the summary generated by the evaluation Quality process;

[0019] Step 1.4. Summary optimization: If the evaluation results Indicates that the summary needs to be optimized, then: ,in For the updated summary instruction, For the optimized summary, According to the evaluation results the process of updating summary instructions;

[0020] Step 1.5: Final summary: Optimized summary This is the final output of entity enhancement.

[0021] As a further improvement of the present invention, the step 2 specifically comprises the following steps:

[0022] Step 2.1, design prompt template: ,in For the prompt template, For textual mention information, For visual information, represents a function for designing a prompt template based on text information and visual information. The prompt template is used to guide the multimodal large language model to generate enhanced mention descriptions;

[0023] Step 2.2: Generate enhanced semantic description: ,in is the semantic description of the initial enhancement, Represents text information and visual information The fusion result of Indicates the use of multimodal large language models and prompt templates Process the fused information and generate a function to enhance the semantic description;

[0024] Step 2.3: Evaluate the enhanced semantic information description: ,in To evaluate the results, Represents the enhanced semantic description generated by the evaluation A function of quality, evaluating content including semantic completeness, accuracy, and consistency with the original information;

[0025] Step 2.4, optimize the prompt template: If the evaluation result Indicates that optimization is needed, then: ,in For the updated prompt template, According to the evaluation results Function to update the prompt template;

[0026] Step 2.5: Regenerate enhanced semantic description: Use the updated hint template Regenerate enhanced mention description: , in, Represents text information and visual information The fusion result, Mention description for regenerated enhancements;

[0027] Step 2.6: Final enhanced semantic description: Optimized enhanced mention description It is the final output of the mention enhancement process, which contains the fusion of textual semantic information and visual feature information, and is used for subsequent entity linking tasks.

[0028] As a further improvement of the present invention, the step 3 specifically comprises the following steps:

[0029] Step 3.1: Initialize the candidate entity set: Use string matching algorithm to match each text mention information Processing is performed to generate an initial set of candidate entities by comparing the character similarity between textual mentions and entities in the knowledge base. ,The set contains entities that have character overlap with the text mentions;

[0030] Step 3.2: Use a pre-trained embedding model: Use a language model pre-trained on a large-scale text corpus to embed text mention information and the candidate entity set Each entity in is mapped to the vector space to obtain the embedding vector representation of the text mention and the embedding vector representation of the candidate entity ,Vectors are used to capture the semantic features of text;

[0031] Step 3.3. Retrieval and re-ranking: Compute the enhanced mention description The embedding vector and candidate entity set The cosine similarity between the embedding vectors of each entity in is used to obtain the similarity score, and the candidate entities are sorted in descending order according to the similarity score to obtain the sorted candidate entity set. ;

[0032] Step 3.4: Expand the candidate entity set: Based on the enhanced mention description , perform similarity retrieval in the knowledge base, select entities whose similarity scores exceed the preset threshold, and add them to the initial candidate set Add the high similarity entities other than to the candidate set to obtain the expanded candidate entity set ;

[0033] Step 3.5: Optimize the candidate entity set: Apply the entity disambiguation algorithm to analyze the semantic relationship and co-occurrence pattern between entities, eliminate irrelevant candidate entities, and obtain the final optimized candidate entity set ,The set contains both the related entities of the initial match and the ,candidate entities obtained after semantic expansion, thus improving the recall and ,precision of entity linking.

[0034] As a further improvement of the present invention, in step 3, the process of retrieval enhancement is specifically as follows:

[0035] ;

[0036] is the initial candidate entity set obtained by string matching in step 3.1, is the enhanced mention description obtained in step 2.5, is the expanded candidate entity set after retrieval, corresponding to the expanded candidate entity set in step 3.4. is a retrieval function that retrieves and re-ranks candidate entities based on the similarity between the mention description information and the embedding vectors of the candidate entities;

[0037] Expanded candidate entity set The final optimized candidate entity set is obtained through entity disambiguation: is the entity disambiguation function, which is used to optimize the candidate entity set, corresponding to the optimization process in step 3.5; is the final optimized set of candidate entities, i.e., the output result of step 3.5, which will be used for the decision-making selection in step 4.

[0038] As a further improvement of the present invention, the step 4 specifically includes the following steps:

[0039] Step 4.1: Candidate entity set: Retrieve the enhanced candidate entity set , including multiple candidate entities related to the text mention;

[0040] Step 4.2: Use the large language model for selection: Use the pre-trained and fine-tuned large language model as a selector to evaluate each candidate entity to determine which entity best matches the text mention;

[0041] Step 4.3, Evaluation and Comparison: The large language model evaluates each candidate entity in the following way:

[0042] (1) Calculate semantic similarity: Use cosine similarity to calculate mention descriptions The similarity score with the embedding vector of the candidate entity ;

[0043] (2) Calculate context relevance: Calculate the context relevance score based on the context information of the mention description and the candidate entity ;

[0044] (3) Calculate attribute matching: Compare the attribute features of the mentioned description and the candidate entity to obtain the attribute matching score ;

[0045] (4) Comprehensive score: The above scores are combined in a weighted manner to obtain the final score. ,in is the weight coefficient;

[0046] Step 4.4: Select the best match: Based on the evaluation results of step 4.3, select the final score The highest candidate entity is taken as the link result;

[0047] Step 4.5: Use prompt templates: Design prompt templates , contains the objectives, evaluation criteria and constraints of the entity linking task, which are used to guide the selection of large language models;

[0048] Step 4.6: Consider multimodal information: Integrate textual and visual features in the decision process, including:

[0049] (1) Text features: text description, attribute information, and context information of the entity;

[0050] (2) Visual features: spatial structure, appearance features, and visual context information related to entities;

[0051] (3) Feature fusion: Multimodal fusion of text and visual features to generate comprehensive feature representation ;

[0052] Step 4.7, iterative optimization: If the confidence of the link result is lower than the preset threshold , the following optimizations are performed: (1) adjusting the content and structure of the prompt template; (2) updating the parameters of the large language model, including adjusting the attention weight and updating the model parameters by automatically building a correct instruction dataset for entity connection; (3) optimizing the similarity calculation method, including adjusting the feature weight and improving the metric function; until the confidence of the link result exceeds the threshold or reaches the maximum number of iterations;

[0053] Step 4.8: Output the final result: The decision selection process outputs an entity link result that best matches the text mention , the result meets the preset confidence requirements and completes the entire entity linking task.

[0054] As a further improvement of the present invention, in step 4, the process of decision-making selection is specifically as follows:

[0055] ;

[0056] in is the selected best entity, i.e., the linking result in step 4.8; is the final optimized candidate entity set in step 3.5, is the enhanced mention description obtained in step 2.5, It is the prompt template designed in step 4.5; is the multimodal feature fusion result in step 4.6, Represents an entity, Representative representation from the candidate entity set Select the entity that maximizes the output score of the large language model; It is a large language model used to select the best entity. Its decision process includes: Construct the input representation; calculate the comprehensive score defined in step 4.3 , at the confidence threshold Select the entity with the highest score under the constraint; if the confidence requirement is not met, perform the iterative optimization process of step 4.7.

[0057] The beneficial effects of the present invention are:

[0058] The present invention improves the accuracy and efficiency of multimodal entity linking by utilizing the capabilities of a large language model and a multimodal large language model to perform entity enhancement, semantic information enhancement, retrieval enhancement, and decision-making selection; specifically, the following aspects are embodied:

[0059] (1) Entity enhancement uses a large language model to solve the problem of long entity descriptions and redundant information, which helps to refine the key features of the entity and thus provide more accurate entity information during the linking process.

[0060] (2) Semantic information enhancement: A multimodal large language model is used to generate high-quality information mention descriptions based on image and text contexts. This helps to supplement the semantic information that may be missing in the mention information and improve the recognizability of the mention information.

[0061] (3) Retrieval enhancement: By calculating the similarity between the mention and the embedded representation of the entity, the K entities most relevant to the mention are retrieved, thereby narrowing the scope of candidate entities and improving the accuracy of subsequent selection.

[0062] (4) Decision-making selection uses the large language model as a selector to select the entity that best matches the mention from the retrieved candidate entities. This step utilizes the powerful capabilities of the large language model in text understanding and reasoning to provide clear task guidance, thereby improving the accuracy of problem definition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a framework diagram of an embodiment of the present invention;

[0064] Figure 2 Schematic diagram of entity linking accuracy in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] Example 1

[0067] like Figure 1 As shown, a multimodal entity connection method based on data enhancement includes:

[0068] 1. Entity enhancement based on large language models. This process aims to improve the quality of entity descriptions for better entity linking. The process is as follows: In the multimodal entity linking task, the entity descriptions in the knowledge base may be long and contain a lot of redundant information. This not only increases the complexity of processing, but may also affect the accuracy of entity linking. Use large language models to summarize entity descriptions. By designing specific instructions, LLMs are able to generate concise and informative summaries that capture the core features of entities. In order to guide LLMs to generate high-quality summaries, instructions need to be carefully designed. These instructions should clearly indicate the goals of the summary, such as length limits, emphasis on key information, etc. LLMs process entity descriptions according to the designed instructions, extract key information, and generate summaries. This process involves natural language processing and text summarization techniques, such as text compression, keyword extraction, etc. The generated summaries may be further optimized to ensure that they are not only concise, but also accurately reflect the main characteristics of the entity.

[0069] The specific process is as follows:

[0070] For each entity , containing the name of its entity and the entity description For a given summary instruction, For the generated summary, is the optimized summary.

[0071] Instruction design: ,in Indicates that according to the original entity and description The process of designing summary instructions.

[0072] Generate summary: ,in Indicates the use of LLMs based on entities and descriptions and instructions The process of generating summaries.

[0073] Summary Assessment: in Represents the summary generated by the evaluation quality process.

[0074] Summary optimization: If the evaluation results Indicates that the summary needs to be optimized, then: ,in For the updated summary instruction According to the evaluation results Update the process of summary instructions.

[0075] Final summary: The optimized summary It is the final output of entity enhancement.

[0076] 2. Semantic information enhancement based on multimodal large language models. Information enhancement based on multimodal large language models aims to enrich the description of text mentions with visual information and improve the accuracy of entity linking. Input information: For each text mention, the system considers not only the text content itself, but also the images or other visual information related to it. The process is as follows: Multimodal large language models are used to process this multimodal information. These models can understand the text content and extract relevant features from the visual information. Multimodal large language models generate a more comprehensive information description by fusing text and visual features. This helps capture the rich semantics of information in a multimodal context. The information description generated by the multimodal large language model aims to refine and emphasize the key aspects of the information, which may include the main features of the entity, relevant attributes, or specific details related to the visual information, including the name, type, functional and behavioral characteristics, physical or structural attributes of the entity, and visual features such as spatial structure, appearance morphology, color texture, etc. extracted from visual information. Similar to entity enhancement, mention enhancement may also use specific prompt templates to guide the multimodal large language model to generate more accurate and relevant information descriptions. The generated enhanced information description is then used in the entity linking process to match candidate entities to determine the best link.

[0077] The multimodal information is expressed as: For text related Information, is the visual information associated with the mention (e.g., image features), Enhanced information for fusing text and visual information using multimodal large language models.

[0078] Design Tips Template: ,in For the prompt template, For textual mention information, For visual information, Represents a function that designs a prompt template based on textual information and visual information. The prompt template is used to guide the multimodal large language model to generate enhanced mention descriptions.

[0079] Generate enhanced semantic description: ,in is the semantic description of the initial enhancement, Represents text information and visual information The fusion result of Indicates the use of multimodal large language models and prompt templates The fused information is processed and a function is generated to enhance the semantic description.

[0080] Evaluate the enhanced semantic information description: ,in To evaluate the results, Represents the enhanced semantic description generated by the evaluation A function of quality, evaluating content including semantic completeness, accuracy, and consistency with the original information;

[0081] Optimize prompt template (if necessary): If the evaluation result Indicates that optimization is needed, then: ,in For the updated prompt template, According to the evaluation results Function that updates the prompt template.

[0082] Regenerate enhanced semantic descriptions: Use updated hint templates Regenerate enhanced mention description: , in, Represents text information and visual information The fusion result, Mention description for regenerated enhancements.

[0083] Final enhanced semantic description: Optimized enhanced mention description It is the final output of the mention enhancement process, which contains the fusion of textual semantic information and visual feature information, and is used for subsequent entity linking tasks.

[0084] 3. Retrieval enhancement aims to expand and optimize the candidate entity set through retrieval technology, thereby improving the accuracy of entity linking. The detailed process of retrieval enhancement is as follows:

[0085] Initialization of candidate entity set: Use string matching algorithm to match each text mention information Processing is performed to generate an initial set of candidate entities by comparing the character similarity between textual mentions and entities in the knowledge base. ,The set contains entities that have character overlap with the text mentions.

[0086] Use pre-trained embedding models: Use language models pre-trained on large-scale text corpora to embed text mention information and the candidate entity set Each entity in is mapped to the vector space to obtain the embedding vector representation of the text mention and the embedding vector representation of the candidate entity ,Vectors are used to capture the semantic features of text.

[0087] Retrieval and re-ranking: Computing enhanced mention descriptions The embedding vector and candidate entity set The cosine similarity between the embedding vectors of each entity in is used to obtain the similarity score, and the candidate entities are sorted in descending order according to the similarity score to obtain the sorted candidate entity set. .

[0088] Expanding the set of candidate entities: Based on the enhanced mention description , perform similarity retrieval in the knowledge base, select entities whose similarity scores exceed the preset threshold, and add them to the initial candidate set Add the high similarity entities other than to the candidate set to obtain the expanded candidate entity set .

[0089] Optimize the candidate entity set: After the expansion of the candidate entity set Apply the entity disambiguation algorithm to analyze the semantic relationship and co-occurrence pattern between entities, eliminate irrelevant candidate entities, and obtain the final optimized candidate entity set ,The set contains both the related entities of the initial match and the ,candidate entities obtained after semantic expansion, thus improving the recall and ,precision of entity linking.

[0090] The retrieval enhancement process can be expressed by the following formula: ; is the initial candidate entity set obtained by string matching, is the enhanced mention description obtained, is the set of candidate entities after retrieval expansion, corresponding to the set of expanded candidate entities, is a retrieval function that retrieves and re-ranks candidate entities based on the similarity between the mention description information and the embedding vectors of the candidate entities;

[0091] Expanded candidate entity set The final optimized candidate entity set is obtained through entity disambiguation: , is the entity disambiguation function, which is used to optimize the candidate entity set; It is the final optimized candidate entity set.

[0092] 4. Decision selection uses a large language model to select the best match from the candidate entity set obtained in the retrieval enhancement step. The specific process is as follows:

[0093] Candidate entity set: retrieve the enhanced candidate entity set , including multiple candidate entities related to the text mention.

[0094] Use a large language model for selection: Using a pre-trained and fine-tuned large language model as a selector, each candidate entity is evaluated to determine which one best matches the textual mention.

[0095] Evaluation and Comparison: The large language model evaluates each candidate entity in the following way:

[0096] (1) Calculate semantic similarity: Use cosine similarity to calculate mention descriptions The similarity score with the embedding vector of the candidate entity ;

[0097] (2) Calculate context relevance: Calculate the context relevance score based on the context information of the mention description and the candidate entity ;

[0098] (3) Calculate attribute matching: Compare the attribute features of the description and the candidate entity to obtain the attribute matching score ;

[0099] (4) Comprehensive score: The above scores are combined in a weighted manner to obtain the final score. ,in is the weight coefficient.

[0100] Select the best match: Based on the evaluation results, select the final score The top candidate entity is taken as the link result.

[0101] Using prompt templates: Designing prompt templates , contains the objectives, evaluation criteria and constraints of the entity linking task, which are used to guide the selection of large language models.

[0102] Consider multimodal information: Integrate textual and visual features in the decision-making process, including:

[0103] (1) Text features: text description, attribute information, and context information of the entity;

[0104] (2) Visual features: spatial structure, appearance features, and visual context information related to entities;

[0105] (3) Feature fusion: Multimodal fusion of text and visual features to generate comprehensive feature representation .

[0106] Iterative optimization: If the confidence of the link result is lower than the preset threshold , the following optimizations are performed: (1) adjusting the content and structure of the prompt template; (2) updating the parameters of the large language model, including adjusting the attention weights and updating the model parameters by automatically building a correct instruction dataset for entity connection; (3) optimizing the similarity calculation method, including adjusting feature weights and improving the measurement function; until the confidence of the link result exceeds the threshold or the maximum number of iterations is reached.

[0107] Output final result: The decision selection process outputs an entity link result that best matches the text mention , the result meets the preset confidence requirements and completes the entire entity linking task.

[0108] The decision-making selection process can be expressed by the following formula: ;in is the best entity selected, is the final optimized candidate entity set, is the enhanced mention description obtained, It is a prompt template for design; is the result of multimodal feature fusion. Representative representation from the candidate entity set Select the entity that maximizes the output score of the large language model; It is a large language model used to select the best entity. Its decision process includes: Construct input representation; compute composite score , at the confidence threshold The entity with the highest score is selected under the constraint of ; if the confidence requirement is not met, an iterative optimization process is performed.

[0109] Example 2

[0110] Scenario setting: Assume that we are processing data from an intelligent traffic monitoring system. This system contains image data of vehicles and related text descriptions. Our task is to correctly link the vehicles captured in the monitoring to the corresponding entities in the vehicle information knowledge base.

[0111] Entity enhancement process:

[0112] 1. Original entity description:

[0113] In the vehicle knowledge base, the description of a high-end electric vehicle may contain a lot of detailed information, including its complete technical specifications, performance parameters, various configuration models, etc.

[0114] 2. Summary Generation: Using a large language model, we generate a concise summary:

[0115] "This is a luxury pure electric sedan that adopts a dual-motor all-wheel drive system, has a streamlined body design, and is equipped with an intelligent driving assistance system. Features include a panoramic glass roof, hidden door handles, and an intelligent central control system. The body size meets the standards of medium and large sedans."

[0116] Semantic information enhancement process:

[0117] 1. Original text information:

[0118] "Monitoring point A12 captured a white high-end electric sedan with a modern body design and an intelligent central control system."

[0119] 2. Image analysis:

[0120] The multimodal large language model analyzes surveillance images and identifies key features of the vehicle, including body contours, design details, and unique appearance characteristics.

[0121] 3. Enhanced semantic information description:

[0122] Combining text and image information, the multimodal large language model generates an enhanced semantic information description:

[0123] "Surveillance point A12 captured a white luxury electric sedan with a streamlined body outline, a panoramic glass roof, hidden door handles, and a large-size intelligent central control system. The overall size belongs to the mid-to-large sedan level."

[0124] Retrieval enhancement process:

[0125] 1. Initial candidate entity set:

[0126] Through string matching, we obtain a candidate set of entities containing multiple high-end electric sedans that meet the characteristics.

[0127] 2. Retrieval and rearrangement:

[0128] Using the pre-trained embedding model in the automotive domain, the candidate entity set was retrieved and re-ranked according to the similarity between the enhanced mention description and the candidate entity, with the most matching car models at the top.

[0129] Decision making process:

[0130] 1. Candidate Entity Assessment:

[0131] The large language model evaluates the retrieved enhanced set of candidate entities by comparing how well the specific features of each candidate model match the observed features.

[0132] 2. Choose the best match:

[0133] The large language model selects the car model with the most matching features as the best match based on the content of the enhanced mention description, and this selection is supported by both image and text information.

[0134] Final result:

[0135] Through the above process, the vehicle information captured by the monitoring system is successfully linked to the corresponding entity in the vehicle knowledge base. This process not only takes into account the detailed text information, but also utilizes the visual features of the vehicle and the powerful capabilities of the large language model, improving the accuracy and efficiency of entity linking in complex intelligent transportation scenarios.

[0136] like Figure 2 As shown in the figure, the accuracy of entity linking is 93.4% under the given data enhancement framework through entity linking experiments. If there is no data processing for entity enhancement (process 1), the accuracy of entity linking is 91.2%; if there is no data processing for semantic information enhancement (process 2), the accuracy of entity linking is 89.1%; if there is no data processing for retrieval enhancement (process 3), the accuracy of entity linking is 82.7%; if there is no decision-making selection processing (process 4), the accuracy of entity linking is 83.5%. The results confirm the effectiveness of the data enhancement framework for entity linking.

[0137] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A multimodal entity connection method based on data enhancement, characterized in that: The following steps are involved: Step 1: Use a large language model to enhance entities: Use a large language model to summarize the entity description, design instructions to clearly indicate the goal of the summary, process the entity description through the designed instructions, extract key information and generate a summary, and optimize the summary; Step 2: Enhance semantic information using a multimodal large language model: The multimodal large language model generates an enhanced information description by fusing text and visual features. The information description is used to refine and emphasize key aspects of the information, including the name, type, function and behavior characteristics, physical or structural attributes of the entity, and the visual features of spatial structure, appearance morphology, color texture extracted from visual information. A prompt template is used to guide the multimodal large language model to generate a more accurate and relevant information description. The generated enhanced information description is used to match candidate entities to determine the best link during the entity linking process. Step 3: Retrieval enhancement: Expand and optimize the candidate entity set through retrieval technology to improve the accuracy of entity linking; The step 3 specifically comprises the following steps: Step 3.1: Initialize the candidate entity set: Use the string matching algorithm to match each text mention information m text_i By comparing the character similarity between the text mention and the entity in the knowledge base, an initial candidate entity set ε is generated. init ,The set contains entities that have character overlap with the text mentions; Step 3.2: Use a pre-trained embedding model: Use a language model pre-trained on a large-scale text corpus to embed the text mention information m text_i and the candidate entity set ε init Each entity in is mapped to the vector space, and the embedding vector representation V mentioned in the text is obtained. mention and the embedding vector representation V of the candidate entity entity ,Vectors are used to capture the semantic features of text; Step 3.3, Retrieval and Re-ranking: Calculate the enhanced mention description m si The embedding vector of init The cosine similarity between the embedding vectors of each entity in is used to obtain the similarity score, and the candidate entities are sorted in descending order according to the similarity score to obtain the sorted candidate entity set ε rank ; Step 3.4: Expand the candidate entity set: Based on the enhanced mention description m si , perform similarity retrieval in the knowledge base, select entities whose similarity scores exceed the preset threshold, and add these entities to the initial candidate set ε init The high similarity entities other than are added to the candidate set to obtain the expanded candidate entity set ε expand ; Step 3.5: Optimize the candidate entity set: Optimize the candidate entity set ε expand By applying the entity disambiguation algorithm, the irrelevant candidate entities are eliminated by analyzing the semantic relationship and co-occurrence pattern between entities, and the final optimized candidate entity set ε' is obtained. The set includes both the initially matched related entities and the candidate entities obtained through semantic expansion, thereby improving the recall rate and accuracy of entity linking. In step 3, the retrieval enhancement process is as follows: e expand =e init ∪f Retriverl (e init ,m si ) ε init is the initial candidate entity set obtained by string matching in step 3.1, m si is the enhanced mention description obtained in step 2.5, ε expand is the expanded candidate entity set after retrieval, corresponding to the expanded candidate entity set in step 3.4, f Retriverl is a retrieval function that retrieves and re-ranks candidate entities based on the similarity between the mention description information and the embedding vectors of the candidate entities; The expanded candidate entity set ε expand The final optimized candidate entity set is obtained through entity disambiguation: ε' = f Disambiguate (ε expand ), f Disambiguate is the entity disambiguation function, which is used to optimize the candidate entity set, corresponding to the optimization process in step 3.5; ε' is the final optimized candidate entity set, that is, the output result of step 3.5, which will be used for the decision-making selection in step 4; Step 4: Decision selection uses a large language model to select the best match from the set of candidate entities obtained through retrieval enhancement.

2. The multimodal entity connection method based on data enhancement according to claim 1 is characterized in that: The step 1 specifically comprises the following steps: Step 1.1, instruction design: I = f instruction (E i ), where I is the given instruction, f instruction Indicates that according to the original entity and description E i (e name_i , e disc_i ) The process of designing instructions, E i Represents an entity, e name_i Indicates the name of the entity, e disc_i Represents a description of an entity; Step 1.2, generate summary: S = f summarize (E i , I), where f summarize Indicates the use of a large language model based on entity names and descriptions E i (e name_i , e disc_i ) and instruction I generate summary S; Step 1.

3. Summary evaluation: Ev = f evalute (S), where Ev represents the evaluation result, f evalute represents the process of evaluating the quality of the generated summary S; Step 1.4, summary optimization: If the evaluation result Ev indicates that the summary needs to be optimized, then: I' = f update_instruction (Ev, I), O = f summarize (E i , I'), where I' is the updated summary instruction, O is the optimized summary, and f update_instruction Represents the process of updating the summary instruction according to the evaluation result Ev; Step 1.5, final summary: The optimized summary O is the final output of entity enhancement.

3. The multimodal entity connection method based on data enhancement according to claim 2 is characterized in that: The step 2 specifically includes the following steps: Step 2.

1. Design prompt template: I mention =f instruction (m text_i , v i ), where I mention For the prompt template, m text_i For textual mention information, v i is the visual information, f instruction represents a function for designing a prompt template based on text information and visual information. The prompt template is used to guide the multimodal large language model to generate enhanced mention descriptions; Step 2.2: Generate enhanced semantic description: m' si =f mention_augment (C(m text_i , v i ), I mention ), where m' si is the initial enhanced semantic description, C(m text_i , v i ) indicates text information m text_i and visual information i The fusion result, f mention_augment Represents the use of multimodal large language models and prompt templates I mention Process the fused information and generate a function to enhance the semantic description; Step 2.3: Evaluate the enhanced semantic information description: Fv = f evalute (m' si ), where Fv is the evaluation result, f evalute Represents the enhanced semantic description m' generated by the evaluation si A function of quality, evaluating content including semantic completeness, accuracy, and consistency with the original information; Step 2.4, optimization prompt template: If the evaluation result Fv indicates that optimization is needed, then: I' mention =f update_instruction (Fv, I mention ), where I' mention is the updated prompt template, f update_instruction Represents a function for updating the prompt template according to the evaluation result Fv; Step 2.5: Regenerate the enhanced semantic description: Use the updated hint template I' mention Regenerate enhanced mention information description: m si =f mention_augment (C(m text_i , v i ), I' mention ), where C(m text_i , v i ) indicates text information m text_i and visual information i The fusion result, m si Mention description for regenerated enhancements; Step 2.6, final enhanced semantic description: the optimized enhanced mention description m si It is the final output of the mention enhancement process, which contains the fusion of textual semantic information and visual feature information, and is used for subsequent entity linking tasks.

4. The multimodal entity connection method based on data enhancement according to claim 1, characterized in that: The step 4 specifically comprises the following steps: Step 4.1, candidate entity set: retrieve the enhanced candidate entity set ε', including multiple candidate entities related to the text mention; Step 4.2: Use the large language model for selection: Use the pre-trained and fine-tuned large language model as a selector to evaluate each candidate entity to determine which entity best matches the text mention; Step 4.3, Evaluation and Comparison: The large language model evaluates each candidate entity in the following way: (1) Calculate semantic similarity: Use cosine similarity to calculate the mention description m si The similarity score S between the embedding vector of the candidate entity sem ; (2) Calculate context relevance: Calculate the context relevance score S based on the context information of the mention description and the candidate entity. context ; (3) Calculate attribute matching: Compare the attribute features of the mentioned description and the candidate entity to obtain the attribute matching score S attr ; (4) Comprehensive score: The above scores are combined in a weighted manner to obtain the final score S final =w1S sem +w2S context +w3S attr , where w1, w2, w3 are weight coefficients; Step 4.4: Select the best match: Based on the evaluation results of step 4.3, select the final score S final The highest candidate entity is taken as the link result; Step 4.5: Use the prompt template: Design the prompt template P template , contains the objectives, evaluation criteria and constraints of the entity linking task, which are used to guide the selection of large language models; Step 4.6: Consider multimodal information: Integrate textual and visual features in the decision process, including: (1) Text features: text description, attribute information, and context information of the entity; (2) Visual features: spatial structure, appearance features, and visual context information associated with entities; (3) Feature fusion: Multimodal fusion of text and visual features to generate a comprehensive feature representation F multi ; Step 4.7, iterative optimization: If the confidence of the link result is lower than the preset threshold θ conf , the following optimizations are performed: (1) adjusting the content and structure of the prompt template; (2) updating the parameters of the large language model, including adjusting the attention weight and updating the model parameters by automatically building a correct instruction dataset for entity connection; (3) optimizing the similarity calculation method, including adjusting the feature weight and improving the metric function; until the confidence of the link result exceeds the threshold or reaches the maximum number of iterations; Step 4.8: Output the final result: The decision selection process outputs an entity link result E that best matches the text mention * , the result meets the preset confidence requirements and completes the entire entity linking task.

5. The multimodal entity connection method based on data enhancement according to claim 4 is characterized in that: In step 4, the decision-making process is as follows: Where E * is the selected best entity, i.e., the link result in step 4.8; ε' is the final optimized candidate entity set in step 3.5, m si is the enhanced mention description obtained in step 2.5, P template is the prompt template designed in step 4.5; F multi is the result of multimodal feature fusion in step 4.6, entity represents entity, The representative entity that selects the entity with the largest output score of the large language model from the candidate entity set ε'; LLM is a large language model used to select the best entity, and its decision process includes: according to the prompt template P template Construct the input representation; calculate the comprehensive score S defined in step 4.3 final , at the confidence threshold θ conf Select the entity with the highest score under the constraint; if the confidence requirement is not met, perform the iterative optimization process of step 4.7.

Images