Multimodal fusion entity retrieval enhancement generation method and device

Abstract

The embodiment of the invention provides a multi-modal fusion entity retrieval enhancement generation method and device, and the method comprises the steps: carrying out the blocking and adaptive text extraction of multi-modal data in an offline stage, obtaining the text block data corresponding to each modal data, extracting the entity and relation of the text block data according to a language model, constructing an entity triple, and carrying out the segmentation and adaptive text extraction of the entity triple. Fusing the entity triad with the text block data to obtain an offline knowledge graph, and constructing a data index; in the present stage, a query statement of a user is received, text block data most similar to the query statement are retrieved in a knowledge graph through a data index, after entity aggregation is carried out on the retrieved text block data, the text block data are reordered according to retrieval scores, an entity aggregation result is obtained, entity ordering is carried out according to the entity aggregation result, and the entity aggregation result is obtained. By means of the multi-modal retrieval method and device, the efficiency and accuracy of multi-modal retrieval can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a multimodal fusion entity retrieval enhancement generation method and apparatus. Background Technology

[0002] In today's era of rapid information technology development, the explosive growth of multimodal data (including text, images, tables, audio, video, and sensor data) has brought unprecedented challenges and opportunities to information retrieval and generation technologies. Traditional text retrieval-enhanced generation systems mainly focus on single-modal data processing, making it difficult to meet users' needs for comprehensive cross-modal information retrieval and generation.

[0003] Multimodal data processing faces two core challenges: difficulties in multimodal data fusion and low efficiency in knowledge graph construction. Existing Retrieval Augmentation (RAG) systems suffer from insufficient dynamic query capabilities and low relevance of search results when processing multimodal data. Specifically, existing systems mostly employ static retrieval mechanisms (such as Elasticsearch inverted indexes), which struggle to support complex query types such as filtering, statistics, and calculations. Traditional retrieval methods do not fully consider the global semantic relationships of entities (such as PageRank scores), resulting in insufficient matching between search results and user needs.

[0004] To address the aforementioned issues, there is an urgent need for a multimodal fusion-based entity retrieval enhancement generation method that can effectively integrate multimodal data and improve the efficiency and accuracy of multimodal retrieval. Summary of the Invention

[0005] To address the problems in the prior art, this application provides a multimodal fusion entity retrieval enhancement generation method and apparatus, which can improve the efficiency and accuracy of multimodal retrieval.

[0006] To solve at least one of the above problems, this application provides the following technical solution: Firstly, this application provides a multimodal fusion-based entity retrieval enhancement generation method, including: In the offline data processing stage, the preset multimodal data is divided into blocks to determine the block data of each modality. Based on the characteristics of each modality, text extraction is performed on the block data to determine the corresponding text block data. Entity recognition and relation extraction are performed on the text block data according to the preset language model to determine the corresponding entity triples. The entity triples are fused with the text block data to determine the offline knowledge graph and construct the corresponding inverted index and vector index. In the online data processing stage, a multimodal query statement input by the user is received, the multimodal query statement is vectorized, the corresponding query vector is determined, and a multi-way hybrid retrieval is performed on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results. The preliminary retrieval results include all entity triples, text block data, and retrieval scores related to the query vector. Entity aggregation is performed on the preliminary retrieval results, and the text block data after entity aggregation is reordered according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results. The entity aggregation results include entities and text block data. The entities are reordered based on the entity aggregation results. Prompt words are constructed based on the reordered entity ranking results. The prompt words are input into a preset language model to obtain the text answer generated by the language model. The text answer is verified based on the offline knowledge graph. The text answer with a confidence level higher than the threshold is output as the final search result.

[0007] Furthermore, the step of dividing the preset multimodal data into blocks to determine the blocks of data for each modality includes: For text modal data, a segmentation strategy based on semantics or fixed-length tokens is used to divide the data and determine the corresponding text segments. For image modal data or table modal data, each image or table entity is independently divided into a block, and the corresponding image block data or table block data is determined respectively. For audio modal data, segmentation is performed according to preset time windows or speech segments to determine the corresponding audio block data; For video modal data, segmentation is performed according to key frame sequences or scene change nodes to determine the corresponding video block data; For sensor modal data or structured data modal data, divide the data into blocks according to data acquisition points, time series or structured fields, and determine the corresponding sensor block data or structure block data respectively.

[0008] Furthermore, the step of extracting text from the segmented data based on each modal characteristic to determine the corresponding text block data includes: Text information is extracted from the image block data and the table block data using optical character recognition technology; The audio segment data is translated into text information using automatic speech recognition technology; Text information is extracted from the video block data using video frame extraction, keyframe optical character recognition, or video content analysis techniques. Text information is extracted from the sensor block data and the structure block data through key field parsing, data transformation, or structured query methods. All extracted text information is stored in a unified format according to the text block data, and the corresponding text block data is determined.

[0009] Further, the step of performing a multi-path hybrid retrieval on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results includes: Based on the query vector in the offline knowledge graph, full-text retrieval of text block data is performed according to the inverted index to determine the corresponding full-text retrieval score, and semantic similarity retrieval of text block data is performed according to the vector index to determine the corresponding semantic similarity retrieval score. After normalizing the full-text search score and the semantic similarity search score, they are weighted and fused according to preset weights to obtain the corresponding preliminary search results. The preliminary search results include relevant text block data, the search score corresponding to the text block data, and relevant entity triples. The relevant entity triples include entity information, attribute information, and relation information from the text block data.

[0010] Further, the preliminary search results are subjected to entity aggregation, and the text block data after entity aggregation is re-ranked according to the search score and the text block data score in the offline knowledge graph to determine the corresponding entity aggregation result. The entity aggregation result includes entities and text block data, including: The text block data, attribute information and relation information pointing to the same entity in the preliminary search results are merged and integrated to determine the aggregation unit with the entity as the core. The retrieval scores of all text block data in the aggregation unit and the data scores of all entities associated with the text block data in the offline knowledge graph are weighted to determine the corresponding text block data re-ranking score. Based on the re-ranking score, the text block data in the aggregation unit is re-ranked to determine the corresponding entity aggregation result, which includes entities and text block data.

[0011] Further, the entity reordering based on the entity aggregation result includes: If the query type of the query statement is a normal query, then the entities are reordered according to the entity aggregation results to determine the corresponding reordered entity sorting result; If the query type of the query statement is a complex query, then a SPARQL query is executed on the entity aggregation result, and the entities are reordered according to the entity aggregation result after the SPARQL query to determine the corresponding reordered entity sorting result. The entity reordering process includes calculating a comprehensive score for each entity and the query statement based on the triplet matching degree, text semantic similarity, and image feature similarity of each entity in the entity aggregation result; and sorting the entities in the entity aggregation result in descending order according to the comprehensive score to generate a reordered entity ranking result.

[0012] Furthermore, the step of constructing prompt words based on the reordered entity sorting results includes: Based on the reordered entity sorting results, a predetermined number of entities are selected in descending order of entity priority. For each selected entity, based on its attribute priority and relation priority, high-priority attributes and relations are selected from the set of triples corresponding to that entity to determine the corresponding entity triple subset; The query statement, the entity triplet subset, and the text block data corresponding to the entity are combined to determine the prompt words required for the final answer.

[0013] Secondly, this application provides a multimodal fusion-based retrieval enhancement generation apparatus, comprising: The offline data processing module is used in the offline data processing stage to perform block processing on the preset multimodal data, determine the block data of each modality, extract text from the block data based on the characteristics of each modality, determine the corresponding text block data, perform entity recognition and relation extraction on the text block data according to the preset language model, determine the corresponding entity triples, fuse the entity triples with the text block data, determine the offline knowledge graph and construct the corresponding inverted index and vector index; An online data processing module is used in the online data processing stage to receive multimodal query statements input by users, vectorize the multimodal query statements, determine the corresponding query vector, perform multi-way hybrid retrieval on the query vector based on the inverted index and the vector index, determine the corresponding preliminary retrieval results, the preliminary retrieval results include all entity triples, text block data and retrieval scores related to the query vector, perform entity aggregation on the preliminary retrieval results, and reorder the text block data after entity aggregation according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results, the entity aggregation results include entities and text block data; The complex query result determination module is used to reorder entities based on the entity aggregation results, construct prompt words based on the reordered entity ranking results, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than a threshold as the final search result.

[0014] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal fusion entity retrieval enhancement generation method.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal fusion entity retrieval enhancement generation method.

[0016] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal fusion entity retrieval enhancement generation method.

[0017] As can be seen from the above technical solution, this application provides a multimodal fusion entity retrieval enhancement generation method and apparatus. In the offline stage, multimodal data is segmented and adaptively extracted to obtain text block data corresponding to each modality. Entities and relations of the text block data are extracted according to the language model to construct entity triples. The entity triples are fused with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, the user's query statement is received, and the most similar text block data to the query statement is retrieved in the knowledge graph through the data index. After entity aggregation of the retrieved text block data, the text block data is re-ranked according to the retrieval score to obtain the entity aggregation result. If the query statement is a complex query, the entity aggregation result is further sorted, and then prompt words are constructed and input into the language model to obtain the text answer. This can improve the efficiency and accuracy of multimodal retrieval. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the multimodal fusion entity retrieval enhancement generation method in this application embodiment; Figure 2 This is a structural diagram of the multimodal fusion retrieval enhancement generation device in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0020] Figure label: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0023] Considering the difficulties in fusion and low efficiency in knowledge graph construction, as well as the insufficient dynamic query capabilities and low relevance of search results in the retrieval enhancement generation system, this application provides a multimodal fusion entity retrieval enhancement generation method and apparatus. In the offline stage, multimodal data is segmented and adaptively extracted to obtain text block data corresponding to each modality. Entities and relations are extracted from the text block data according to the language model, and entity triples are constructed. The entity triples are fused with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, the user's query statement is received, and the most similar text block data to the query statement is retrieved from the knowledge graph through the data index. After entity aggregation of the retrieved text block data, the text block data is re-ranked according to the search score to obtain the entity aggregation result. If the query statement is a complex query, the entity aggregation result is further sorted, and prompt words are constructed and input into the language model to obtain the text answer. This improves the efficiency and accuracy of multimodal retrieval.

[0024] To improve the efficiency and accuracy of multimodal retrieval, this application provides an embodiment of a multimodal fusion-based entity retrieval enhancement generation method, see [link to embodiment]. Figure 1 The multimodal fusion entity retrieval enhancement generation method specifically includes the following: Step S101: In the offline data processing stage, the preset multimodal data is divided into blocks to determine the block data of each modality. Based on the characteristics of each modality, text extraction is performed on the block data to determine the corresponding text block data. Entity recognition and relation extraction are performed on the text block data according to the preset language model to determine the corresponding entity triples. The entity triples are fused with the text block data to determine the offline knowledge graph and construct the corresponding inverted index and vector index. Optionally, in this embodiment, the system first performs unified block processing on the preset multimodal data.

[0025] Multimodal data includes text, images, tables, audio, video, and sensor data. The segmentation strategy is tailored to the characteristics of each modality. Text data is divided into semantic segments or fixed lengths (e.g., 512 tokens) to ensure semantic integrity; image and table data are processed as independent units, preserving their structural information; audio data is segmented into fixed time windows (e.g., 30 seconds) or natural speech segments; video data is segmented based on keyframes or scene change nodes; and sensor data is segmented according to time series or data acquisition points. The core function of this step is to transform heterogeneous, continuous multimedia data into discrete, processable, standardized data units, laying the foundation for subsequent unified text extraction and semantic analysis.

[0026] After data segmentation, text extraction is performed on the segmented data based on the characteristics of each modality. For image and table segments, Optical Character Recognition (OCR) technology is used to extract text information; for audio segments, Automatic Speech Recognition (ASR) technology is used to translate speech content into text; for video segments, text content is extracted by analyzing keyframes or combining speech transcription and subtitle information; sensor or structured data is converted into readable text descriptions by parsing its key fields, metadata, or numerical records. All extracted text is organized into a unified structured format (such as JSON) to ensure that text information from different sources has a consistent representation. This step achieves semantic conversion from non-textual modalities to textual modalities, building an interoperable text bridge between multimodal data and providing clean and standardized input for subsequent knowledge extraction.

[0027] Next, a pre-defined Large Language Model (LLM) is used to perform deep semantic analysis on the aforementioned text blocks, executing entity recognition and relation extraction tasks. The model receives the merged text content, identifies the entities mentioned in the text (such as people, places, concepts, etc.), and further determines the semantic relationships between these entities (such as "belongs to," "located in," "created in," etc.), finally outputting structured triples in the form of (subject, predicate, object). This process achieves automated conversion from unstructured text to structured knowledge, significantly improving the coverage and efficiency of knowledge extraction, and avoiding the high dependence on manual annotation in traditional methods.

[0028] For example, using a Large Language Model (LLM) to perform entity recognition and relation extraction on merged text, generating triples:

[0029] in: Subject: Represents the entity in the triple; Predicate: Represents an attribute or relation in a triple; Object: Represents the value in the triplet; Text: Indicates the input text.

[0030] Subsequently, the extracted entity triples are fused with the original text block data. The triples are imported into a knowledge graph construction engine and modeled according to the RDF (Resource Description Framework) specification to form an offline knowledge graph with entities as nodes and relations as edges. Simultaneously, the original text block data is associated with the graph as supplementary attributes or sources of evidence for the entity nodes. This fusion process not only constructs a structured knowledge network rich in semantic relationships but also preserves the contextual information of the original data, enabling the knowledge graph to possess both semantic accuracy and traceability.

[0031] Finally, a dual-mode indexing mechanism is constructed for the knowledge graph and its associated text data. On the one hand, a full-text search index is built based on inverted index (ES) technology to support efficient keyword matching; on the other hand, an embedding model is used to convert the text into high-dimensional vectors and construct a vector index (FAISS) to support approximate retrieval based on semantic similarity. The construction of these two indexes provides efficient and flexible data access capabilities for the online retrieval stage, enabling it to simultaneously support both exact matching and semantic similarity retrieval modes.

[0032] Step S101, through a systematic process of segmentation, extraction, fusion, and indexing, successfully transforms the original multimodal heterogeneous data into a structured knowledge graph and an efficient retrieval index. Its core effect lies in breaking down the semantic barriers between multimodal data, constructing a unified knowledge representation, and providing a solid data foundation for accurate and dynamic retrieval and generation in the online phase.

[0033] Step S102: In the online data processing stage, a multimodal query statement input by the user is received, the multimodal query statement is vectorized, the corresponding query vector is determined, and a multi-way hybrid retrieval is performed on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results. The preliminary retrieval results include all entity triples, text block data, and retrieval scores related to the query vector. Entity aggregation is performed on the preliminary retrieval results, and the text block data after entity aggregation is reordered according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results. The entity aggregation results include entities and text block data. Optionally, in this embodiment, during the online data processing, a multimodal query statement input by the user is first received. This query statement may contain information in the form of text, images, audio, or other modalities.

[0034] The system parses queries using a multimodal parsing module to identify the various information components they contain. For example, for queries containing both text and images, the system extracts textual and visual features respectively. Subsequently, the system uses a pre-trained multimodal embedding model to map these heterogeneous features into a unified vector space, generating a corresponding query vector representation. This vectorization process ensures that information from different modalities can be used for similarity calculation and retrieval within the same semantic space.

[0035] After obtaining the query vector, the system simultaneously utilizes the inverted index and vector index built in the offline phase for multi-path hybrid retrieval. The inverted index, based on keyword matching, can quickly recall text blocks and entity triples containing specific words in the query; while the vector index searches for relevant content from a semantic level by calculating the cosine similarity between the query vector and vectors in the index. The two retrieval paths are executed in parallel, generating candidate result sets based on keyword matching and semantic similarity, respectively.

[0036] Specifically, the calculation process of the inverted index (ES) candidate result set:

[0037] in: D: Target text; : term Word frequencies in text D; The number of terms in the text; : The average length of all texts; Inverse document frequency; The word frequency saturation parameter is typically set between 1.2 and 2.0. : Length normalization parameter, usually taken as 0.5~0.8.

[0038] Specifically, the calculation process of the candidate result set for the vector index (FAISS):

[0039] Where q and d are the embedding vectors of the query and the text, respectively.

[0040] The two search results are then merged. By assigning weights (e.g., 0.5 each) to the inverted index search score and the vector search score, a comprehensive search score (RS) is calculated for each candidate text chunk and entity triple. This score reflects the overall degree of matching between the candidate content and the user query in terms of both direct match and semantic relevance. At this point, the system obtains a preliminary search result set, which includes all query-related entity triples, text chunk data, and their corresponding search scores.

[0041] Specifically, the preliminary search score calculation is as follows:

[0042] Text Blocking The retrieval score is used to evaluate the direct matching degree of text chunks. Next, entity aggregation is performed on the preliminary search results. The core purpose of this step is to merge and integrate information that points to the same entity but is scattered across different text blocks.

[0043] By identifying entities mentioned in text chunks and triples, entity aggregation groups all attributes, relationships, and text chunks mentioning the same entity together to form a complete entity view. For example, if multiple text chunks mention different information about "Company X" (such as establishment date, main business, CEO, etc.), entity aggregation will associate all this information with the entity "Company X". This process greatly enriches the information completeness at the entity level, laying a data foundation for subsequent processing.

[0044] Based on entity aggregation, the text chunks in the aggregation results are further reordered. That is, all retrieved text chunks associated with the entity "Company X" are reordered.

[0045] The ranking at this point not only relies on the search score (RS) obtained during the initial retrieval, but also incorporates the global importance score of the text chunks. The global importance score of the text chunks is represented by the PageRank score (PR) calculated from the knowledge graph in the offline stage.

[0046] Retrieval score (RS) reflects the direct relevance of a text chunk to the current query and is a measure of local relevance; while PageRank score (PR) reflects the global importance and centrality of the text chunk in the entire knowledge graph and measures the value of the information it contains.

[0047] Specifically, the calculation process for the global importance score (PR) of text chunks is as follows:

[0048] in: d: Damping coefficient (usually taken as 0.85), N=|E| is the total number of entities; : Pointing to an entity A collection of text blocks; : Pointing to an entity Other entity sets; Block The set of all entities it points to; :entity The set of all entities it points to; Normalization: Denominator Ensure a balanced weight distribution.

[0049]

[0050] in: M = |C| represents the total number of blocks; : Points to blocks entity set Optimize the ranking of text chunks by combining the retrieval score of text chunks with the PageRank score of related entities:

[0051] in: : The new score for the sorting of the i-th text block; : The retrieval score of the i-th text block, used to evaluate the direct matching degree of the text block; : The weighted average of the PageRank scores of all entities associated with the i-th text chunk, used to evaluate the global relevance of the text chunk; Weighting parameter, used to balance local and global scores.

[0052] After calculating the new scores for the text blocks, the text blocks are reordered in descending order based on the new scores.

[0053] The re-ranking process effectively overcomes the limitations of relying solely on keyword matching or semantic similarity. For example, a text chunk might achieve a high search score because it contains multiple keywords from the query, but it might itself be just a general passage with low information density. On the other hand, another text chunk might have a slightly lower direct literal match to the query, but it comes from a very important document (with a high PageRank value) and contains more in-depth and authoritative details. By combining RS and PR for re-ranking, the system can elevate such high-quality, high-value content to a higher position, thereby greatly optimizing the quality of search results and ensuring that the information returned to users is not only relevant but also important and reliable.

[0054] Finally, the output is a structured entity aggregation result, which includes all entities related to the user query and their rich attributes, and also provides a list of textual evidence supporting these entities, sorted by the new rating.

[0055] Step S103: Reorder entities based on the entity aggregation result, construct prompt words based on the reordered entity ranking result, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than the threshold as the final search result.

[0056] Optionally, in this embodiment, the user's query statement is first dynamically determined. If the current query type is a normal query (such as "find an entity"), entity reordering is performed directly based on the entity aggregation result of step S102, and suggestion words are generated. If the current query type is a complex query (such as filtering, statistics, calculation), a SPARQL query is generated, and entity reordering is performed based on the SPARQL query result, and suggestion words are generated.

[0057] Optionally, in this embodiment, this step mainly describes the steps for processing complex statements.

[0058] For example, consider the difference between simple and complex statements.

[0059] In a simple query, a user might directly ask about the revenue of Pineapple Company. Based on the entity aggregation result from step S102, the entities "Pineapple Company" and "revenue" can already be obtained. Based on the entity aggregation result from step S102, entity reordering can be performed and prompt words can be generated.

[0060] In complex queries, users might ask for "the top three subsidiaries with sales exceeding 100 million and the highest market share in 2023, and their main products." Such queries require the system to not only identify entities like "subsidiaries," "sales," and "products," but also to understand filtering, sorting, and statistical operations such as "exceeding 100 million," "highest," and "top three," and to perform logical connections and comparisons between these entities. Therefore, directly using preliminary search and aggregation results to construct prompts may lead to inaccurate or redundant answers generated by large language models due to overly complex and unclear input information.

[0061] To address this issue, this step introduces a deep processing flow that incorporates semantic understanding and structured reasoning. It transforms natural language queries into structured query commands (SPARQL) that the knowledge graph can understand and execute, and then re-ranks the initially retrieved entities. The entire process aims to solve complex information needs that simple keyword matching or vector similarity retrieval cannot handle.

[0062] Specifically, the system automatically constructs SPARQL queries based on information obtained in the "multimodal parsing" and "entity aggregation" steps. For example, when faced with a query like "find all departments with sales exceeding 100 million yuan in 2023 and ranking in the top three in market share," the system needs to: Identify query intent and constraints: The target entity is identified as "department", and multiple filtering conditions are extracted - "2023", "sales exceeding 100 million yuan", "market share ranked among the top three".

[0063] Mapping knowledge graph elements: Map "department" to a type of entity in the knowledge graph, map "sales revenue" and "market share" to attributes or relationships of these entities, and map "2023" to a time-limited condition.

[0064] Constructing a structured query: This ultimately results in a well-structured SPARQL query. This query instructs the knowledge graph: "Find all entities of type 'Department' that must have one 'Sales' record for 2023 with a value greater than 100 million yuan; sort them in descending order by their 'Market Share' for the same year, and return only the top three." The generated SPARQL query is sent to the RDF knowledge graph database for execution. The knowledge graph engine efficiently traverses the graph structure, performing pattern matching, filtering, joining, and sorting according to the query logic, ultimately returning a precise list of entities that meet all constraints and their related data.

[0065] The execution result of a SPARQL query is already a highly relevant and pre-selected set of entities. However, to provide optimal information priority during the answer generation phase, the system performs a second round of re-ranking. This ranking logic differs from the previous ranking based on search scores; it focuses more on comparing entity attribute values ​​and the degree to which query conditions are met.

[0066] Specifically, the entity re-ranking mechanism uses a multi-dimensional, quantifiable comprehensive evaluation system to accurately score each entity in the aggregation result.

[0067] First, a core evaluation dimension is constructed. The first is triple matching, which measures the degree of matching between the structured knowledge (such as attribute values) associated with the entity and the query conditions. For example, does the "sales revenue" attribute value of entity "Subsidiary A" truly "exceed 100 million"? The second is textual semantic similarity, which measures the semantic relevance between the unstructured text describing the entity and the user's query text. This is calculated using algorithms such as cosine similarity to capture deep semantic relationships. The third is multimodal feature similarity. If the query or entity involves images, videos, or other content, the similarity of their visual features needs to be calculated. By assigning different weights (e.g., 0.5, 0.3, 0.2) to these three dimensions, the system calculates a comprehensive score for each entity and ranks them in descending order. The effect of this approach is that it significantly optimizes the quality of information input into the large model, prioritizing and highlighting the most relevant entities that best meet complex query conditions, thus laying a solid and high-quality data foundation for subsequent answer generation.

[0068] Specifically, the reordering mechanism calculation formula is as follows:

[0069] in: e: Entity data to be sorted (including lists of triples, text representations, and image representations); q: Query (including text representation and image representation); avg_triplet_score: The average matching score of all triples for this entity; TextSimilarity: The semantic similarity between entity text and query text (e.g., CosSim); ImageSimilarity: Visual similarity (e.g., cosine similarity) between the entity image and the query image. , , Weight parameters (default) , , (This can be dynamically adjusted based on the query type).

[0070] After the entity score is calculated, the entities are rearranged in descending order based on the score.

[0071] Next, a prompt is constructed based on the reordered entity ranking. The construction of the prompt involves several key parts: a clear restatement of the user's original question, entities in descending order of importance and their key attributes and relationships (from the reordering results), relevant textual evidence blocks (to provide context), and rules or instructions required to perform complex operations (e.g., "Please perform statistics and list the top three"). Therefore, a crucial optimization step in the construction process is dynamic pruning.

[0072] Based on indicators such as entity priority, attribute relevance, and relationship strength, redundant information that is ranked low or irrelevant is automatically filtered out, ensuring that the generated Prompt is of reasonable length, high information density, and highlights the key points. On the one hand, this reduces unnecessary token consumption, lowering the computational load and API call costs of large models; on the other hand, it avoids interference from irrelevant information, significantly improving the accuracy and relevance of the generated answers, making them more closely aligned with the core requirements of complex queries.

[0073] The specific pruning process includes:

[0074] in: PrunedTriples: The set of entity triples after pruning; Triples: The original set of triples (S, P, O); EntityPriority: A list of entity priorities (e.g., sorted by importance); AttributePriority: A list of attribute priorities (e.g., sorted by relevance); RelationPriority: A list of relationship priorities (e.g., sorted by strength); Filter: Filters redundant triples based on priority.

[0075] The Prompt generation process is optimized by dynamically pruning entity triplet data.

[0076] Next, the constructed Prompt is input into a pre-defined large language model. Leveraging its natural language understanding and generation capabilities, the large model performs deep reasoning and fusion on the structured entity information, unstructured text evidence, and operational instructions in the Prompt, ultimately generating a coherent, natural textual answer that directly addresses the user's question.

[0077] For example, in response to the above query, the model might generate: "Based on 2023 data, the top three subsidiaries with sales exceeding 100 million and the highest market share are: 1. Subsidiary A (main product: smartphones), market share X%; 2. Subsidiary B (main product: cloud computing services), market share Y%; 3. Subsidiary C (main product: smart home), market share Z%." However, the inherent "illusion" problem of large language models may lead to the generation of seemingly reasonable content that does not actually conform to existing knowledge. Therefore, this solution introduces a crucial verification step based on offline knowledge graphs.

[0078] The text answer generated by the model is semantically consistent and deeply semantically validated against the validated RDF knowledge graph built offline. This checks whether the facts claimed in the answer (e.g., "Subsidiary A's sales revenue is 150 million") are consistent with the triples recorded in the knowledge graph (e.g., <Subsidiary A><Sales Revenue>"150 million"). The system calculates a confidence score (between 0 and 1) for the answer. If the score is higher than a preset threshold (e.g., 0.7), it indicates that the generated text answer has high reliability, and the system outputs it as the final search result to the user. The result can be multimodal, containing not only the text answer but also potential source links pointing to the original entities and text chunks supporting the answer, enhancing the interpretability and credibility of the result.

[0079] If the confidence level falls below the threshold after verification, the system triggers a protection mechanism, determining that the currently generated text answer is unreliable. In this case, the system will not output a potentially incorrect text answer, but instead will only return verified structured data results (such as entity lists and attribute values) and related text evidence blocks, proactively prompting the user that "the question may be unclear or too broad," guiding the user to refine their query. This step significantly improves the accuracy and reliability of the system's output, guiding users to obtain the necessary information more effectively through proactive feedback.

[0080] The specific verification process is as follows:

[0081] in: Answer: The text answer generated by the large model; RDFValidation(Answer, KG): The semantic consistency verification score between the answer and the RDF knowledge graph (KG) (0 ≤ Score ≤ 1). AnswerScore: The final score for the text answers generated by the large model.

[0082] When the AnswerScore is below the confidence threshold (e.g., 0.3), the generated text answer is considered unreliable. In this case, the text answer is removed from the final answer, and only entity and text chunk data are provided. At the same time, the system prompts the user that the question is unclear or the scope of the question is too broad, and guides the user to ask the question again.

[0083] When the AnswerScore is higher than the confidence threshold, the text answer is linked to the entity and text chunks to form the final answer, which includes multimodal data such as entities and text chunks.

[0084] This example demonstrates how this embodiment solves the problem of answer generation under complex queries by using a dual offline and online data processing approach: building a knowledge graph and index offline, and performing question queries and complex question entity reordering online.

[0085] As described above, the multimodal fusion entity retrieval enhancement generation method provided in this application can, in the offline stage, segment and adaptively extract text from multimodal data to obtain text block data corresponding to each modality. Based on the language model, entities and relations of the text block data are extracted to construct entity triples. The entity triples are then fused with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, the system receives user queries, retrieves the most similar text block data to the query from the knowledge graph using the data index, aggregates the retrieved text block data into entities, and re-sorts the text block data according to the retrieval score to obtain entity aggregation results. If the query type is a complex query, the entity aggregation results are further sorted, prompt words are constructed, and input into the language model to obtain the text answer. This improves the efficiency and accuracy of multimodal retrieval.

[0086] In one embodiment of the multimodal fusion entity retrieval enhancement generation method of this application, it may further include the following: Step S201: Divide the text modal data using a segmentation strategy based on semantics or fixed-length tokens to determine the corresponding text segment data; Step S202: For image modal data or table modal data, each image or table entity is independently divided into a block, and the corresponding image block data or table block data is determined respectively; Step S203: Segment the audio modal data according to a preset time window or speech segment to determine the corresponding audio block data; Step S204: Segment the video modal data according to the keyframe sequence or scene change nodes to determine the corresponding video block data; Step S205: Divide the sensor modal data or structured data modal data into blocks according to data acquisition points, time series or structured fields, and determine the corresponding sensor block data or structure block data respectively.

[0087] Optionally, in this embodiment, this step involves adopting a targeted segmentation strategy based on the inherent characteristics of different modal data to transform continuous, heterogeneous raw data into discretized, standardized data units that retain key semantics, thus laying the foundation for subsequent unified processing.

[0088] For text modal data, a segmentation strategy based on semantics or fixed-length tokens is adopted. Semantic segmentation ensures that each segment contains a relatively complete semantic unit by identifying paragraph boundaries, punctuation patterns, or using natural language processing models to determine semantic integrity. Fixed-length token segmentation, on the other hand, mechanically divides the data according to a preset number of tokens (e.g., 512), ensuring processing efficiency and data regularity. This approach maintains the semantic coherence of the text content while meeting the technical requirements for input length in subsequent processing.

[0089] For image and table modal data, each image or table is independently divided into a block unit. This processing method fully respects the characteristics of images and tables as holistic information carriers; the visual elements in images and the row and column relationships in tables all possess indivisible structured semantics. Independent block division ensures the structural integrity and semantic independence of these modal data, creating the necessary conditions for accurate extraction of text content through subsequent OCR technology and avoiding semantic distortion caused by information confusion across images or tables.

[0090] For audio modal data, the system segments the data according to preset time windows (e.g., 30 seconds) or speech segments (e.g., speaker switching, semantic segment intervals). Segmentation by duration ensures processing efficiency and data uniformity, suitable for audio materials with uniformly distributed content; segmentation by speech segment better conforms to the natural structure of audio content, and can identify and preserve semantic boundaries such as dialogue turns and topic transitions. This segmentation method transforms continuous audio signals into discrete time segments, providing clear processing units for subsequent speech recognition and text translation, while maintaining the semantic coherence of the audio content.

[0091] Video modal data processing involves segmenting the data according to keyframe sequences or scene transition nodes. Keyframe sequence segmentation is based on extracting keyframes with significant visual content changes, ensuring that each segment represents a relatively stable visual content unit. Scene transition segmentation identifies natural scene boundaries in the video by analyzing features such as shot transitions, background changes, and motion patterns. This effectively solves the problems of large video data volume and high information redundancy, transforming continuous video streams into a set of semantically representative segments, providing an efficient processing foundation for subsequent video content analysis and text extraction.

[0092] For sensor data and structured data, the data can be segmented by data acquisition point, time series, or structured field. Segmentation by acquisition point is suitable for distributed sensor networks, preserving spatial correlation; segmentation by time series is suitable for time-series data, retaining temporal dependencies; and segmentation by structured field is suitable for structured data such as database tables, maintaining logical relationships. This process transforms machine-generated data into processing units with clearly defined contextual boundaries, ensuring that subsequent text processing accurately reflects the original structure and semantic relationships of the data.

[0093] Through step S205, this embodiment successfully transforms raw data from multiple modalities into a unified and standardized set of data units through this series of targeted block processing, providing structurally consistent and semantically complete input data for subsequent text extraction, knowledge extraction, and knowledge graph construction.

[0094] In one embodiment of the multimodal fusion entity retrieval enhancement generation method of this application, it may further include the following: Step S301: Extract text information from the image block data and the table block data using optical character recognition technology; Step S302: The audio block data is translated into text information using automatic speech recognition technology; Step S303: Extract text information from the video block data using video frame extraction, keyframe optical character recognition, or video content analysis techniques; Step S304: Extract text information from the sensor block data and the structure block data through key field parsing, data conversion or structured query methods; Step S305: Store all extracted text information in the unified format of the text block data to determine the corresponding text block data.

[0095] Optionally, in this embodiment, a specialized text extraction technique is used for the segmented data of different modalities to transform non-text content into a unified text representation.

[0096] For image and table segmented data, the system employs Optical Character Recognition (OCR) technology for text extraction. Through image preprocessing, text region detection, character segmentation, and recognition, the system transforms visual text information in the image into editable and searchable text content. For table data, the OCR system can also recognize the table structure, preserve row and column relationships, and ensure that the extracted text not only contains content information but also maintains its original structured features.

[0097] For audio segmented data, the system employs Automatic Speech Recognition (ASR) technology for text translation. The ASR system uses components such as acoustic models, language models, and decoders to convert the speech content in the audio signal into corresponding text representations. During processing, the system considers influencing factors such as different languages, dialects, accents, and background noise, and uses deep learning models to improve recognition accuracy. For long audio segments, the system also performs preprocessing such as speaker separation and speech enhancement to ensure the quality and readability of the translated text.

[0098] Text extraction from segmented video data employs a multi-technology fusion strategy. First, keyframe images are obtained using video frame extraction technology. Then, OCR technology is applied to these keyframes to extract text information such as subtitles, titles, and chart annotations. Simultaneously, video content analysis technology is used, employing methods such as scene recognition, object detection, and action recognition to generate descriptive text metadata. For videos containing audio tracks, ASR technology is applied simultaneously to extract audio content. This multi-pronged approach ensures the comprehensive capture of the rich information contained within the video.

[0099] For both sensor block data and structured block data, the system extracts text information using methods such as key field parsing, data transformation, and structured querying. Sensor data typically contains key fields such as timestamps, measured values, and device identifiers. The system generates descriptive text records by parsing the semantic meaning of these fields. Structured data is converted into natural language descriptions or text representations that retain the original structural relationships through database queries and XML / JSON parsing. This processing method enables machine-readable structured data to be transformed into a human-understandable and machine-processable text format.

[0100] Finally, the system standardizes and stores all extracted text information in a uniform format. Regardless of whether the text is extracted from images, audio, video, or sensor data, it is converted into a consistent text block data structure.

[0101] Through step S305, this embodiment achieves semantic unification of multimodal data. Through specialized extraction techniques, the semantic content and structural information of the original data are preserved to the greatest extent.

[0102] In one embodiment of the multimodal fusion entity retrieval enhancement generation method of this application, it may further include the following: Step S401: Based on the query vector in the offline knowledge graph, perform full-text retrieval of the text block data according to the inverted index to determine the corresponding full-text retrieval score, and perform semantic similarity retrieval of the text block data according to the vector index to determine the corresponding semantic similarity retrieval score. Step S402: After normalizing the full-text search score and the semantic similarity search score, they are weighted and fused according to preset weights to obtain the corresponding preliminary search results. The preliminary search results include relevant text block data, the search score corresponding to the text block data, and relevant entity triples. The relevant entity triples include entity information, attribute information, and relation information from the text block data.

[0103] Optionally, in this embodiment, this step is a multi-path hybrid retrieval and result fusion mechanism.

[0104] Specifically, two different retrieval strategies are executed in parallel on the user query vector. First, full-text retrieval is performed using the inverted index (ES). This process quickly matches all text blocks in the knowledge base containing keywords from the query. Its retrieval score quantifies the degree of matching between the text block and the query at the lexical level, focusing on literal matching and word frequency statistics. Simultaneously, the system uses the Facial Similarity Index (FAISS) for semantic similarity retrieval. This process measures the semantic relevance between the query vector and the text block embedding by calculating the cosine similarity between them. This effectively captures the deep semantic relationships beyond keywords between the query and the text block, such as synonyms, contextual meaning, or topic relevance.

[0105] The two different dimensions and units of retrieval scores are normalized and mapped to the same numerical range (e.g., [0,1]) to ensure their weights are comparable. Then, a weighted fusion is performed according to preset weights (e.g., 0.5 for full-text retrieval and 0.5 for semantic retrieval) to calculate a comprehensive retrieval score (RS) for each candidate text block, achieving a balance between retrieval precision and recall.

[0106] Full-text search ensures high accuracy and interpretability, accurately capturing keyword matches; while semantic search enhances the system's generalization ability, recalling content that, although not containing the query keywords, is highly semantically relevant. The initial search result set generated by the fusion of the two includes relevant text block data and their comprehensive scores, as well as automatically associating the entity, attribute, and relation triples contained in these text blocks.

[0107] Through step S402, this embodiment successfully integrates full-text search and semantic search to obtain a preliminary search result set, which provides a comprehensive and accurate data foundation for subsequent entity aggregation and reordering.

[0108] In one embodiment of the multimodal fusion entity retrieval enhancement generation method of this application, it may further include the following: Step S501: Merge and integrate the text block data, attribute information and relation information pointing to the same entity information in the preliminary search results to determine the aggregation unit with the entity as the core; Step S502: Weight the retrieval scores of all text block data in the aggregation unit and the data scores of all entities associated with the text block data in the offline knowledge graph to determine the corresponding text block data re-ranking score, and re-rank the text block data in the aggregation unit based on the re-ranking score to determine the corresponding entity aggregation result, wherein the entity aggregation result includes entities and text block data.

[0109] Optionally, in this embodiment, this step is an entity aggregation step.

[0110] The initial search results are analyzed in depth to identify all mentioned entities. For each unique entity, the system merges and integrates relevant information scattered across different text blocks. This includes the entity's attribute descriptions in different contexts (such as name, location, time, etc.), assertions of relationships with other entities (such as "belongs to," "located in," "created in"), and all original text block data containing information about that entity. This step breaks down data silos and constructs a rich, structured view of entities, laying a solid foundation for subsequent in-depth processing and accurate sorting.

[0111] Next, the quality of text block data is evaluated within the aforementioned entity aggregation unit, and the presentation order is optimized. A weighted scoring strategy that integrates local relevance and global importance is adopted.

[0112] Specifically, a new score is calculated for each text block, which is a weighted combination of its original retrieval score (representing the direct match with the current query) and the PageRank score of its associated entity (representing the entity's global importance and authority within the entire knowledge graph). This new re-ranking score comprehensively measures both the immediate relevance and long-term value of the text block.

[0113] Based on this new reordering score, all text blocks within an aggregation unit are sorted in descending order. High-quality textual evidence that is not only directly relevant to the user's query but also originates from important entities in the knowledge graph is prioritized for presentation, improving system reliability and user experience.

[0114] Through step S502, this embodiment successfully improved the quality and reliability of search results based on text block reordering.

[0115] In one embodiment of the multimodal fusion entity retrieval enhancement generation method of this application, it may further include the following: Step S601: If the query type of the query statement is a normal query, then the entities are reordered according to the entity aggregation result to determine the corresponding reordered entity sorting result. Step S602: If the query type of the query statement is a complex query, then execute a SPARQL query on the entity aggregation result, and reorder the entities according to the entity aggregation result after the SPARQL query to determine the corresponding reordered entity sorting result. Step S603: The entity re-ranking includes calculating a comprehensive score between each entity and the query statement based on the triplet matching degree, text semantic similarity, and image feature similarity of each entity in the entity aggregation result; and sorting the entities in the entity aggregation result in descending order according to the comprehensive score to generate a re-ranked entity ranking result. Optionally, in this embodiment, the user's query statement is first dynamically determined. If the current query type is a normal query (such as "find an entity"), entity reordering is performed directly based on the entity aggregation result of step S502 above, and suggestion words are generated. If the current query type is a complex query (such as filtering, statistics, calculation), a SPARQL query is generated, and entity reordering is performed based on the SPARQL query result, and suggestion words are generated.

[0116] To address the issue of large language models generating inaccurate or redundant answers in complex queries due to overly complex and unclear input information, this step introduces a deep processing flow that incorporates semantic understanding and structured reasoning. This transforms natural language queries into structured query commands (SPARQL) that the knowledge graph can understand and execute, and then re-ranks the initially retrieved entities based on this.

[0117] Specifically, the system automatically constructs SPARQL query statements based on the information obtained in the "multimodal parsing" and "entity aggregation" steps. The generated SPARQL query statements are sent to the RDF knowledge graph database for execution. The knowledge graph engine efficiently traverses the graph structure, performing pattern matching, filtering, joining, and sorting according to the query logic, ultimately returning a precise list of entities that meet all constraints and their related data.

[0118] Next, the entity re-ranking mechanism. Based on a multi-dimensional and quantifiable comprehensive evaluation system, each entity in the aggregation result is accurately scored.

[0119] Specifically, a comprehensive score is calculated for each entity in the entity aggregation result.

[0120] The overall score is a weighted fusion of three core indicators: First, triple matching degree, which measures the degree to which the triples (attribute-value pairs) contained in the entity match the query intent. For example, for the query "electric vehicle with the longest range", the matching degree of each "range" value in the entity with the query is calculated. Second, textual semantic similarity, which assesses the semantic relevance of the entity by comparing the vector representations of the entity-related text (such as descriptive paragraphs) and the query text. Third, image feature similarity, which calculates similarity through visual feature vectors if the query contains images or entity-related images (e.g., if the query image is "red sports car", then the entity that is red and is a sports car model will receive a higher score). These three dimensions cover the multimodal information of the entity from the perspectives of structured attributes, unstructured text, and visual content, respectively. The system combines the three into a single overall score through preset or dynamically adjusted weight coefficients (e.g., triple weight 0.5, text weight 0.3, image weight 0.2), comprehensively quantifying the multimodal relevance between the entity and the query.

[0121] Entities are reordered in descending order based on the comprehensive score. The highest-scoring entity indicates a high degree of match with the user query across different modalities and is therefore placed at the top of the results list. This multimodal fusion ranking mechanism effectively avoids single-modal bias (such as text matching but irrelevant images), ensuring that the returned entity results not only meet the literal and semantic requirements of the query but also maintain consistency in visual and structural attributes, thus improving the accuracy of entity ranking results and user experience.

[0122] Through step S602, this embodiment successfully reorders entities by constructing entity evaluation indicators, obtaining entity ranking results, which lays the foundation for outputting accurate prompt words in the future.

[0123] In one embodiment of the multimodal fusion entity retrieval enhancement generation method of this application, it may further include the following: Step S701: Based on the reordered entity sorting results, select a predetermined number of entities in descending order of entity priority; Step S702: For each selected entity, based on its attribute priority and relation priority, filter out the high-priority attributes and relations from the set of triples corresponding to that entity to determine the corresponding entity triple subset; Step S703: Combine the query statement, the entity triplet subset, and the text block data corresponding to the entity to determine the prompt words required for the corresponding final answer.

[0124] Optionally, this step is essentially a pruning process during the generation of prompt words.

[0125] Specifically, after obtaining the entity reordering results, prompt words are constructed based on the reordering results to generate the final query answer. The construction of prompt words requires accuracy and conciseness to reduce the burden on the model to generate answers and reduce resource waste. Therefore, the crucial optimization step is dynamic pruning.

[0126] In this step, quality filtering and focus aggregation are performed first. Based on the comprehensive score obtained after entity reordering, all candidate entities are prioritized from high to low. Subsequently, the system does not include all entities in the prompts, but selects a predetermined number of high-priority entities according to a preset strategy (such as the top N entities or all entities with scores exceeding a threshold). This ensures that subsequent processing is only performed on the most relevant and important entities, effectively avoiding interference from low-quality or irrelevant entities to the model.

[0127] Next, deep refinement and noise reduction are performed. For each high-priority entity selected in the previous step, its numerous associated attributes and relationships are further simplified. Based on predefined or dynamically calculated attribute priorities (e.g., "sales revenue" has high priority for statistical queries) and relationship priorities (e.g., the "subsidiary" relationship has high priority for enterprise queries), the system selects the most critical and relevant subset from the complete set of triples corresponding to that entity. This ensures that every attribute or relationship in the final input model highly contributes to answering user queries, significantly improving the information efficiency of subsequent generation processes.

[0128] Finally, the system structurally combines the user's original query, the entity triplet subset obtained after deep refinement, and the original text evidence blocks corresponding to these entities (used to provide contextual background) into the final large model prompt words.

[0129] Through step S703, this embodiment successfully provides the large language model with all the core elements necessary to answer questions: clear task instructions (user questions), refined structured knowledge (entity triples), and supporting textual evidence, directly guiding it to generate high-quality textual answers that are relevant to the question, factually accurate, and traceable.

[0130] To improve the efficiency and accuracy of multimodal retrieval, this application provides an embodiment of a multimodal fusion retrieval enhancement generation apparatus for implementing all or part of the aforementioned multimodal fusion entity retrieval enhancement generation method. See [link to embodiment]. Figure 2 The multimodal fusion-based retrieval enhancement generation device specifically includes the following components: Offline data processing module 10 is used in the offline data processing stage to perform block processing on preset multimodal data, determine the block data of each modality, extract text from the block data based on the characteristics of each modality, determine the corresponding text block data, perform entity recognition and relation extraction on the text block data according to a preset language model, determine the corresponding entity triples, fuse the entity triples with the text block data, determine the offline knowledge graph and construct the corresponding inverted index and vector index; The online data processing module 20 is used in the online data processing stage to receive multimodal query statements input by users, vectorize the multimodal query statements, determine the corresponding query vector, perform multi-way hybrid retrieval on the query vector based on the inverted index and the vector index, determine the corresponding preliminary retrieval results, the preliminary retrieval results include all entity triples, text block data and retrieval scores related to the query vector, perform entity aggregation on the preliminary retrieval results, and reorder the text block data after entity aggregation according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results, the entity aggregation results include entities and text block data; The complex query result determination module 30 is used to reorder entities based on the entity aggregation result, construct prompt words based on the reordered entity ranking result, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than a threshold as the final search result.

[0131] As described above, the multimodal fusion retrieval enhancement generation device provided in this application embodiment can, in the offline stage, perform block segmentation and adaptive text extraction on multimodal data to obtain text block data corresponding to each modality. It then extracts entities and relations from the text block data based on a language model, constructs entity triples, and fuses these entity triples with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, it receives user query statements, retrieves the most similar text block data to the query statement in the knowledge graph through the data index, performs entity aggregation on the retrieved text block data, and re-sorts the text block data according to the retrieval score to obtain entity aggregation results. If the query type is a complex query, it continues to sort the entity aggregation results, constructs prompt words, and inputs them into the language model to obtain the text answer. This improves the efficiency and accuracy of multimodal retrieval.

[0132] From a hardware perspective, in order to improve the efficiency and accuracy of multimodal retrieval, this application provides an embodiment of an electronic device for implementing all or part of the entity retrieval enhancement generation method of multimodal fusion, wherein the electronic device specifically includes the following: The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the multimodal fusion entity retrieval enhancement generation method and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the multimodal fusion entity retrieval enhancement generation method in the present embodiment, and the contents of the embodiments of the multimodal fusion entity retrieval enhancement generation method are incorporated herein, and repeated details will not be described again.

[0133] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0134] In practical applications, parts of the multimodal fusion entity retrieval enhancement generation method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0135] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0136] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0137] In one embodiment, the multimodal fusion entity retrieval enhancement generation method functionality can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: Step S101: In the offline data processing stage, the preset multimodal data is divided into blocks to determine the block data of each modality. Based on the characteristics of each modality, text extraction is performed on the block data to determine the corresponding text block data. Entity recognition and relation extraction are performed on the text block data according to the preset language model to determine the corresponding entity triples. The entity triples are fused with the text block data to determine the offline knowledge graph and construct the corresponding inverted index and vector index. Step S102: In the online data processing stage, a multimodal query statement input by the user is received, the multimodal query statement is vectorized, the corresponding query vector is determined, and a multi-way hybrid retrieval is performed on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results. The preliminary retrieval results include all entity triples, text block data, and retrieval scores related to the query vector. Entity aggregation is performed on the preliminary retrieval results, and the text block data after entity aggregation is reordered according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results. The entity aggregation results include entities and text block data. Step S103: Reorder entities based on the entity aggregation result, construct prompt words based on the reordered entity ranking result, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than the threshold as the final search result.

[0138] As described above, the electronic device provided in this application embodiment, in the offline stage, performs block segmentation and adaptive text extraction on multimodal data to obtain text block data corresponding to each modality. Entities and relations of the text block data are extracted according to the language model to construct entity triples. The entity triples are then fused with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, the device receives user query statements, retrieves the most similar text block data to the query statement in the knowledge graph through the data index, performs entity aggregation on the retrieved text block data, and re-sorts the text block data according to the retrieval score to obtain entity aggregation results. If the query type is a complex query, the entity aggregation results are further sorted, and prompt words are constructed and input into the language model to obtain the text answer. This improves the efficiency and accuracy of multimodal retrieval.

[0139] In another embodiment, the multimodal fusion entity retrieval enhancement generation method can be configured separately from the central processing unit 9100. For example, the multimodal fusion entity retrieval enhancement generation method can be configured as a chip connected to the central processing unit 9100, and the function of the multimodal fusion entity retrieval enhancement generation method can be realized through the control of the central processing unit.

[0140] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0141] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0142] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0143] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0144] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0145] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0146] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0147] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is also coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0148] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multimodal fusion entity retrieval enhancement generation method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multimodal fusion entity retrieval enhancement generation method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: Step S101: In the offline data processing stage, the preset multimodal data is divided into blocks to determine the block data of each modality. Based on the characteristics of each modality, text extraction is performed on the block data to determine the corresponding text block data. Entity recognition and relation extraction are performed on the text block data according to the preset language model to determine the corresponding entity triples. The entity triples are fused with the text block data to determine the offline knowledge graph and construct the corresponding inverted index and vector index. Step S102: In the online data processing stage, a multimodal query statement input by the user is received, the multimodal query statement is vectorized, the corresponding query vector is determined, and a multi-way hybrid retrieval is performed on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results. The preliminary retrieval results include all entity triples, text block data, and retrieval scores related to the query vector. Entity aggregation is performed on the preliminary retrieval results, and the text block data after entity aggregation is reordered according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results. The entity aggregation results include entities and text block data. Step S103: Reorder entities based on the entity aggregation result, construct prompt words based on the reordered entity ranking result, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than the threshold as the final search result.

[0149] As described above, the computer-readable storage medium provided in this application embodiment, in the offline stage, performs block segmentation and adaptive text extraction on multimodal data to obtain text block data corresponding to each modality. Entities and relations of the text block data are extracted according to the language model to construct entity triples. The entity triples are then fused with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, the system receives user query statements, retrieves the most similar text block data to the query statement in the knowledge graph through the data index, performs entity aggregation on the retrieved text block data, and re-sorts the text block data according to the retrieval score to obtain entity aggregation results. If the query type is a complex query, the entity aggregation results are further sorted, prompt words are constructed, and input into the language model to obtain the text answer. This improves the efficiency and accuracy of multimodal retrieval.

[0150] Embodiments of this application also provide a computer program product capable of implementing all steps of the multimodal fusion entity retrieval enhancement generation method with the execution subject being a server or client in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the multimodal fusion entity retrieval enhancement generation method. For example, the computer program / instruction implements the following steps: Step S101: In the offline data processing stage, the preset multimodal data is divided into blocks to determine the block data of each modality. Based on the characteristics of each modality, text extraction is performed on the block data to determine the corresponding text block data. Entity recognition and relation extraction are performed on the text block data according to the preset language model to determine the corresponding entity triples. The entity triples are fused with the text block data to determine the offline knowledge graph and construct the corresponding inverted index and vector index. Step S102: In the online data processing stage, a multimodal query statement input by the user is received, the multimodal query statement is vectorized, the corresponding query vector is determined, and a multi-way hybrid retrieval is performed on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results. The preliminary retrieval results include all entity triples, text block data, and retrieval scores related to the query vector. Entity aggregation is performed on the preliminary retrieval results, and the text block data after entity aggregation is reordered according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results. The entity aggregation results include entities and text block data. Step S103: Reorder entities based on the entity aggregation result, construct prompt words based on the reordered entity ranking result, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than the threshold as the final search result.

[0151] As described above, the computer program product provided in this application embodiment, in the offline stage, performs block segmentation and adaptive text extraction on multimodal data to obtain text block data corresponding to each modality. Entities and relations of the text block data are extracted according to the language model, entity triples are constructed, and the entity triples are fused with the text block data to obtain an offline knowledge graph and construct a data index. In the current stage, the system receives user query statements, retrieves the most similar text block data to the query statement in the knowledge graph through the data index, performs entity aggregation on the retrieved text block data, and re-sorts the text block data according to the retrieval score to obtain entity aggregation results. If the query type of the query statement is a complex query, the entity aggregation results are further sorted, and prompt words are constructed and input into the language model to obtain the text answer. This improves the efficiency and accuracy of multimodal retrieval.

[0152] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0154] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0156] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multimodal fusion-based entity retrieval enhancement generation method, characterized in that, The method includes: In the offline data processing stage, the preset multimodal data is divided into blocks to determine the block data of each modality. Based on the characteristics of each modality, text extraction is performed on the block data to determine the corresponding text block data. Entity recognition and relation extraction are performed on the text block data according to the preset language model to determine the corresponding entity triples. The entity triples are fused with the text block data to determine the offline knowledge graph and construct the corresponding inverted index and vector index. In the online data processing stage, a multimodal query statement input by the user is received, the multimodal query statement is vectorized, the corresponding query vector is determined, and a multi-way hybrid retrieval is performed on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results. The preliminary retrieval results include all entity triples, text block data, and retrieval scores related to the query vector. Entity aggregation is performed on the preliminary retrieval results, and the text block data after entity aggregation is reordered according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results. The entity aggregation results include entities and text block data. The entities are reordered based on the entity aggregation results. Prompt words are constructed based on the reordered entity ranking results. The prompt words are input into a preset language model to obtain the text answer generated by the language model. The text answer is verified based on the offline knowledge graph. The text answer with a confidence level higher than the threshold is output as the final search result.

2. The multimodal fusion entity retrieval enhancement generation method according to claim 1, characterized in that, The step of dividing the preset multimodal data into blocks to determine the blocks of data for each modality includes: For text modal data, a segmentation strategy based on semantics or fixed-length tokens is used to divide the data and determine the corresponding text segments. For image modal data or table modal data, each image or table entity is independently divided into a block, and the corresponding image block data or table block data is determined respectively. For audio modal data, segmentation is performed according to preset time windows or speech segments to determine the corresponding audio block data; For video modal data, segmentation is performed according to key frame sequences or scene change nodes to determine the corresponding video block data; For sensor modal data or structured data modal data, divide the data into blocks according to data acquisition points, time series or structured fields, and determine the corresponding sensor block data or structure block data respectively.

3. The multimodal fusion entity retrieval enhancement generation method according to claim 2, characterized in that, The step of extracting text from the segmented data based on each modal characteristic to determine the corresponding text block data includes: Text information is extracted from the image block data and the table block data using optical character recognition technology; The audio segment data is translated into text information using automatic speech recognition technology; Text information is extracted from the video block data using video frame extraction, keyframe optical character recognition, or video content analysis techniques. Text information is extracted from the sensor block data and the structure block data through key field parsing, data transformation, or structured query methods. All extracted text information is stored in a unified format according to the text block data, and the corresponding text block data is determined.

4. The multimodal fusion entity retrieval enhancement generation method according to claim 1, characterized in that, The step of performing a multi-path hybrid retrieval on the query vector based on the inverted index and the vector index to determine the corresponding preliminary retrieval results includes: Based on the query vector in the offline knowledge graph, full-text retrieval of text block data is performed according to the inverted index to determine the corresponding full-text retrieval score, and semantic similarity retrieval of text block data is performed according to the vector index to determine the corresponding semantic similarity retrieval score. After normalizing the full-text search score and the semantic similarity search score, they are weighted and fused according to preset weights to obtain the corresponding preliminary search results. The preliminary search results include relevant text block data, the search score corresponding to the text block data, and relevant entity triples. The relevant entity triples include entity information, attribute information, and relation information from the text block data.

5. The multimodal fusion entity retrieval enhancement generation method according to claim 1, characterized in that, The preliminary search results are aggregated into entities, and the aggregated text block data is reordered based on the search score and the text block data score in the offline knowledge graph to determine the corresponding entity aggregation result. The entity aggregation result includes entities and text block data, including: The text block data, attribute information and relation information pointing to the same entity in the preliminary search results are merged and integrated to determine the aggregation unit with the entity as the core. The retrieval scores of all text block data in the aggregation unit and the data scores of all entities associated with the text block data in the offline knowledge graph are weighted to determine the corresponding text block data re-ranking score. Based on the re-ranking score, the text block data in the aggregation unit is re-ranked to determine the corresponding entity aggregation result, which includes entities and text block data.

6. The multimodal fusion entity retrieval enhancement generation method according to claim 1, characterized in that, The entity reordering based on the entity aggregation result includes: If the query type of the query statement is a normal query, then the entities are reordered according to the entity aggregation results to determine the corresponding reordered entity sorting result; If the query type of the query statement is a complex query, then a SPARQL query is executed on the entity aggregation result, and the entities are reordered according to the entity aggregation result after the SPARQL query to determine the corresponding reordered entity sorting result. The entity reordering process includes calculating a comprehensive score for each entity and the query statement based on the triplet matching degree, text semantic similarity, and image feature similarity of each entity in the entity aggregation result, and sorting the entities in the entity aggregation result in descending order according to the comprehensive score to generate a reordered entity ranking result.

7. The multimodal fusion entity retrieval enhancement generation method according to claim 1, characterized in that, The step of constructing prompt words based on the reordered entity sorting results includes: Based on the reordered entity sorting results, a predetermined number of entities are selected in descending order of entity priority. For each selected entity, based on its attribute priority and relation priority, high-priority attributes and relations are selected from the set of triples corresponding to that entity to determine the corresponding entity triple subset; The query statement, the entity triplet subset, and the text block data corresponding to the entity are combined to determine the prompt words required for the final answer.

8. A multimodal fusion-based retrieval enhancement generation device, characterized in that, The device includes: The offline data processing module is used to segment the preset multimodal data, determine the segmented data of each modality, extract text from the segmented data based on the characteristics of each modality, determine the corresponding text block data, perform entity recognition and relation extraction on the text block data according to the preset language model, determine the corresponding entity triples, fuse the entity triples with the text block data, determine the offline knowledge graph, and construct the corresponding inverted index and vector index. An online data processing module is used to receive multimodal query statements input by users, vectorize the multimodal query statements, determine the corresponding query vector, perform multi-way hybrid retrieval on the query vector based on the inverted index and the vector index, determine the corresponding preliminary retrieval results, the preliminary retrieval results include all entity triples, text block data and retrieval scores related to the query vector, perform entity aggregation on the preliminary retrieval results, and reorder the text block data after entity aggregation according to the retrieval scores and the text block data scores in the offline knowledge graph to determine the corresponding entity aggregation results, the entity aggregation results include entities and text block data; The complex query result determination module is used to reorder entities based on the entity aggregation results, construct prompt words based on the reordered entity ranking results, input the prompt words into a preset language model to obtain the text answer generated by the language model, verify the text answer based on the offline knowledge graph, and output the text answer with a confidence level higher than a threshold as the final search result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal fusion entity retrieval enhancement generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal fusion entity retrieval enhancement generation method according to any one of claims 1 to 7.

Images