Multi-modal map enhanced retrieval method and dialogue system based on feature fusion optimization

Abstract

The invention discloses a feature fusion optimization-based multi-modal map enhancement retrieval method and a dialogue system. The method comprises the following steps of: respectively carrying out pre-training and fine tuning on a visual model and a language model by utilizing a domain image and text data; constructing a knowledge graph based on the text data in the knowledge base and constructing a vector database containing associated image data; performing semantic analysis and optimization on the original query of the user by using the language model and forming a structured retrieval intention; searching related sub-graphs, text semantic vector information and associated image data based on the search intention; encoding the sub-images into knowledge contexts, inputting the knowledge contexts into a dynamic prompt generator to generate visual prompts, and extracting enhanced visual features from the associated image data through a visual model; and inputting the subgraph, the text semantic vector information and the enhanced visual features into a language model for collaborative reasoning, and generating and outputting a final answer. According to the method, deep fusion and accurate retrieval of multi-modal knowledge can be realized, and the accuracy and efficiency are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a multimodal graph enhancement retrieval method and dialogue system based on feature fusion optimization. Background Technology

[0002] With the rapid development of information technology, knowledge bases in various fields (such as healthcare, finance, education, and cultural heritage) have undergone tremendous changes. Early knowledge bases primarily relied on text-based entry-style storage, offering limited functionality and only meeting basic knowledge retrieval needs. However, with the widespread adoption of advanced technologies such as high-precision scanning, 3D modeling, and image acquisition, knowledge bases in various fields have undergone significant upgrades. They have evolved from simple text storage to complex collections integrating multiple data types, including text, images, 3D models, and audio, exhibiting typical multimodal data characteristics. This evolution in data format has provided rich material for in-depth knowledge mining in various fields, while also posing new challenges to retrieval technologies, including cross-modal association, precise positioning, and semantic understanding.

[0003] Faced with this challenge, while current mainstream single-modal retrieval technologies can achieve efficient retrieval within their respective modalities, they fall short when processing multimodal knowledge bases. Their technical bottleneck lies in the modal barriers existing between different modalities of data. These barriers stem from fundamental differences in the representation, feature dimensions, and semantic expression logic of different modalities: text data uses character sequences to convey semantic information, image data presents visual features based on pixel matrices, and audio data carries auditory information with waveform signals at its core. These differences prevent the system from directly establishing deep semantic connections between different modalities, making it difficult to accurately meet users' complex retrieval needs in real-world retrieval scenarios.

[0004] To compensate for the lack of semantic association, knowledge graph technology, as a structured knowledge representation and association query tool, alleviates the problem of missing semantic association in traditional retrieval to some extent. By systematically modeling core entities in specific domains (such as diseases, drugs, and genes in the medical field; institutions, products, and customers in the financial field), entity attributes (such as drug indications and product yields), and relationships between entities (such as the correspondence between diseases and symptoms, and the lending relationship between institutions and customers), knowledge graphs can construct logically clear knowledge networks, enabling accurate queries and reasoning based on entity associations. However, the construction and reasoning of traditional knowledge graphs still revolve around textual semantics, and their ability to model non-textual modal information such as images, audio, and 3D models is severely insufficient. There is a lack of effective technical means to transform the semantic information in these non-textual data into structured knowledge that can be recognized and associated by the knowledge graph.

[0005] Furthermore, the terminology systems in specific fields often possess extremely strong professionalism and relatively high ambiguity, which become important obstacles for retrieval technologies to accurately understand requirements. In various professional fields, a large number of concepts have the characteristic of "similar in form but different in meaning". Their surface expressions are similar, but their core connotations and applicable scenarios are significantly different, and precise differentiation must be carried out to ensure the effectiveness of retrieval results. Taking the field of cultural heritage as an example, when a user mentions the term "ding", it may refer to different shapes such as square ding, round ding, and li-style ding. These shape differences are not only reflected in the appearance structure but also contain different technological characteristics and cultural connotations of different eras. Therefore, if traditional retrieval systems only rely on text labels and cannot understand and associate specific visual morphological features, it will inevitably lead to generalization and deviation of retrieval results. If the retrieval system cannot accurately identify such detailed differences, it will result in the generalization of retrieval results.

[0006] Therefore, the existing technologies have core bottlenecks such as difficulty in breaking the modality barrier, insufficient representation of multi-modal information in knowledge graphs, and lack of fine-grained semantics of professional terms. There is an urgent need for an innovative retrieval method that can deeply integrate multi-source heterogeneous features, enhance the multi-modal representation ability of knowledge graphs, and achieve accurate cross-modal semantic alignment and understanding. Summary of the Invention

[0007] In view of the above, the purpose of the present invention is to provide a multi-modal graph enhanced retrieval method and dialogue system based on feature fusion optimization, which realizes accurate cross-modal association by deeply integrating visual features and knowledge graph semantics, and improves the semantic understanding ability of professional terms and fuzzy queries through dynamic prompting and interactive optimization. Finally, an intelligent framework capable of accurate retrieval and multi-modal reasoning is constructed, significantly improving the accuracy and efficiency of domain knowledge retrieval and intelligent question answering.

[0008] To achieve the above invention purpose, the technical solutions provided by the present invention are as follows: In the first aspect, a multi-modal graph enhanced retrieval method based on feature fusion optimization provided by an embodiment of the present invention includes the following steps: Pre-train and fine-tune the visual model and language model respectively using domain image and text data; After segmenting the text data in the knowledge base, use the fine-tuned language model to extract entities and relationships to construct a knowledge graph. At the same time, associate and index the image data in the knowledge base with the corresponding entities, and store them together with the semantic vectors of entities and relationships in a vector database; Receive the user's original query, perform semantic parsing and active optimization processing on it using the fine-tuned language model, and further extract the target entity and relationship to form a structured retrieval intention; Based on the search intent, graph queries and semantic searches are performed in the knowledge graph and vector database respectively to obtain relevant subgraphs, text semantic vector information and associated image data. Subgraphs are encoded into knowledge context inputs to a dynamic cue generator to produce visual cues. Enhanced visual features are then extracted from the visual cues and associated image data using a pre-trained visual model. The subgraph, text semantic vector information, and enhanced visual features are combined to form a multimodal context, which is then input into a fine-tuned language model for collaborative reasoning to generate the final answer and output it.

[0009] Preferably, the pre-training and fine-tuning of the visual model and language model using domain image and text data respectively includes: A mask autoencoder framework is used to pre-train a vision model based on the Vision Transformer architecture. By randomly masking a high proportion of image patches in the input image, and by utilizing the encoder and lightweight decoder of the vision model to work together, the pixel values ​​of the masked regions are reconstructed, enabling the model to have the ability to learn the structured representation of the image from the visible context and have a deep understanding of visual features. We use an instruction dataset built from domain-specific text data to perform supervised fine-tuning on a general-purpose large language model. The fine-tuning objectives include entity recognition and relation understanding, enabling the model to have a deep understanding of domain terminology, knowledge structure, and semantic logic.

[0010] Preferably, the step of segmenting and processing the text data in the knowledge base and then using a fine-tuned language model to extract entities and relations to construct a knowledge graph includes: The text data in the knowledge base is processed by multi-level segmentation, including first splitting it into preliminary segments by paragraph, and then using a sliding window with overlapping areas to split the ultra-long paragraphs in the preliminary segments into fine-grained segments, resulting in several text fragments. Based on the preset entity relation extraction prompt word template, the structured information of the text fragments is extracted using a fine-tuned language model. The extraction results are then subjected to multiple rounds of data optimization, including cleaning, merging, and deduplication, to form a standardized entity and relation dataset and construct a knowledge graph.

[0011] Preferably, the step of associating image data in the knowledge base with corresponding entities includes: In the knowledge graph, the storage path of the image file or a unique identifier is added as an attribute to the entity node of the image type, and the semantic vector of the entity node is associated with the image identifier in the vector database to establish a direct mapping relationship from entity semantics to image.

[0012] Preferably, the step of receiving the user's original query, performing semantic parsing and proactive optimization on it using a fine-tuned language model, and further extracting target entities and relationships to form a structured search intent includes: The fine-tuned language model is used to parse the user's original query and identify key information and potential missing points. Based on entity relationships and common domain interaction scenarios in knowledge graphs, key information and potential missing points are supplemented as prompt words, and several candidate completion directions or explicit statements are generated in combination with prompt words. These candidate results are fed back to the user for confirmation in order to optimize and obtain a semantically clear query expression. Finally, the target entities and relationships are accurately extracted from them to form the structured search intent.

[0013] Preferably, the dynamic cue generator is a lightweight multilayer perceptron, whose input is an encoded subgraph knowledge context vector and whose output is a set of learnable visual cue vectors. When this set of cue vectors is input into the visual model, it is concatenated with the image patch embedding vectors, and the knowledge context-driven image region focusing is achieved by guiding the attention mechanism inside the model.

[0014] Preferably, the dynamic cue generator introduces orthogonality constraints when generating visual cue vectors, so that different cue vectors focus on different semantic regions in the image, ensuring the diversity of cue and comprehensively covering key visual information.

[0015] Secondly, embodiments of the present invention also provide a multimodal graph-enhanced retrieval dialogue system based on feature fusion optimization, which is implemented using the aforementioned multimodal graph-enhanced retrieval method based on feature fusion optimization, including: a model domainization module, a knowledge base construction module, a query understanding module, an association retrieval module, a visual enhancement module, and a multimodal reasoning module; The model domainization module is used to pre-train and fine-tune the visual model and the language model using domain image and text data, respectively. The knowledge base construction module is used to segment and process the text data in the knowledge base, and then use the fine-tuned language model to extract entities and relations to construct a knowledge graph. At the same time, the image data in the knowledge base is associated with the corresponding entities and indexed, and then stored together with the entity semantic vector and the relation semantic vector in the vector database. The query understanding module is used to receive the user's original query, perform semantic parsing and proactive optimization on it using a fine-tuned language model, and further extract the target entities and relationships to form a structured search intent. The association retrieval module is used to perform graph query and semantic retrieval in the knowledge graph and vector database respectively based on the retrieval intent, and obtain relevant subgraphs, text semantic vector information and associated image data; The visual enhancement module is used to encode the subgraph into a knowledge context input dynamic prompt generator to generate visual prompts, and extract enhanced visual features based on the visual prompts and associated image data through a pre-trained visual model. The multimodal reasoning module is used to combine subgraphs, text semantic vector information, and enhanced visual features to form a multimodal context, which is then input into a fine-tuned language model for collaborative reasoning to generate the final answer and output it.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described multimodal map enhancement retrieval method based on feature fusion optimization when executing the computer program.

[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described multimodal graph enhancement retrieval method based on feature fusion optimization.

[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention constructs a multimodal knowledge graph that integrates text and images, and innovatively introduces a collaborative mechanism based on a dynamic prompt generator, a visual model, and a general language model. This enables the deep integration of deep semantic information of objects with fine-grained visual features, effectively solving the problem of insufficient modeling ability of traditional knowledge graphs for non-textual information. It also breaks down the modal barriers in traditional retrieval and establishes a cross-modal semantic bridge.

[0019] (2) This invention adopts a visual pre-training method based on a mask autoencoder framework and a language model fine-tuning method combined with prompt word templates to form a complete domain adaptive technology solution. Domain knowledge modeling and entity relationship extraction can be efficiently completed through small sample learning, which significantly improves the accuracy of knowledge extraction and system scalability in scenarios lacking large-scale labeled data, and effectively solves the problem of scarce data in professional fields.

[0020] (3) In the retrieval process, the present invention improves the accuracy of retrieval by using multiple rounds of dialogue and completion prompts to improve the question, as well as by using different retrieval modes and retrieval algorithms. At the same time, a dynamic prompt generator is used to generate visual prompts to guide the model to focus on the most relevant area of ​​the query in the image, thereby extracting fine-grained visual features for knowledge enhancement. Finally, the entire process from query understanding to multimodal reasoning is optimized, which significantly improves the retrieval accuracy and user experience. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a multimodal map enhancement retrieval method based on feature fusion optimization provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a sliding window for segmenting long paragraphs provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating how an Embedding model maps text to a unified semantic space, as provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of a process for querying and retrieving relevant fragments provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a multimodal graph-enhanced retrieval dialogue system based on feature fusion optimization provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0024] The inventive concept of this invention is as follows: In view of the problems of existing technologies, such as the difficulty in breaking down modal barriers, insufficient representation of multimodal information by knowledge graphs, and lack of fine-grained semantics of professional terms, the embodiments of this invention provide a multimodal graph-enhanced retrieval method and dialogue system based on feature fusion optimization. Through a complete technology chain of domain adaptive training, multimodal knowledge base construction, query intent parsing optimization, associative retrieval, and knowledge-guided visual feature enhancement, deep fusion of text, image, and graph information is achieved, which significantly improves the retrieval accuracy and semantic understanding ability in complex professional scenarios.

[0025] like Figure 1 As shown in the embodiment, a multimodal graph enhancement retrieval method based on feature fusion optimization is provided, including the following steps: S1 utilizes domain image and text data to pre-train and fine-tune the visual model and language model, respectively.

[0026] S1.1, Visual model pre-training.

[0027] In this embodiment, taking the cultural heritage field as an example, the training data used consists of images of over 2,000 artifacts already displayed in museums. Each artifact has photos taken from different angles, requiring image filtering and cleaning to generate the final training set. Using this domain-specific data, a visual model based on the VisionTransformer (ViT) architecture is pre-trained using the Masked Autoencoder (MAE) framework, enabling ViT to deeply understand domain-specific visual patterns. For images, the MAE method first converts them into a series of non-overlapping Patch Embedding sequences. , where vector It is the first It is derived from a linear transformation of a patch. The total number of vectors in the sequence, then in a high proportion Randomly mask patches and replace them with learnable mask tokens. The ViT encoder only processes visible patch embeddings and outputs features. The expression is as follows: , in, It corresponds to the location encoding of the visible patch. , For indexed sets, This is the set of all learnable parameters in the ViT encoder (including the weights, biases, etc. of the MSA (Multi-Head Self-Attention) layer and the FFN (Feed-Forward Network) layer). The ViT decoder is a lightweight Transformer whose input consists of two parts: the features output by the encoder. And a mask token added for all masked patch locations, by encoding these two parts with the complete location. The concatenation and decoder outputs the reconstructed patch features. Finally, a linear layer projects these features back into the pixel space to obtain the predicted pixel values ​​of the masked patch. .

[0028] Pre-training targets visual features in the deep learning domain and employs the following loss function. : , in, The total number of image patches that are being masked. The square of the L2 norm. Normalization function is used to normalize pixel data in order to stabilize training and better handle pixel values. Represents the first in the original image The pixel values ​​of each patch. Indicates the decoder's predicted first The pixel values ​​of each patch.

[0029] S1.2, Language Model Fine-tuning.

[0030] In this embodiment, the language model adopts the general model Qwen3-NEXT-14B open source from Alibaba Qianwen. The text data is cleaned, entity names are marked using entity annotation tools (such as BIO format), and corresponding instruction datasets are constructed. The model is fine-tuned with the goal of recognizing the corresponding objects, thereby enhancing the general language model's understanding of specific knowledge domains.

[0031] Fine-tuning targets specific object recognition capabilities and employs the cross-entropy loss function. : , in, The total number of tokens. For the label category (Y - is a certain type of object, N - is not a certain type of object). For true labels (one-hot encoded) To predict the current number of times the model is used. Each token is categorized The probability of.

[0032] S2 processes the text data in the knowledge base into segments and then uses a fine-tuned language model to extract entities and relations to construct a knowledge graph. At the same time, the image data in the knowledge base is associated with the corresponding entities and indexed, and then stored together with the entity semantic vectors and relation semantic vectors in the vector database.

[0033] In this embodiment, the finely tuned language model and the designed prompt word template are used to segment the existing text data in the knowledge base and then extract entities and relations, as well as establish a graph and entity-relation vector library.

[0034] First, the textual materials in the knowledge base are split into paragraphs, and then excessively long paragraphs are processed using... Figure 2 The sliding window shown is subdivided into different text blocks. The specific steps include: using line breaks in the text ( ), consecutive spaces or typesetting marks (such as section symbols like "I.", "(1)", etc.) to judge paragraph separation, conduct preliminary paragraph segmentation, and then remove pure spaces, punctuation marks or paragraphs with no actual content. Then, according to the model's capabilities (the maximum input length of the retrieval model) and text density, define the "extra-long" standard (single paragraph > 800 tokens), set the size of the corresponding sliding window (300 tokens) and step size (200 tokens), so that the overlapping part is 100 tokens, ensuring that cross-paragraph semantics are not lost, and successfully splitting the longer segments into corresponding small segments.

[0035] Based on the previously divided text segments, combined with the entity relationship extraction prompt template designed specifically, input the segment content item by item into the general model for structured information extraction. This step needs to ensure that the template clearly defines the entity types (such as cultural relic names, dynasties to which they belong, excavation locations, core materials, pattern styles, etc.) and relationship types (such as "belong to", "contain", "adopt", etc.), guiding the general model to output results strictly in the format of "entity pair + relationship type", reducing ambiguity and format deviation.

[0036] After obtaining the preliminary extraction results, the quality needs to be optimized through multiple rounds of data processing: in the cleaning link, correct the abbreviations and typos of entity names (such as standardize "Tri-color Glazed Pottery Horse" to avoid variants such as "Tang Horse" and "Tri-color Horse"), and complete the missing entity information in the relationship expression; in the merging link, integrate different associated information of the same entity to form a complete entry; in the deduplication link, remove duplicate entity pairs and relationship combinations, and finally obtain a standardized and consistent entity and relationship dataset.

[0037] Subsequently, use Cypher statements to batch write this structured data into the graph database. By defining the attributes of nodes (entities) (text information related to the entity) and the types of edges (relationships), construct a knowledge graph with clear associations between entities, intuitively presenting the semantic network of a specific domain. To achieve semantic-level precise matching during retrieval (such as associating "Tang Dynasty colored pottery" with "Tri-color Glazed Pottery"), as Figure 3 shown in the process, introduce the Qwen3-Embedding-8B-GGUF model open-sourced by Alibaba Cloud Qianwen as the Embedding model, perform vector encoding on the entity names and relationship types in the graph database, and map them to a high-dimensional semantic vector space. The generated vectors will be synchronously stored in the vector database to establish a vector index library of entities and relationships, providing data support for subsequent semantic retrieval based on vector similarity, and enhancing the understanding ability of the retrieval system for fuzzy queries and synonymous expressions.

[0038] Meanwhile, in the knowledge graph, the storage path of the image file or a unique identifier is added as an attribute to the entity node of the image type, and the semantic vector of the entity node is associated with the image identifier in the vector database to establish a direct mapping relationship from entity semantics to image.

[0039] S3 receives the user's original query, performs semantic parsing and proactive optimization on it using a fine-tuned language model, and further extracts the target entities and relationships to form a structured search intent.

[0040] In actual interactions, user input often exhibits ambiguity (such as brief descriptions, incomplete information, or unclear intent), which directly affects the accuracy of subsequent retrieval and responses. Therefore, to address this issue, it is necessary to leverage the semantic understanding and generation capabilities of language models to proactively optimize user input.

[0041] In this embodiment, a finely tuned language model parses the user's original query to identify key information and potential missing points. Then, based on entity relationships in the knowledge graph and common domain interaction scenarios, corresponding prompts are generated. The speech model combines these prompts to generate several possible completion directions or explicit expressions. These candidate contents are then fed back to the user, guiding them to confirm or further supplement the information until clear and explicit input is obtained. Ultimately, the target entities and relationships are accurately extracted from the user's query, forming a structured search intent. This process lays the foundation for subsequent accurate matching based on knowledge graphs and vector retrieval, ultimately ensuring the relevance and accuracy of the answer.

[0042] S4, based on the search intent, performs graph query and semantic search in the knowledge graph and vector database respectively to obtain relevant subgraphs, text semantic vector information and associated image data.

[0043] In this embodiment, a retrieval algorithm is used to search and sort the established entity relationship vector library and graph to obtain relevant subgraphs, text semantic vector information and associated image data.

[0044] Search process as follows Figure 4 As shown, the first step after inputting the query is keyword extraction. This step separates two types of keywords from the query: high-level keywords (hl_keywords) and low-level keywords (ll_keywords). High-level keywords are mostly abstract concepts or relational descriptions, while low-level keywords are specific entity names.

[0045] Different search modes are initiated based on keyword type. If high-level keywords are extracted, a global relational search is launched; if low-level keywords are retrieved, a local entity search is triggered. In the entity search phase, the system first performs a vector similarity query using the entity vector library. Let the vector of the entity to be searched be... ( (where the dimension is vector), the entity vector library stores the set of entity vectors. Candidate entities are filtered by calculating cosine similarity: , Rank the similarity scores The entities are used as initial matching nodes. Then, the node association degree is calculated (…). The connection strength of an entity in a knowledge graph is defined as the strength of that connection, and the formula is: , in, The in-degree of a node (the number of relations pointing to this entity). This represents the average weight of the relationships (reflecting the importance of the relationships). This is a balancing coefficient. Based on the relevance threshold, other entities strongly associated with the target entity, their corresponding text fragments, and relational data are filtered out.

[0046] In the relation retrieval process, the relation vector library stores vector representations of relation types: For the relation description vector to be retrieved Similarly, cosine similarity is used to match the corresponding relation edges: , After matching the target relation, the connection strength of the relation is calculated. The relation is defined as the weighted sum of the frequency of its occurrence in the graph and the importance of the associated entities: , in, The frequency of the occurrence of the relationship. It is the sum of the degrees of association of the entities connected by the relation. To adjust parameters, highly relevant entity relationship pairs and their corresponding text and image information are selected based on connection strength. Finally, the top-N results are selected for the context construction stage, ensuring that the subsequently generated answers conform to both semantic relevance and the structural logic of the knowledge graph.

[0047] S5 encodes the subgraph into a knowledge context input dynamic cue generator to produce visual cues, and extracts enhanced visual features based on the visual cues and associated image data through a pre-trained visual model.

[0048] In this embodiment, the acquired subgraph information is encoded to obtain a global knowledge vector. , As a vector dimension, it accurately summarizes the domain knowledge context relevant to the query. This information is then fed into the dynamic prompt generator designed in this invention. . It is a lightweight multilayer perceptron (MLP) that can adapt to dynamically changing knowledge context. Real-time generation A unique 3D visual cue vector .

[0049] To ensure the effectiveness and diversity of these cue vectors, orthogonality constraints are introduced. : , Among them, superscript For transpose, It is the identity matrix. It is the square of the Frobenius norm.

[0050] Through orthogonality constraints This forces different cue vectors to remain independent, allowing each cue to contribute unique semantic guidance. (Generated) Then, the patch embedding of the original image is... And the learnable CLS tokens are concatenated to form the complete input sequence of the ViT encoder. Within the ViT encoder, the multi-head self-attention (MSA) mechanism is key to achieving focus. During MSA interaction, the image patch's query will deeply interact with the knowledge hint's key: , in, and These are linear projection functions used to transform an input vector into query and key vectors. It is a scaling factor, where This refers to the dimension of the Query / Key vector. Based on the above mechanism, ViT dynamically adjusts its attention weights to different regions of the image, prompting ViT to focus on and extract key visual region features that are highly relevant to the current knowledge context, thereby obtaining a problem-related visual image, which is then used as contextual input to the language model.

[0051] S6 combines subgraphs, text semantic vectors, and enhanced visual features to form a multimodal context, which is then input into a finely tuned language model for collaborative reasoning to generate the final answer and output it.

[0052] In this embodiment, the retrieved text semantic vectors, knowledge subgraph structure data, and image features enhanced by dynamic visual cues are uniformly encoded and input into a domain-fine-tuned language model to achieve deep collaboration and complementarity of cross-modal information, ultimately generating an accurate answer that conforms to domain knowledge and incorporates visual evidence.

[0053] In summary, the multimodal graph enhancement retrieval method based on feature fusion optimization provided by this invention innovatively achieves deep fusion of text, image, and graph information, effectively solving problems such as modal barriers, insufficient knowledge graph representation, and semantic loss of professional terms. It significantly improves retrieval accuracy and semantic understanding capabilities in complex professional scenarios and can be widely applied to various professional fields that require in-depth processing of multimodal knowledge, such as cultural heritage protection, medical diagnosis, and financial management.

[0054] Based on the same inventive concept, such as Figure 5 As shown, this embodiment of the invention also provides a multimodal graph-enhanced retrieval dialogue system 500 based on feature fusion optimization, including: a model domainization module 510, a knowledge base construction module 520, a query understanding module 530, an association retrieval module 540, a visual enhancement module 550, and a multimodal reasoning module 560.

[0055] The model domainization module 510 is used to pre-train and fine-tune the visual model and the language model using domain image and text data, respectively.

[0056] The knowledge base construction module 520 is used to segment and process the text data in the knowledge base, and then use the fine-tuned language model to extract entities and relations to construct a knowledge graph. At the same time, the image data in the knowledge base is associated with the corresponding entities and indexed, and then stored together with the entity semantic vector and the relation semantic vector in the vector database.

[0057] The query understanding module 530 receives the user's original query, performs semantic parsing and proactive optimization on it using a fine-tuned language model, and further extracts the target entities and relationships to form a structured search intent.

[0058] The association retrieval module 540 is used to perform graph query and semantic retrieval in the knowledge graph and vector database respectively based on the retrieval intent, and obtain relevant subgraphs, text semantic vector information and associated image data.

[0059] The visual enhancement module 550 is used to encode the subgraph into a knowledge context input dynamic cue generator to produce visual cues, and to extract enhanced visual features based on the visual cues and associated image data through a pre-trained visual model.

[0060] The multimodal reasoning module 560 is used to combine subgraphs, text semantic vector information, and enhanced visual features to form a multimodal context, which is then input into a fine-tuned language model for collaborative reasoning to generate the final answer and output it.

[0061] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described multimodal map enhancement retrieval method based on feature fusion optimization when executing the computer program.

[0062] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described multimodal map enhancement retrieval method based on feature fusion optimization.

[0063] It should be noted that the multimodal graph enhancement retrieval dialogue system, electronic device, and computer-readable storage medium based on feature fusion optimization provided in the above embodiments all belong to the same inventive concept as the multimodal graph enhancement retrieval method based on feature fusion optimization. For details of their specific implementation process, please refer to the embodiments of the multimodal graph enhancement retrieval method based on feature fusion optimization, which will not be repeated here.

[0064] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal map enhancement retrieval method based on feature fusion optimization, characterized in that, Includes the following steps: The visual model and language model were pre-trained and fine-tuned using domain image and text data, respectively. After segmenting and processing the text data in the knowledge base, the fine-tuned language model is used to extract entities and relations to construct a knowledge graph. At the same time, the image data in the knowledge base is associated with the corresponding entities and indexed, and then stored together with the entity semantic vector and the relation semantic vector in the vector database. The system receives the user's original query, performs semantic parsing and proactive optimization on it using a fine-tuned language model, and further extracts the target entities and relationships to form a structured search intent. Based on the search intent, graph queries and semantic searches are performed in the knowledge graph and vector database respectively to obtain relevant subgraphs, text semantic vector information and associated image data. Subgraphs are encoded into knowledge context inputs to a dynamic cue generator to produce visual cues. Enhanced visual features are then extracted from the visual cues and associated image data using a pre-trained visual model. The subgraph, text semantic vector information, and enhanced visual features are combined to form a multimodal context, which is then input into a fine-tuned language model for collaborative reasoning to generate the final answer and output it.

2. The multimodal map enhancement retrieval method based on feature fusion optimization according to claim 1, characterized in that, The pre-training and fine-tuning of the visual model and language model using domain image and text data respectively includes: A mask autoencoder framework is used to pre-train a vision model based on the Vision Transformer architecture. By randomly masking a high proportion of image patches in the input image, and by utilizing the encoder and lightweight decoder of the vision model to work together, the pixel values ​​of the masked regions are reconstructed, enabling the model to have the ability to learn the structured representation of the image from the visible context and have a deep understanding of visual features. We use an instruction dataset built from domain-specific text data to perform supervised fine-tuning on a general-purpose large language model. The fine-tuning objectives include entity recognition and relation understanding, enabling the model to have a deep understanding of domain terminology, knowledge structure, and semantic logic.

3. The multimodal map enhancement retrieval method based on feature fusion optimization according to claim 1, characterized in that, The process of segmenting and processing text data in the knowledge base, followed by extracting entities and relations using a fine-tuned language model to construct a knowledge graph, includes: The text data in the knowledge base is processed by multi-level segmentation, including first splitting it into preliminary segments by paragraph, and then using a sliding window with overlapping areas to split the ultra-long paragraphs in the preliminary segments into fine-grained segments, resulting in several text fragments. Based on the preset entity relation extraction prompt word template, the structured information of the text fragments is extracted using a fine-tuned language model. The extraction results are then subjected to multiple rounds of data optimization, including cleaning, merging, and deduplication, to form a standardized entity and relation dataset and construct a knowledge graph.

4. The multimodal map enhancement retrieval method based on feature fusion optimization according to claim 1, characterized in that, The step of associating and indexing image data in the knowledge base with corresponding entities includes: In the knowledge graph, the storage path of the image file or a unique identifier is added as an attribute to the entity node of the image type, and the semantic vector of the entity node is associated with the image identifier in the vector database to establish a direct mapping relationship from entity semantics to image.

5. The multimodal map enhancement retrieval method based on feature fusion optimization according to claim 1, characterized in that, The process involves receiving the user's original query, performing semantic parsing and proactive optimization using a fine-tuned language model, further extracting target entities and relationships to form a structured search intent, including: The fine-tuned language model is used to parse the user's original query and identify key information and potential missing points. Based on entity relationships and common domain interaction scenarios in knowledge graphs, key information and potential missing points are supplemented as prompt words, and several candidate completion directions or explicit statements are generated in combination with prompt words. These candidate results are fed back to the user for confirmation in order to optimize and obtain a semantically clear query expression. Finally, the target entities and relationships are accurately extracted from them to form the structured search intent.

6. The multimodal map enhancement retrieval method based on feature fusion optimization according to claim 1, characterized in that, The dynamic cue generator is a lightweight multilayer perceptron. Its input is an encoded subgraph knowledge context vector, and its output is a set of learnable visual cue vectors. When this set of cue vectors is input into the visual model, it is concatenated with the image patch embedding vector. By guiding the attention mechanism inside the model, it achieves knowledge context-driven image region focusing.

7. The multimodal map enhancement retrieval method based on feature fusion optimization according to claim 6, characterized in that, The dynamic cue generator introduces orthogonality constraints when generating visual cue vectors, ensuring that different cue vectors focus on different semantic regions in the image, thus guaranteeing the diversity of cue vectors and comprehensively covering key visual information.

8. A multimodal graph-enhanced retrieval dialogue system based on feature fusion optimization, implemented using the multimodal graph-enhanced retrieval method based on feature fusion optimization as described in any one of claims 1 to 7, characterized in that, include: The module includes a model domainization module, a knowledge base construction module, a query understanding module, an association retrieval module, a visual enhancement module, and a multimodal reasoning module. The model domainization module is used to pre-train and fine-tune the visual model and the language model using domain image and text data, respectively. The knowledge base construction module is used to segment and process the text data in the knowledge base, and then use the fine-tuned language model to extract entities and relations to construct a knowledge graph. At the same time, the image data in the knowledge base is associated with the corresponding entities and indexed, and then stored together with the entity semantic vector and the relation semantic vector in the vector database. The query understanding module is used to receive the user's original query, perform semantic parsing and proactive optimization on it using a fine-tuned language model, and further extract the target entities and relationships to form a structured search intent. The association retrieval module is used to perform graph query and semantic retrieval in the knowledge graph and vector database respectively based on the retrieval intent, and obtain relevant subgraphs, text semantic vector information and associated image data; The visual enhancement module is used to encode the subgraph into a knowledge context input dynamic prompt generator to generate visual prompts, and extract enhanced visual features based on the visual prompts and associated image data through a pre-trained visual model. The multimodal reasoning module is used to combine subgraphs, text semantic vector information, and enhanced visual features to form a multimodal context, which is then input into a fine-tuned language model for collaborative reasoning to generate the final answer and output it.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the multimodal map enhancement retrieval method based on feature fusion optimization as described in any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the multimodal graph enhancement retrieval method based on feature fusion optimization as described in any one of claims 1 to 7.

Images