Multi-modal heterogeneous knowledge fusion construction and semantic enhancement retrieval system based on large model

Abstract

The invention relates to the technical field of multi-modal data processing and semantic retrieval, in particular to a multi-modal heterogeneous knowledge fusion construction and semantic enhancement retrieval system based on a large model, which comprises a data acquisition module, a semantic analysis module, a knowledge fusion module and a retrieval optimization module. Multi-modal data such as texts, images and audios are uniformly expressed and deeply analyzed by introducing a large model technology, a knowledge graph is dynamically constructed, a structure is optimized in combination with a user query intention, and meanwhile accurate sorting and screening are achieved through a semantic enhancement algorithm. According to the method, the semantic comprehension capability and the intelligent level of the system can be improved, the real-time and diversified scene requirements are met, and the accuracy and the adaptability of a retrieval result are remarkably enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of information technology and artificial intelligence technology, and specifically relates to a multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system based on a large model. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and big data technologies, knowledge fusion and semantically enhanced retrieval of multimodal, heterogeneous data have become research hotspots. Efficiently integrating data from diverse sources and modalities, and achieving accurate semantically enhanced retrieval in complex scenarios, is a key area of ​​focus. Existing technologies have limitations in processing efficiency, depth of semantic understanding, and system intelligence, limiting their ability to support the growing demand for applications.

[0003] Currently, a Chinese invention patent with publication number CN118690838B discloses a knowledge fusion method and system for multi-source heterogeneous multimodal data, with a publication date of May 30, 2025. This patent realizes the unified representation and knowledge fusion of multimodal data through steps such as preprocessing, feature extraction, dimensionality reduction, alignment, graph neural network fusion, and knowledge graph integration. However, this technical solution mainly relies on traditional graph neural networks for feature fusion, and has not fully introduced large model technology, resulting in room for improvement in its capabilities in complex semantic understanding and cross-modal association mining. In addition, the semantically enhanced retrieval function of this solution is relatively weak, and fails to fully consider the dynamic changes of user query intentions and the in-depth use of contextual information, which affects the accuracy of the retrieval results and the user experience.

[0004] At the same time, a risk prediction method based on multimodal data fusion is disclosed in a Chinese invention patent with publication number CN117708746B, which was published on April 30, 2024. This patent collects text, image and audio data, combines natural language processing, computer vision and speech analysis technology, builds a multimodal risk prediction model, and improves the prediction accuracy through outlier detection. However, this technical solution focuses on risk prediction in specific fields, and has not formed a general multimodal heterogeneous knowledge fusion framework. Its semantic enhancement capabilities are relatively limited, and its flexibility to adapt to diverse scenarios is insufficient. In addition, the solution has limited performance in the dynamic update of knowledge graphs and semantic reasoning capabilities, making it difficult to meet application scenarios with high real-time requirements.

[0005] The above issues indicate that existing technologies still need improvement in knowledge fusion and semantically enhanced retrieval of multimodal heterogeneous data: first, the in-depth application of large model technology limits the system's semantic understanding capabilities and intelligence level; second, the semantically enhanced retrieval function fails to fully tap into user intent and contextual information; and third, the dynamic updating and reasoning capabilities of the knowledge graph need to be strengthened to better support the needs of real-time and efficient semantically enhanced retrieval. Therefore, the present invention aims to provide a multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system based on large models to address the above issues. Summary of the Invention

[0006] The technical problem addressed by this invention is that related technologies lack deep semantic understanding and dynamic knowledge fusion of multimodal, heterogeneous data, failing to fully tap into user query intent and contextual information, resulting in insufficient semantically enhanced retrieval capabilities. Furthermore, existing technologies have limitations in the dynamic updating and reasoning capabilities of knowledge graphs, making it difficult to meet the needs of real-time and diverse scenarios.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system, including a data acquisition module, a semantic parsing module, a knowledge fusion module and a retrieval optimization module.

[0008] The data acquisition module is used to acquire multimodal heterogeneous data in the target domain, including text data, image data and audio data, and extract contextual information related to the data; the semantic analysis module is used to uniformly represent and semantically analyze the collected multimodal data, generate semantic vectors and annotate semantic tags; the knowledge fusion module deeply processes the semantic vectors by introducing large model technology, constructs a dynamic knowledge graph, and adjusts the structure of the knowledge graph according to user query intentions and contextual information; the retrieval optimization module uses a semantic enhancement algorithm to sort and filter the nodes in the knowledge graph and output the optimal retrieval results.

[0009] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system described in the present invention, the text data includes natural language text and its meta-information, the image data includes pixel matrix and its annotation information, and the audio data includes waveform data and its spectral features; the context information includes timestamp, geographic location and user behavior records; the target domain is represented as a data set under a specific application scenario, such as medical diagnosis, financial risk control or intelligent customer service.

[0010] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system described in the present invention, the semantic vector generation logic of the semantic parsing module includes: obtaining the original features of the input data, converting the text data into word embedding vectors, converting the image data into convolution feature vectors, and converting the audio data into Mel-spectrum feature vectors; normalizing each type of feature vector so that its value is distributed between 0 and 1; calculating the association weights between different modal features through a multi-head attention mechanism, and splicing the weighted feature vectors into a unified semantic vector; generating semantic labels according to the distribution characteristics of the semantic vectors, and the number of semantic labels is consistent with the preset classification categories.

[0011] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantic-enhanced retrieval system described in the present invention, the semantic tags are represented as semantic categories of data in the target domain, and the semantic categories include entity classes, relationship classes, and event classes; the entity classes are represented as specific objects in the data, such as names of people, places, or items; the relationship classes are represented as associations between objects in the data, such as causal relationships, subordinate relationships, or parallel relationships; the event classes are represented as specific actions or state changes described in the data, such as transaction behaviors, disease diagnoses, or weather changes.

[0012] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantic-enhanced retrieval system described in the present invention, the dynamic knowledge graph construction logic of the knowledge fusion module includes: inputting semantic vectors as nodes into the graph neural network, calculating the similarity matrix between nodes; constructing an initial knowledge graph based on the similarity matrix, and setting the edge weights between nodes as similarity values; introducing large-model technology to iteratively optimize the initial knowledge graph, and adjusting the embedded representation of the nodes through self-supervised learning; dynamically adjusting the structure of the knowledge graph according to user query intentions and contextual information, and adding or deleting nodes and their corresponding edges.

[0013] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system described in the present invention, wherein: historical query records in the target field and their corresponding retrieval results are obtained, and query intent and context information are extracted; the first value is set as the ideal query intent, and the second value is set as the ideal context information, and the retrieval results corresponding to the ideal query intent and ideal context information are obtained, which are recorded as ideal retrieval results; the difference between the query intent and context information of the historical query record and the first value and the second value is calculated, which is recorded as the intent difference and the context difference; the difference between the corresponding retrieval result and the ideal retrieval result is calculated, which is recorded as the result difference; with the intent difference and the context difference as independent variables and the result difference as the dependent variable, a multivariate regression analysis is performed to obtain the coefficient of the intent difference and the coefficient of the context difference, and the coefficient of the intent difference and the coefficient of the context difference are set as the weights of the query intent and context information.

[0014] As a preferred solution of the large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system described in the present invention, a semantic enhancement function is constructed according to the weights, and the calculation expression of the semantic enhancement function is: , where S is the result difference, Expressed as i independent variables, For the i The weight corresponding to the independent variable; input the current query intention and context information into the semantic enhancement model, obtain the current result difference, obtain the current query record, obtain the retrieval result corresponding to the current query record, sum the current result difference with the retrieval result, obtain the first sum value, and set the first sum value as the new retrieval result.

[0015] As a preferred solution of the multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system based on a large model described in the present invention, the sorting logic of the retrieval optimization module includes: obtaining all nodes in the dynamic knowledge graph and their corresponding edge weights, performing weighted summation of the semantic vector of the node and its edge weight to obtain the comprehensive score of the node; sorting the nodes in descending order according to the comprehensive score of the node, and selecting the top k nodes. k nodes as candidate nodes; the candidate nodes are screened, nodes that are irrelevant to the user's query intention are eliminated, and the remaining nodes are retained as the final retrieval results.

[0016] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system described in the present invention, wherein: user query intention and context information are input into the retrieval optimization module to obtain candidate nodes and their comprehensive scores; the third value is set as the scoring threshold, the comprehensive score of the candidate node is compared with the scoring threshold, and a first operation is performed according to the comparison result, the first operation including node retention and node removal; when the comprehensive score of the candidate node is greater than or equal to the scoring threshold, the first operation is set to node retention, otherwise, the first operation is set to node removal.

[0017] As a preferred solution for the large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system described in the present invention, a first mapping relationship is established, which is represented by a mapping relationship between query intent, context information and final retrieval results, and the corresponding final retrieval results are obtained by inputting query intent and context information.

[0018] The technical effects of the present invention are as follows: by introducing large model technology to perform deep semantic analysis and dynamic knowledge fusion of multimodal heterogeneous data, the semantic understanding ability and intelligence level of the system are improved; by constructing a dynamic knowledge graph and optimizing it in combination with user query intentions and contextual information, the accuracy and adaptability of semantically enhanced retrieval are enhanced; through the introduction of multiple regression analysis and semantic enhancement functions, dynamic adjustment and optimization of retrieval results are achieved, meeting the needs of real-time and diversified scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the system architecture of the present invention; Figure 2 This is the workflow diagram of the data acquisition module; Figure 3 This is the processing flow chart of the semantic parsing module; Figure 4 Construct a flow chart for the dynamic knowledge graph of the knowledge fusion module; Figure 5 Calculate the logical graph for query intent and context information weights; Figure 6 Node sorting and screening flow chart for the retrieval optimization module; Figure 7 This is a schematic diagram of the application of semantic enhancement function; Figure 8 This is a schematic diagram of the first mapping relationship.

[0020] The accompanying drawings are numbered as follows: 1. Data acquisition module; 2. Semantic analysis module; 3. Knowledge fusion module; 4. Retrieval optimization module; 5. Dynamic knowledge graph; 6. Semantic vector; 7. Query intent; 8. Contextual information. DETAILED DESCRIPTION

[0021] The present invention provides a multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system based on a large model. Figure 1 To the attached Figure 8 The system consists of a data acquisition module 1, a semantic analysis module 2, a knowledge fusion module 3 and a retrieval optimization module 4. The modules are logically connected to each other to achieve data flow and processing.

[0022] like Figure 1 As shown, the data acquisition module 1 is the first link of the system, and its main function is to obtain multimodal heterogeneous data in the target field and extract context information. In actual operation, the data acquisition module 1 is connected to the external data source through an interface, and receives various data forms including text data, image data and audio data. Among them, text data is input in the form of natural language text and its meta-information, image data is input in the form of pixel matrix and its annotation information, and audio data is input in the form of waveform data and its spectral characteristics. Context information comes from auxiliary data such as timestamps, geographic locations and user behavior records. These data are organized into a unified format and output through the internal processing flow of the data acquisition module 1. The specific process is as follows. Figure 2 As shown in Figure 2, the data acquisition module 1 first categorizes and stores the received data, and associates and binds the context information of each type of data with its main data to form a complete data unit. Subsequently, these data units are passed to the semantic analysis module 2 for further processing.

[0023] The core task of the semantic analysis module 2 is to uniformly represent and semantically analyze the multimodal data from the data acquisition module 1. Figure 3 As shown, the workflow of the semantic parsing module 2 is divided into multiple steps. First, module 2 extracts original features from the data unit, converts text data into word embedding vectors, converts image data into convolution feature vectors, and converts audio data into Mel spectrum feature vectors. After these feature vectors are normalized, the numerical distribution is limited to between 0 and 1 to ensure that the feature vectors of different modal data are comparable. Next, module 2 uses a multi-head attention mechanism to calculate the association weights between different modal features. This process generates a unified semantic vector 6 by weighted summation of various feature vectors, and generates semantic labels based on the distribution characteristics of the semantic vector 6. The number of semantic labels is consistent with the preset classification categories, which are used to identify the semantic categories of data in the target domain, including entity classes, relationship classes, and event classes. The semantic parsing module 2 passes the generated semantic vector 6 and semantic labels to the knowledge fusion module 3.

[0024] The knowledge fusion module 3 deeply processes the semantic vector 6 by introducing the large model technology to construct a dynamic knowledge graph 5. Figure 4 As shown in Figure 3, the workflow of the knowledge fusion module 3 includes node similarity calculation, initial knowledge graph construction, and iterative optimization based on the large model. Module 3 inputs the semantic vector 6 as a node into the graph neural network and calculates the similarity matrix between nodes. The values ​​in the similarity matrix are used to set the edge weights between nodes to construct the initial knowledge graph. Subsequently, module 3 introduces the large model technology to iteratively optimize the initial knowledge graph and adjust the embedded representation of the node through self-supervised learning. In this process, module 3 is based on the user query meaning. Figure 7 The structure of the knowledge graph 5 is dynamically adjusted by using context information 8, adding or deleting nodes and their corresponding edges. In addition, module 3 also determines the query intent through multiple regression analysis. Figure 7 and the weight of context information 8, the specific logic is as follows Figure 5 Module 3 first obtains the historical query records and corresponding search results in the target field, extracts the query meaning, and Figure 7 and context information 8. Subsequently, module 3 records the retrieval results corresponding to the ideal query intent and the ideal context information as the ideal retrieval results, and calculates the query intent of the historical query records. Figure 7 The difference between the context information 8 and the ideal value. Through multiple regression analysis, module 3 obtains the query meaning. Figure 7 and the weight of context information 8, and use it to construct the semantic enhancement function. The specific application of semantic enhancement function is as follows Figure 7 As shown, module 3 converts the current query into Figure 7 The context information 8 is input into the semantic enhancement model to obtain the current result difference, which is then summed with the retrieval result to finally generate a new retrieval result.

[0025] The search optimization module 4 is responsible for sorting and filtering the nodes in the knowledge graph 5 using the semantic enhancement algorithm and outputting the optimal search results. Figure 6 As shown in Figure 4, the workflow of module 4 includes comprehensive score calculation, candidate node sorting, and filtering irrelevant nodes. Module 4 first obtains all nodes in the dynamic knowledge graph 5 and their corresponding edge weights, and performs weighted summation of the node's semantic vector 6 and its edge weight to obtain the node's comprehensive score. Based on the comprehensive score, module 4 sorts the nodes in descending order and selects the top k nodes as candidate nodes. Subsequently, module 4 filters the candidate nodes and eliminates those that are irrelevant to the user's query. Figure 7 Irrelevant nodes are selected and the remaining nodes are retained as the final search results. In actual operation, module 4 sets a scoring threshold and compares the comprehensive score of the candidate node with the scoring threshold. When the comprehensive score of the candidate node is greater than or equal to the scoring threshold, the node is retained; otherwise, the node is eliminated. The final output result of module 4 is compared with the query intent through the first mapping relationship. Figure 7 Establish a corresponding relationship with the context information 8, such as Figure 8 The first mapping relationship clearly shows the query meaning. Figure 7 , the logical connection between context information 8 and the final retrieval results.

[0026] In the above-described implementation, the data acquisition module 1, semantic analysis module 2, knowledge fusion module 3, and retrieval optimization module 4 work together through logical connections to achieve deep semantic understanding and dynamic knowledge fusion of multimodal heterogeneous data. In practical application scenarios, such as medical diagnosis, the system can generate a dynamic knowledge graph 5 and optimize search results by analyzing a patient's medical records, medical images, and voice recordings, combined with contextual information such as timestamps and geographic location. Similarly, in the field of financial risk control, the system can integrate transaction records, user behavior data, and audio call content, and dynamically adjust search results through semantic enhancement functions to meet the needs of real-time and diverse scenarios.

[0027] In order to better enable relevant personnel in this technical field to fully understand and implement the present invention, the specific implementation principle of the present invention is further supplemented below with reference to a specific application scenario.

[0028] In the field of medical diagnosis, the system generates a dynamic knowledge graph5 by analyzing the patient's medical records, medical images, and voice recordings, combined with contextual information such as timestamps and geographic locations, and optimizes search results to support doctors' diagnostic decisions. The specific operation steps are as follows: First, Data Acquisition Module 1 acquires multimodal, heterogeneous data from the hospital information system. For example, patients' electronic medical records are input as natural language text, medical images are input as pixel matrices and their annotations, and voice recordings are input as waveform data and their spectral features. Simultaneously, the system extracts contextual information related to the data, including the timestamp of the patient's visit, the hospital's geographic location, and the doctor's operational records. After classification and storage, this data is formed into complete data units and passed to Semantic Parsing Module 2.

[0029] Subsequently, the semantic parsing module 2 performs a unified representation and semantic parsing of the multimodal data in the data unit. Module 2 first extracts raw features, converting medical record text into word embedding vectors, medical images into convolutional feature vectors, and voice recordings into mel-spectrogram feature vectors. These feature vectors are normalized, and their numerical distribution is constrained to range from 0 to 1 to ensure comparability of feature vectors from different modalities. Next, module 2 utilizes a multi-head attention mechanism to calculate the association weights between features from different modalities and generates a unified semantic vector 6 by weighted summation of each type of feature vector. Based on the distribution characteristics of the semantic vector 6, module 2 generates semantic labels to identify semantic categories within the target domain, such as entity classes (patient names, disease names), relationship classes (the relationship between causes and symptoms), and event classes (diagnostic behaviors). These semantic vectors 6 and semantic labels are passed to the knowledge fusion module 3.

[0030] The knowledge fusion module 3 uses the large model technology to deeply process the semantic vector 6 and construct a dynamic knowledge graph 5. The module 3 inputs the semantic vector 6 as a node into the graph neural network and calculates the similarity matrix between the nodes. The values ​​in the similarity matrix are used to set the edge weights between the nodes to construct the initial knowledge graph. Subsequently, the module 3 uses the large model technology to iteratively optimize the initial knowledge graph and adjust the embedded representation of the nodes through self-supervised learning. In this process, the module 3 uses the user query meaning to Figure 7 The structure of the knowledge graph 5 is dynamically adjusted based on the context information 8. For example, when a doctor queries for a diagnosis of a disease, module 3 will Figure 7 (disease name) and context information 8 (patient age, visit time) to add or delete relevant nodes and their corresponding edges. In addition, module 3 determines the query meaning through multiple regression analysis. Figure 7 Specifically, module 3 obtains historical query records and their corresponding search results in the target field, extracts query meaning, and Figure 7 and context information 8, and calculates the difference between it and the ideal value. Through multiple regression analysis, module 3 obtains the query meaning. Figure 7 and context information 8, and use them to construct semantic enhancement function. Module 3 Figure 7 The context information 8 is input into the semantic enhancement model to obtain the current result difference, which is then summed with the retrieval result to finally generate a new retrieval result.

[0031] Finally, the search optimization module 4 uses the semantic enhancement algorithm to sort and filter the nodes in the knowledge graph 5 and output the optimal search results. Module 4 first obtains all nodes in the dynamic knowledge graph 5 and their corresponding edge weights, and performs a weighted summation of the node's semantic vector 6 and its edge weight to obtain the node's comprehensive score. Based on the comprehensive score, module 4 sorts the nodes in descending order and selects the top k nodes as candidate nodes. Subsequently, module 4 filters the candidate nodes and eliminates nodes that do not match the user's query intent. Figure 7 For example, when a doctor searches for treatment options for a certain disease, module 4 will remove nodes that are not related to the disease and retain the remaining nodes as the final search results. In actual operation, module 4 sets a scoring threshold and compares the comprehensive score of the candidate node with the scoring threshold. When the comprehensive score of the candidate node is greater than or equal to the scoring threshold, the node is retained; otherwise, the node is removed. The final output result of module 4 is compared with the query intent through the first mapping relationship. Figure 7 Establish a corresponding relationship with the context information 8, clearly showing the query meaning Figure 7 , the logical connection between context information 8 and the final retrieval results.

[0032] Through the above steps, the system can achieve deep semantic understanding and dynamic knowledge fusion of multimodal, heterogeneous data, and optimize search results based on user query intent and contextual information. For example, in medical diagnosis scenarios, the system can quickly retrieve diagnostic plans and treatment recommendations related to the patient's condition, meeting the needs of real-time and diverse scenarios. Similarly, in the field of financial risk control, the system can dynamically adjust search results by integrating transaction records, user behavior data, and audio call content to support risk prediction and decision-making.

[0033] Any content not described in detail in the specification belongs to the prior art known to those skilled in the art, and the model parameters of each electrical appliance are not specifically limited, and conventional equipment can be used. In this technical solution, electrical control components not mentioned are not shown in the figures because they belong to the prior art and will not be described here.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal heterogeneous knowledge fusion and semantic enhancement retrieval system based on a large model, characterized by: The system comprises a data acquisition module (1), a semantic analysis module (2), a knowledge fusion module (3) and a retrieval optimization module (4); the data acquisition module (1) is used to acquire multimodal heterogeneous data in a target domain, including text data, image data and audio data, and extract contextual information related to the data; the semantic analysis module (2) is used to uniformly represent and semantically analyze the acquired multimodal data, generate semantic vectors (6) and annotate semantic tags; the knowledge fusion module (3) deeply processes the semantic vectors (6) by introducing large model technology, constructs a dynamic knowledge graph (5), and adjusts the structure of the knowledge graph (5) according to the user's query intention (7) and contextual information (8); the retrieval optimization module (4) uses a semantic enhancement algorithm to sort and filter the nodes in the knowledge graph (5) and outputs the optimal retrieval result.

2. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 1, characterized in that: The text data includes natural language text and its meta-information, the image data includes pixel matrix and its annotation information, and the audio data includes waveform data and its spectral features; the context information includes timestamp, geographic location and user behavior records; the target domain is represented as a data set in a specific application scenario.

3. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 1, characterized in that: The semantic vector (6) generation logic of the semantic parsing module (2) includes: obtaining the original features of the input data, converting the text data into a word embedding vector, converting the image data into a convolution feature vector, and converting the audio data into a Mel spectrum feature vector; normalizing each feature vector so that its value is distributed between 0 and 1; The association weights between different modal features are calculated through a multi-head attention mechanism, and the weighted feature vectors are concatenated into a unified semantic vector (6). Semantic labels are generated based on the distribution characteristics of the semantic vector (6), and the number of semantic labels is consistent with the preset classification categories.

4. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 3, characterized in that: The semantic tags are represented as semantic categories of data in the target domain, and the semantic categories include entity classes, relationship classes, and event classes; the entity classes are represented as specific objects in the data; the relationship classes are represented as associations between objects in the data; and the event classes are represented as specific actions or state changes described in the data.

5. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 1, characterized in that: The dynamic knowledge graph (5) construction logic of the knowledge fusion module (3) includes: inputting the semantic vector (6) as a node into the graph neural network and calculating the similarity matrix between the nodes; constructing the initial knowledge graph based on the similarity matrix and setting the edge weights between the nodes as the similarity values; introducing the large model technology to iteratively optimize the initial knowledge graph and adjust the embedded representation of the nodes through self-supervised learning; dynamically adjusting the structure of the knowledge graph (5) according to the user query intention (7) and context information (8), and adding or deleting nodes and their corresponding edges.

6. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 1, characterized in that: The knowledge fusion module (3) determines the weights of the query intent (7) and the context information (8) through multiple regression analysis. The specific logic includes: obtaining historical query records in the target field and their corresponding retrieval results, extracting the query intent (7) and the context information (8); setting the first value as the ideal query intent, setting the second value as the ideal context information, obtaining the retrieval results corresponding to the ideal query intent and the ideal context information, and recording them as ideal retrieval results; calculating the difference between the query intent (7) and the context information (8) of the historical query record and the first value and the second value, and recording them as the intent difference and the context difference; calculating the difference between the corresponding retrieval result and the ideal retrieval result, and recording them as the result difference; using the intent difference and the context difference as independent variables and the result difference as the dependent variable, performing multiple regression analysis to obtain the coefficient of the intent difference and the coefficient of the context difference, and setting the coefficient of the intent difference and the coefficient of the context difference as the weights of the query intent (7) and the context information (8).

7. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 6, characterized in that: A semantic enhancement function is constructed according to the weights, and the calculation expression of the semantic enhancement function is: , where S is the result difference, Expressed as i independent variables, For the i The weight corresponding to the independent variable is input into the semantic enhancement model; the current query intention (7) and context information (8) are input to obtain the current result difference, the current query record is obtained, the retrieval result corresponding to the current query record is obtained, the current result difference is summed with the retrieval result to obtain a first sum value, and the first sum value is set as a new retrieval result.

8. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 1, characterized in that: The sorting logic of the retrieval optimization module (4) includes: obtaining all nodes in the dynamic knowledge graph (5) and their corresponding edge weights, performing weighted summation of the semantic vector (6) of the node and its edge weight to obtain the comprehensive score of the node; sorting the nodes in descending order according to the comprehensive score of the node, and selecting the top k nodes. k nodes as candidate nodes; the candidate nodes are screened, nodes that are irrelevant to the user's query intention (7) are eliminated, and the remaining nodes are retained as the final retrieval results.

9. The large-model-based multimodal heterogeneous knowledge fusion construction and semantically enhanced retrieval system according to claim 8, characterized in that: Inputting the user query intention (7) and context information (8) into the retrieval optimization module (4) to obtain candidate nodes and their comprehensive scores; setting the third value as the score threshold, comparing the comprehensive score of the candidate node with the score threshold, and performing a first operation based on the comparison result, wherein the first operation includes node retention and node removal; when the comprehensive score of the candidate node is greater than or equal to the score threshold, setting the first operation to node retention, otherwise, setting the first operation to node removal.

10. The large-model-based multimodal heterogeneous knowledge fusion construction and semantic enhancement retrieval system according to claim 1, characterized in that: A first mapping relationship is established, wherein the first mapping relationship is represented as a mapping relationship between the query intention (7), the context information (8) and the final retrieval result. By inputting the query intention (7) and the context information (8), the corresponding final retrieval result is obtained.

Images