Knowledge Graph Optimization Method Based on Large Model and Multimodal Data Fusion

Abstract

This invention discloses a knowledge graph optimization method based on the fusion of large models and multimodal data, comprising the following steps: S1. Constructing a multimodal dataset; S2. Forming a preprocessed multimodal dataset; S3. Generating a unified semantic vector representation; S4. Inputting the unified semantic vector representation into an adaptive Poisson distribution model to dynamically model the arrival rate and distribution characteristics of the multimodal data, and determining adaptive parameters for data sampling and updating; S5. Generating a fused semantic representation set; S6. Using the fused data representation to perform entity extraction and relation recognition, constructing a preliminary knowledge graph; S7. Automatically verifying, removing redundant information, and adaptively adjusting the structure of the knowledge graph's nodes, edges, and attributes, forming an optimized knowledge graph. This invention can adjust the data acquisition and graph update frequency according to real-time data semantic changes, ensuring that the graph construction process is semantically driven and time-sensitive.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, and in particular to a knowledge graph optimization method based on the fusion of large models and multimodal data. Background Technology

[0002] With the rapid development of artificial intelligence, big data, and the Internet of Things, knowledge graphs, as an important tool capable of structurally representing massive amounts of information and revealing semantic relationships between entities, have been widely applied in search engines, intelligent question answering, recommendation systems, and the healthcare field. To enhance the breadth and depth of knowledge graph construction, researchers have begun to explore the introduction of multimodal data such as images, audio, and video to achieve comprehensive modeling of complex semantics. However, existing knowledge graph construction methods mainly rely on structured or semi-structured text data, and have limited processing capabilities for non-textual modal data, resulting in significant deficiencies in the information completeness and semantic richness of the generated graphs.

[0003] Specifically, existing methods generally suffer from the following shortcomings in multimodal data processing: First, there is a lack of a unified semantic alignment mechanism among multimodal data, making it difficult to effectively fuse features of different modalities in the same vector space; second, traditional modeling methods cannot dynamically respond to changes in multimodal data in time and semantics, and lack flexible data update and sampling strategies, resulting in long graph construction cycles and poor timeliness; third, existing methods for optimizing knowledge graph structures are mostly based on static rules or single strategies, making it difficult to balance semantic consistency and structural rationality, and easily introducing redundant nodes and erroneous relationships, affecting the usability and accuracy of the graph.

[0004] In summary, existing technologies have significant shortcomings in multimodal semantic unified modeling, adaptive sampling and update mechanisms, and graph structure optimization. There is an urgent need for a novel knowledge graph optimization method that can integrate the semantic encoding capabilities of large models with the multimodal data perception capabilities, in order to break through the current bottlenecks in heterogeneous information integration and dynamic knowledge modeling. Summary of the Invention

[0005] One objective of this invention is to propose a knowledge graph optimization method based on the fusion of large models and multimodal data. This invention can adjust the data acquisition and graph update frequency according to real-time data semantic changes, ensuring that the graph construction process is semantically driven and time-sensitive.

[0006] A knowledge graph optimization method based on the fusion of large models and multimodal data according to an embodiment of the present invention includes the following steps:

[0007] S1. Automatically collect various multimodal data, including text, images, audio, and video, through multiple channels to construct a multimodal dataset;

[0008] S2. Preprocess the multimodal dataset to form a preprocessed multimodal dataset;

[0009] S3. Utilize a pre-trained large model to perform multi-level feature extraction and semantic encoding on the pre-processed multimodal dataset to generate a unified semantic vector representation;

[0010] S4. Input the unified semantic vector representation into the adaptive Poisson distribution model to dynamically model the arrival rate and distribution characteristics of multimodal data, and determine the adaptive parameters for data sampling and updating;

[0011] S5. Based on unified semantic vector representation and adaptive parameters, a multimodal data fusion algorithm is used to perform semantic alignment and feature integration on each modality of data to generate a fused semantic representation set;

[0012] S6. Entity extraction and relation recognition are performed using the fused data representation to construct a preliminary knowledge graph, wherein the entities and their relations in the constructed knowledge graph are based on the fused data representation;

[0013] S7. Implement a structure optimization algorithm based on adaptive Poisson distribution reinforcement learning on the preliminary knowledge graph to automatically verify the nodes, edges and their attributes, remove redundant information and adaptively adjust the structure to form an optimized knowledge graph.

[0014] Optionally, S1 includes the following steps:

[0015] S11. Set the multimodal data acquisition channel set M = {m t ,m i ,m a ,m v}, where m t m i m a m v These represent the acquisition channels used to collect text, image, audio, and video data, respectively. Acquisition rules are configured based on heterogeneous information sources to construct a multimodal data stream.

[0016] S12. For each modal data s k ∈M, a distributed task scheduling mechanism is used to trigger the acquisition module in parallel, and the data stream is collected in a polling manner according to the preset sampling period to form a time-series multimodal dataset;

[0017] S13. Summarize all multimodal data acquisition results into the original dataset to generate a multimodal dataset D with a consistent structure. raw :

[0018]

[0019] Where, d k This represents the k-th multimodal data sample, where N is the total number of samples collected, and t k Indicates the data collection timestamp, v k Indicates the original data content, l k A logical identifier indicating the source of the data.

[0020] Optionally, S2 includes the following steps:

[0021] S21. For the multimodal dataset D raw Each multimodal data sample d in k Based on its modal data s k Each module calls the corresponding modality processing module to perform modality-related noise filtering operations, resulting in a denoised multimodal dataset.

[0022] S22. The denoised multimodal dataset is processed according to the modal data s. k Mapping to a unified format set generates a format-standardized multimodal dataset;

[0023] S23. Perform data cleaning on the data samples in the multimodal dataset, removing data samples with invalid formats, missing fields, abnormal collection, and duplicate logical identifiers, to form a preprocessed multimodal dataset D. clean .

[0024] Optionally, S3 includes the following steps:

[0025] S31. Transfer the preprocessed multimodal dataset D clean Input a pre-trained large model, and define the feature extraction process of the pre-trained large model as a multi-level semantic mapping function F. emb Multi-level semantic mapping function F emb Composed of multiple feature coding layers It is constructed sequentially, where each feature encoding layer E l The semantic coding unit is based on a pre-trained large model. It extracts and encodes features from the input data feature vector to generate semantic feature representations at different levels. The outputs of all feature coding layers are connected to form a unified multi-level feature vector.

[0026] S32. The unified multi-level feature vector is processed by dimensionality reduction and nonlinear mapping through the feature fusion unit of a pre-trained large model to generate a unified semantic vector representation of standard dimension, forming a unified semantic vector representation set V that can simultaneously represent the semantic information of multimodal data such as text, image, audio, and video. emb :

[0027]

[0028] in, Represents the k-th multimodal data sample d k After multi-level semantic mapping function F emb The semantic vector representation obtained afterwards.

[0029] Optionally, S4 includes the following steps:

[0030] S41. Represent the unified semantic vector representation set V emb The improved adaptive Poisson distribution model is input, and the semantic activity function Λ(·) is defined to characterize the potential influence of the k-th multimodal data sample on the newly added entities and relations in the knowledge graph at time t.

[0031]

[0032] in, For the k-th sample at time t k The semantic activity, where Λ0 is the baseline semantic activity, and β is the semantic activity. j (t) represents the structural weight of the j-th feature at time t, dynamically estimated by combining the rate of change of the entity or relation corresponding to this feature in the current knowledge graph. Let J be the projection function of the j-th semantic feature, and J be the dimension of the unified semantic vector.

[0033] S42. Constructing a joint Poisson intensity function that integrates semantic novelty and structural heterogeneity based on the semantic activity function:

[0034]

[0035] in, Let θ represent the optimal Poisson modeling parameters, and t represent the set of Poisson modeling parameters to be optimized. k Let n be the timestamp of the data sample. k This represents the number of samples within the sampling window. The KL divergence between the current sample and the semantics of existing nodes in the knowledge graph reflects semantic novelty. γ is an adjustment parameter that controls the weight of the difference. This indicates that the k-th sample is at time t. k The semantic activity value under the set of Poisson modeling parameters to be optimized;

[0036] S43. Based on the optimal Poisson modeling parameters Combined with the current knowledge graph structure state G t =(E t ,R t ) Calculate the data sampling period T samp With the map update threshold δ upd E t For entity sets, R tFor a set of relations:

[0037]

[0038] in, ρ is the average semantic activity at time t, ρ is the structure density adjustment factor used to adjust the sampling frequency according to the relation density, and η is the update sensitivity factor. This represents the average semantic deviation of the current batch of samples.

[0039] S44. Set the data sampling period T samp With the map update threshold δ upd As an adaptive parameter for the multimodal data fusion module and knowledge graph update mechanism.

[0040] Optionally, S5 includes the following steps:

[0041] S51. Based on the unified semantic vector representation set V emb With the adaptive parameter set {T samp ,δ upd Construct a multimodal semantic alignment mapping mechanism. For each multimodal data sample, based on the modality type s of the multimodal data sample... k Select the corresponding semantic alignment strategy and set the semantic vector of the semantic alignment strategy. Mapping to a shared semantic alignment space enables semantic features from different modalities to be aligned in the same vector space, forming a semantic alignment vector.

[0042] S52. Align the semantic vector Combined with the semantic activity of the corresponding samples and its KL divergence with the existing semantics of knowledge graphs In the input feature fusion mechanism, features are weighted and integrated based on the importance and novelty of multimodal semantic contributions, and the weights of each modality in the fused representation are dynamically adjusted to generate the fused data semantic representation. Construct the fused semantic representation set V fused .

[0043] Optionally, S6 includes the following steps:

[0044] S61. Combine the fused semantic representation set V fused As input, a predefined entity extraction mechanism is used to analyze the semantic representation of each data item. Based on the semantic features in the fused data representation, a candidate entity set E is identified and extracted. cand ;

[0045] S62. Combine the fused semantic representation V fused Same candidate entity set E candAs input, a relation recognition mechanism is used to detect semantic associations between candidate entities, identify and extract a candidate relation set R. cand ;

[0046] S63. Based on candidate entity set E cand With candidate relation set R cand Constructing a preliminary knowledge graph G init =(E init ,R init ),in For the initially determined set of entities, This is a preliminary set of relationships, and all entities and their relationships are constructed based on the fused data representation.

[0047] Optionally, S7 includes the following steps:

[0048] S71. The initially constructed knowledge graph G init The input structure optimization module constructs a graph structure optimization strategy set Π={π} based on an improved adaptive Poisson distribution reinforcement learning algorithm. θ}, where π θ This is a parameterized optimization strategy used to dynamically adjust the structure of nodes, edges, and their attributes in a knowledge graph.

[0049] S72. Define the state space S, action space A, and reward function R for graph optimization. The state space S represents the semantic confidence distribution of each node and relation in the current graph structure. The action space A includes node retention, edge update, attribute adjustment, and redundant removal structural operations. The reward function R is constructed based on semantic consistency score and structural sparsity.

[0050] S73. Semantic activity obtained from adaptive Poisson distribution modeling results With the map update threshold δ upd The action sampling probability and update frequency in the graph structure optimization are dynamically adjusted to make the reinforcement learning strategy time-sensitive and semantically driven.

[0051] S74. In each round of structural optimization, the optimization strategy π is adopted. θ Perform action a on the current graph structure t ∈A, and based on the current state s t ∈S to obtain reward r t =R(s) t ,a t The parameter θ is updated according to the policy gradient method, so that the structural adjustment process is continuously optimized.

[0052] S75. Repeat the structural optimization iterations until the convergence condition is met, and output the final optimized knowledge graph G. opt=(E opt ,R opt ).

[0053] Optionally, the final optimized knowledge graph construction satisfies:

[0054] Reserved nodes: semantic liveness Furthermore, it possesses structural connectivity with the core subgraph of the graph;

[0055] Remove nodes: semantic activity And there are no high-confidence related edges;

[0056] Edge retention: Both connected nodes meet the retention condition, and the semantic confidence of the edge is greater than or equal to the set edge confidence threshold;

[0057] Merging edges or attributes: Multiple semantically repetitive edges pointing to the same entity, or multiple redundant attribute values ​​are merged into a unified attribute representation after structural reduction.

[0058] The beneficial effects of this invention are:

[0059] (1) This invention inputs heterogeneous modal data of text, image, audio and video into a pre-trained large model, designs a multi-level semantic mapping function, realizes deep encoding of cross-modal semantic features, outputs a unified semantic vector representation, effectively captures the contextual semantics and structural features in different modalities through a multi-level encoding structure, and significantly reduces the semantic projection error between modalities by means of a unified semantic space alignment mechanism.

[0060] (2) This invention designs a semantic activity function based on a unified semantic vector, which combines feature dynamic weights and the potential update value of KL divergence modeling samples. By optimizing the Poisson modeling parameters, it dynamically generates the data sampling period and the map update threshold. It can adjust the data collection and map update frequency according to real-time data semantic changes, ensuring that the map construction process is semantically driven and time-sensitive.

[0061] (3) This invention proposes a graph structure optimization algorithm based on adaptive Poisson distribution reinforcement learning. It establishes a dynamic reward function with semantic confidence and structural sparsity as the core, and designs a time-sensitive action sampling mechanism by combining the current semantic activity and update threshold. The optimization strategy can automatically identify and adjust redundant nodes, weakly related edges and attribute conflicts in the knowledge graph, forming an optimization path with good convergence and strong interpretability. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0063] Figure 1This is a flowchart of a knowledge graph optimization method based on the fusion of large models and multimodal data proposed in this invention. Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0065] refer to Figure 1 A knowledge graph optimization method based on the fusion of large models and multimodal data includes the following steps:

[0066] S1. Automatically collect various multimodal data, including text, images, audio, and video, through multiple channels to construct a multimodal dataset;

[0067] S2. Preprocess the multimodal dataset to form a preprocessed multimodal dataset;

[0068] S3. Utilize a pre-trained large model to perform multi-level feature extraction and semantic encoding on the pre-processed multimodal dataset to generate a unified semantic vector representation;

[0069] S4. Input the unified semantic vector representation into the adaptive Poisson distribution model to dynamically model the arrival rate and distribution characteristics of multimodal data, and determine the adaptive parameters for data sampling and updating;

[0070] S5. Based on unified semantic vector representation and adaptive parameters, a multimodal data fusion algorithm is used to perform semantic alignment and feature integration on each modality of data to generate a fused semantic representation set;

[0071] S6. Entity extraction and relation recognition are performed using the fused data representation to construct a preliminary knowledge graph, wherein the entities and their relations in the constructed knowledge graph are based on the fused data representation;

[0072] S7. Implement a structure optimization algorithm based on adaptive Poisson distribution reinforcement learning on the preliminary knowledge graph to automatically verify the nodes, edges and their attributes, remove redundant information and adaptively adjust the structure to form an optimized knowledge graph.

[0073] In this embodiment, S1 includes the following steps:

[0074] S11. Set the multimodal data acquisition channel set M = {m t ,m i ,m a ,m v}, where m t m i m a m vThese represent the acquisition channels used to collect text, image, audio, and video data, respectively. Acquisition rules are configured based on heterogeneous information sources to construct a multimodal data stream.

[0075] S12. For each modal data s k ∈M, a distributed task scheduling mechanism is used to trigger the acquisition module in parallel, and the data stream is collected in a polling manner according to the preset sampling period to form a time-series multimodal dataset;

[0076] S13. Summarize all multimodal data acquisition results into the original dataset to generate a multimodal dataset D with a consistent structure. raw :

[0077]

[0078] Where, d k This represents the k-th multimodal data sample, where N is the total number of samples collected, and t k Indicates the data collection timestamp, v k Indicates the original data content, l k A logical identifier indicating the source of the data.

[0079] In this embodiment, S2 includes the following steps:

[0080] S21. For the multimodal dataset D raw Each multimodal data sample d in k Based on its modal data s k Each module calls the corresponding modality processing module to perform modality-related noise filtering operations, resulting in a denoised multimodal dataset.

[0081] S22. Based on the modal data s, the denoised multimodal dataset... k Mapping to a unified format set generates a format-standardized multimodal dataset;

[0082] S23. Perform data cleaning on the data samples in the multimodal dataset, removing data samples with invalid formats, missing fields, abnormal collection, and duplicate logical identifiers, to form a preprocessed multimodal dataset D. clean .

[0083] In this embodiment, S3 includes the following steps:

[0084] S31. Transfer the preprocessed multimodal dataset D clean Input a pre-trained large model, and define the feature extraction process of the pre-trained large model as a multi-level semantic mapping function F. emb Multi-level semantic mapping function F emb Composed of multiple feature coding layers It is constructed sequentially, where each feature encoding layer E l The semantic coding unit is based on a pre-trained large model. It extracts and encodes features from the input data feature vector to generate semantic feature representations at different levels. The outputs of all feature coding layers are connected to form a unified multi-level feature vector.

[0085] S32. The unified multi-level feature vectors are processed by dimensionality reduction and non-linear mapping through the feature fusion unit of a pre-trained large model to generate a unified semantic vector representation of standard dimensions, forming a unified semantic vector representation set V that can simultaneously represent the semantic information of multimodal data such as text, images, audio, and video. emb :

[0086]

[0087] in, Represents the k-th multimodal data sample d k After multi-level semantic mapping function F emb The semantic vector representation obtained afterwards.

[0088] In this embodiment, S4 includes the following steps:

[0089] S41. Represent the unified semantic vector representation set V emb The improved adaptive Poisson distribution model is input, and the semantic activity function Λ(·) is defined to characterize the potential influence of the k-th multimodal data sample on the newly added entities and relations in the knowledge graph at time t.

[0090]

[0091] in, For the k-th sample at time t k The semantic activity, where Λ0 is the baseline semantic activity, and β is the semantic activity. j (t) represents the structural weight of the j-th feature at time t, dynamically estimated by combining the rate of change of the entity or relation corresponding to this feature in the current knowledge graph. Let J be the projection function of the j-th semantic feature, and J be the dimension of the unified semantic vector.

[0092] S42. Constructing a joint Poisson intensity function that integrates semantic novelty and structural heterogeneity based on the semantic activity function:

[0093]

[0094] in, Let θ represent the optimal Poisson modeling parameters, and t represent the set of Poisson modeling parameters to be optimized. k Let n be the timestamp of the data sample. kThis represents the number of samples within the sampling window. The KL divergence between the current sample and the semantics of existing nodes in the knowledge graph reflects semantic novelty. γ is an adjustment parameter that controls the weight of the difference. This indicates that the k-th sample is at time t. k The semantic activity value under the set of Poisson modeling parameters to be optimized;

[0095] S43. Based on the optimal Poisson modeling parameters Combined with the current knowledge graph structure state G t =(E t ,R t ) Calculate the data sampling period T samp With the map update threshold δ upd E t For entity sets, R t For a set of relations:

[0096]

[0097] in, ρ is the average semantic activity at time t, ρ is the structure density adjustment factor used to adjust the sampling frequency according to the relation density, and η is the update sensitivity factor. This represents the average semantic deviation of the current batch of samples.

[0098] S44. Set the data sampling period T samp With the map update threshold δ upd As an adaptive parameter for the multimodal data fusion module and knowledge graph update mechanism.

[0099] In this embodiment, S5 includes the following steps:

[0100] S51. Based on the unified semantic vector representation set V emb With the adaptive parameter set {T samp ,δ upd Construct a multimodal semantic alignment mapping mechanism. For each multimodal data sample, based on the modality type s of the multimodal data sample... k Select the corresponding semantic alignment strategy and set the semantic vector of the semantic alignment strategy. Mapping to a shared semantic alignment space enables semantic features from different modalities to be aligned in the same vector space, forming a semantic alignment vector.

[0101] S52. Align the semantic vector Combined with the semantic activity of the corresponding samples and its KL divergence with the existing semantics of knowledge graphs In the input feature fusion mechanism, features are weighted and integrated based on the importance and novelty of multimodal semantic contributions, and the weights of each modality in the fused representation are dynamically adjusted to generate the fused data semantic representation. Construct the fused semantic representation set V fused .

[0102] In this embodiment, S6 includes the following steps:

[0103] S61. Combine the fused semantic representation set V fused As input, a predefined entity extraction mechanism is used to analyze the semantic representation of each data item. Based on the semantic features in the fused data representation, a candidate entity set E is identified and extracted. cand ;

[0104] S62. Combine the fused semantic representation V fused Same candidate entity set E cand As input, a relation recognition mechanism is used to detect semantic associations between candidate entities, identify and extract a candidate relation set R. cand ;

[0105] S63. Based on candidate entity set E cand With candidate relation set R cand Constructing a preliminary knowledge graph G init =(E init ,R init ),in For the initially determined set of entities, This is a preliminary set of relationships, and all entities and their relationships are constructed based on the fused data representation.

[0106] In this embodiment, S7 includes the following steps:

[0107] S71. The initially constructed knowledge graph G init The input structure optimization module constructs a graph structure optimization strategy set Π={π} based on an improved adaptive Poisson distribution reinforcement learning algorithm. θ}, where π θ This is a parameterized optimization strategy used to dynamically adjust the structure of nodes, edges, and their attributes in a knowledge graph.

[0108] S72. Define the state space S, action space A, and reward function R for graph optimization. The state space S represents the semantic confidence distribution of each node and relation in the current graph structure. The action space A includes node retention, edge update, attribute adjustment, and redundant removal structural operations. The reward function R is constructed based on semantic consistency score and structural sparsity.

[0109] S73. Semantic activity obtained from adaptive Poisson distribution modeling results With the map update threshold δ upd The action sampling probability and update frequency in the graph structure optimization are dynamically adjusted to make the reinforcement learning strategy time-sensitive and semantically driven.

[0110] S74. In each round of structural optimization, the optimization strategy π is adopted. θ Perform action a on the current graph structure t ∈A, and based on the current state s t ∈S to obtain reward r t =R(s) t ,a t The parameter θ is updated according to the policy gradient method, so that the structural adjustment process is continuously optimized.

[0111] S75. Repeat the structural optimization iterations until the convergence condition is met, and output the final optimized knowledge graph G. opt =(E opt ,R opt ).

[0112] In this embodiment, the final optimized knowledge graph construction satisfies:

[0113] Reserved nodes: semantic liveness Furthermore, it possesses structural connectivity with the core subgraph of the graph;

[0114] Remove nodes: semantic activity And there are no high-confidence related edges;

[0115] Edge retention: Both connected nodes meet the retention condition, and the semantic confidence of the edge is greater than or equal to the set edge confidence threshold;

[0116] Merging edges or attributes: Multiple semantically repetitive edges pointing to the same entity, or multiple redundant attribute values ​​are merged into a unified attribute representation after structural reduction.

[0117] Example 1:

[0118] At 9:35 a.m. on November 13, 2024, in the big data laboratory of the neurology department of a hospital in City A, a data acquisition server with the number "BTH-MOD1" started a new round of multimodal acquisition of neurological case data within the hospital. The goal was to construct a dynamic knowledge graph between the "neuropathy-clinical manifestation-treatment method" triplet to serve the hospital's intelligent consultation assistance system.

[0119] Within just two hours, the system collected 13,486 multimodal sample data entries, including 7,850 text medical records, 2,331 MRI scan images, 2,700 audio recordings, and 605 video clips from the live surgery system. The data was packaged by timestamp and sent to the "Unified Data Preprocessing Channel D-PIPE" for denoising, standardization, and modal synchronization labeling preprocessing.

[0120] At 11:46 AM, the system automatically flagged a data sample with significant semantic conflict:

[0121] Medical record number: TRH20241113-01095;

[0122] Time: 2024-11-13 10:24:31;

[0123] Modalities: text, image;

[0124] Text content: "The patient suddenly experienced left-sided limb paralysis within the past two days, and the preliminary diagnosis is brainstem hemorrhage;"

[0125] Image content: MRI image shows a high-density lesion in the right basal ganglia region, which is preliminarily identified as a hemorrhage.

[0126] The method of this invention fuses semantic vectors and finds that the KL divergence between the text semantic vector `[0.13,0.22,0.91,...,0.08]` and the image semantic vector `=[0.10,0.19,0.95,...,0.05]` is 0.18. The system automatically marks it as "intermodal semantic contradiction", and the semantic activity Λ value is 0.91, which is higher than the update threshold of 0.62. The system decides to retain this sample for subsequent fusion judgment of conflict knowledge.

[0127] At 15:12 that afternoon, the data fusion engine initiated an automatic graph update mechanism according to the set 3-hour interval. The fusion module mapped the sample semantic vectors to the shared space and calculated semantic weights. The new representation generated after semantic fusion for this conflicting sample, `[0.12,0.21,0.93,...,0.06]`, was identified as a "bleeding site conflicting sample" with high semantic confidence. The system, through reinforcement learning, determined that this sample was a false alarm of the actual condition, and a new relationship was added to the graph.

[0128] Entity 1: Brainstem hemorrhage

[0129] Entity 2: "Basal ganglia hemorrhage"

[0130] Relationship type: "Exclusion-False Alarm Conflict"

[0131] Time tag: 2024-11-13 10:24.

[0132] Such false alarm relationships were previously completely ignored in traditional graphs, and there was no error correction mechanism. The method of this invention is the first to actively model modal conflicts and annotate graph metadata, providing an important basis for subsequent intelligent diagnosis systems.

[0133] At 12:04 PM on November 15th, the project team simulated a clinical consultation process in the deployment test environment. The doctor entered the question: "The patient presents with sudden left-sided paralysis; in which areas is the bleeding occurring?" Traditional atlases only returned two candidate entities: "brainstem" and "thalamus." However, the atlas generated by this invention returned three candidates: "brainstem," "basal ganglia," and "thalamus," and automatically added confidence ranking and conflict warnings. The confidence results are as follows:

[0134] Brainstem: 0.91;

[0135] Basal section: 0.87 (semantic conflict marker);

[0136] Thalamus: 0.72;

[0137] Based on this, doctors can make more targeted judgments on imaging results and avoid misdiagnosis.

[0138] To further verify the system performance, the research team conducted a week-long A / B control experiment, using the traditional knowledge graph construction process (Text-GCN+EarlyFusion) and the method of this invention to build graphs and conduct question-answering tests on 200,000 multimodal neurology case data collected from 5 hospitals. The test data covered three modalities: text, image, and voice. The experimental scenarios included three indicators: accuracy evaluation of the question-answering system, efficiency evaluation of graph construction, and consistency evaluation of the graph.

[0139] The following is a summary of the comparative experimental data:

[0140]

[0141]

[0142] In one experiment, the knowledge triplet extracted by the traditional method from the record numbered "TRH20241115-05573" was:

[0143] Entity 1: "Thalamic hemorrhage";

[0144] Entity 2: "Speech disorder";

[0145] Relationship: Cause and effect;

[0146] The method of this invention identifies a more complete spectral structure:

[0147] Entity 1: "Thalamic hemorrhage";

[0148] Entity 2: "Insufficient blood supply to the frontal lobe";

[0149] Entity 3: "Speech disorder";

[0150] Relationship 1: "Trigger";

[0151] Relationship 2: "Secondary to";

[0152] Timeline: 2024-11-12 → 2024-11-14;

[0153] The atlas not only identifies more entities and relationships, but also accurately expresses the temporal evolution path, which is of great significance for clinical causal analysis.

[0154] As of November 30, 2024, a hospital's knowledge graph construction team had successfully generated 42 optimized knowledge graphs using the method of this invention, covering eight professional directions including neurology, respiratory medicine, and cardiology. These graphs were deployed in the hospital's intelligent diagnostic decision-making system. After going live, the system automatically recognized the clinical input statement "patient suddenly becomes confused and accompanied by cold hands and feet" and matched it with the entity "brainstem insufficiency" in the graph, with a confidence level of 0.88. This correctly prompted the doctor to prioritize checking the cerebral blood flow perfusion and avoid misdiagnosing it as "diabetic hypoglycemia".

[0155] This embodiment truly demonstrates the practical capabilities of the present invention in processing complex multimodal medical data. From semantic encoding and modality fusion to map optimization, each step provides quantitative data and clear scenarios, fully verifying the core advantages of this method over existing technical solutions in terms of semantic consistency, construction efficiency, and map quality.

[0156] This invention inputs heterogeneous modal data of text, images, audio, and video into a pre-trained large model, designs a multi-level semantic mapping function, realizes deep encoding of cross-modal semantic features, outputs a unified semantic vector representation, effectively captures contextual semantics and structural features in different modalities through a multi-level encoding structure, and significantly reduces semantic projection errors between modalities by using a unified semantic space alignment mechanism.

[0157] This invention designs a semantic activity function based on a unified semantic vector, which combines feature dynamic weights with the potential update value of KL divergence modeling samples. By optimizing Poisson modeling parameters, it dynamically generates data sampling period and map update threshold. It can adjust the data acquisition and map update frequency according to real-time data semantic changes, ensuring that the map construction process is semantically driven and time-sensitive.

[0158] This invention proposes a graph structure optimization algorithm based on adaptive Poisson distribution reinforcement learning. It establishes a dynamic reward function with semantic confidence and structural sparsity as the core, and designs a time-sensitive action sampling mechanism by combining the current semantic activity and update threshold. The optimization strategy can automatically identify and adjust redundant nodes, weakly related edges and attribute conflicts in the knowledge graph, forming an optimization path with good convergence and strong interpretability.

[0159] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A knowledge graph optimization method based on the fusion of large models and multimodal data, characterized in that, Includes the following steps: S1. Automatically collect various multimodal data, including text, images, audio, and video, through multiple channels to construct a multimodal dataset; S2. Preprocess the multimodal dataset to form a preprocessed multimodal dataset; S3. Utilize a pre-trained large model to perform multi-level feature extraction and semantic encoding on the pre-processed multimodal dataset to generate a unified semantic vector representation; S4. Input the unified semantic vector representation into the adaptive Poisson distribution model to dynamically model the arrival rate and distribution characteristics of multimodal data, and determine the adaptive parameters for data sampling and updating; S4 includes the following steps: S41. Represent the unified semantic vector representation set V emb Input the improved adaptive Poisson distribution model. The improved adaptive Poisson distribution model defines the semantic activity function Λ(·) to represent the k-th multimodal data sample at time t. k The potential impact strength on new entities and relations in the knowledge graph: in, For the k-th sample at time t k The semantic activity, where Λ0 is the baseline semantic activity, and β is the semantic activity. j (t k ) represents the j-th feature at time t k The structural weights are dynamically estimated by combining the rate of change of the corresponding entity or relation in the current knowledge graph. Let J be the projection function of the j-th semantic feature, and J be the dimension of the unified semantic vector. S42. Constructing a joint Poisson intensity function that integrates semantic novelty and structural heterogeneity based on the semantic activity function: in, Let θ represent the optimal Poisson modeling parameters, and t represent the set of Poisson modeling parameters to be optimized. k Let n be the timestamp of the data sample. k This represents the number of samples within the sampling window. The KL divergence between the current sample and the semantics of existing nodes in the knowledge graph reflects semantic novelty. γ is an adjustment parameter that controls the weight of the difference. This indicates that the k-th sample is at time t. k The semantic activity value under the set of Poisson modeling parameters to be optimized; S43. Based on the optimal Poisson modeling parameters Combined with the current knowledge graph structure state G t =(E t ,R t ) Calculate the data sampling period T samp With the map update threshold δ upd E t For entity sets, R t For a set of relations: in, For time t k The average semantic activity is given by ρ, which is a structure density adjustment factor used to adjust the sampling frequency based on relation density, and η is an update sensitivity factor. This represents the average semantic deviation of the current batch of samples. S44. Set the data sampling period T samp With the map update threshold δ upd As an adaptive parameter for the multimodal data fusion module and the knowledge graph update mechanism; S5. Based on unified semantic vector representation and adaptive parameters, a multimodal data fusion algorithm is used to perform semantic alignment and feature integration on each modality of data to generate a fused data representation set; S6. Utilize the fused data representation to perform entity extraction and relationship recognition, and construct a preliminary knowledge graph; S7. Implement a structure optimization algorithm based on adaptive Poisson distribution reinforcement learning on the preliminary knowledge graph to automatically verify the nodes, edges and their attributes of the knowledge graph, remove redundant information and adaptively adjust the structure to form an optimized knowledge graph; S7 includes the following steps: S71. The initially constructed knowledge graph G init The input structure optimization module constructs a graph structure optimization strategy set Π={π} based on an improved adaptive Poisson distribution reinforcement learning algorithm. θ }, where π θ To optimize strategies, this is used to dynamically adjust the structure of nodes, edges, and their attributes in the knowledge graph; S72. Define the state space S, action space A, and reward function R for graph optimization. The state space S represents the semantic confidence distribution of each node and relation in the current graph structure. The action space A includes node retention, edge update, attribute adjustment, and redundant removal structural operations. The reward function R is constructed based on semantic consistency score and structural sparsity. S73. Semantic activity obtained from adaptive Poisson distribution modeling results With the map update threshold δ upd The action sampling probability and update frequency in the graph structure optimization are dynamically adjusted to make the reinforcement learning strategy time-sensitive and semantically driven. S74. In each round of structural optimization, the optimization strategy π is adopted. θ Perform action a on the current graph structure t ∈A, and based on the current state s t ∈S to obtain reward r t =R(s) t ,a t The parameter θ is updated according to the policy gradient method, so that the structural adjustment process is continuously optimized. S75. Repeat the structural optimization iterations until the convergence condition is met, and output the final optimized knowledge graph G. opt =(E opt ,R opt ).

2. The knowledge graph optimization method based on the fusion of large models and multimodal data as described in claim 1, characterized in that, S1 includes the following steps: S11. Set the multimodal data acquisition channel set M = {m t ,m i ,m a ,m v }, where m t m i m a m v These represent the acquisition channels used to collect text, image, audio, and video data, respectively. Acquisition rules are configured based on heterogeneous information sources to construct a multimodal data stream. S12. For each modal data s k ∈M, a distributed task scheduling mechanism is used to trigger the acquisition module in parallel, and the data stream is collected in a polling manner according to the preset sampling period to form a time-series multimodal dataset; S13. Summarize all multimodal data acquisition results into the original dataset to generate a multimodal dataset D with a consistent structure. raw : Where, d k This represents the k-th multimodal data sample, where N is the total number of samples collected, and t k Indicates the data collection timestamp, v k Indicates the original data content, l k A logical identifier indicating the source of the data.

3. The knowledge graph optimization method based on the fusion of large models and multimodal data as described in claim 1, characterized in that, S2 includes the following steps: S21. For the multimodal dataset D raw Each multimodal data sample d in k Based on its modal data s k Each module calls the corresponding modality processing module to perform modality-related noise filtering operations, resulting in a denoised multimodal dataset. S22. The denoised multimodal dataset is processed according to the modal data s. k Mapping to a unified format set generates a format-standardized multimodal dataset; S23. Perform data cleaning on the data samples in the standardized multimodal dataset, removing data samples with invalid formats, missing fields, abnormal collection, and duplicate logical identifiers, to form a preprocessed multimodal dataset D. clean .

4. The knowledge graph optimization method based on the fusion of large models and multimodal data as described in claim 1, characterized in that, S3 includes the following steps: S31. Transfer the preprocessed multimodal dataset D clean Input a pre-trained large model, and define the feature extraction process of the pre-trained large model as a multi-level semantic mapping function F. emb Multi-level semantic mapping function F emb Composed of multiple feature coding layers It is constructed sequentially, where each feature encoding layer E l The semantic coding unit is based on a pre-trained large model. It extracts and encodes features from the input data feature vector to generate semantic feature representations at different levels. The outputs of all feature coding layers are connected to form a unified multi-level feature vector. S32. The unified multi-level feature vector is processed by dimensionality reduction and nonlinear mapping through the feature fusion unit of a pre-trained large model to generate a unified semantic vector representation of standard dimension, forming a unified semantic vector representation set V that can simultaneously represent the semantic information of multimodal data such as text, image, audio, and video. emb : in, Represents the k-th multimodal data sample d k After multi-level semantic mapping function F emb The semantic vector representation obtained afterwards.

5. The knowledge graph optimization method based on the fusion of large models and multimodal data as described in claim 1, characterized in that, S5 includes the following steps: S51. Based on the unified semantic vector representation set V emb With the adaptive parameter set {T samp ,δ upd Construct a multimodal semantic alignment mapping mechanism. For each multimodal data sample, based on the modal data s of the multimodal data sample... k Select the appropriate semantic alignment strategy and represent the semantic vector. Mapping to a shared semantic alignment space enables semantic features from different modalities to be aligned in the same vector space, forming a semantic alignment vector. S52. Align the semantic vector Combined with the semantic activity value of the corresponding sample and its KL divergence with the existing semantics of knowledge graphs In the input feature fusion mechanism, features are weighted and integrated based on the importance and novelty of multimodal semantic contributions, and the weights of each modality in the fused representation are dynamically adjusted to generate the fused data representation. Construct the fused data representation set V fused .

6. The knowledge graph optimization method based on the fusion of large models and multimodal data as described in claim 5, characterized in that, S6 includes the following steps: S61. Represent the fused data set V fused As input, a predefined entity extraction mechanism is used to analyze the semantic representation of each data item. Based on the semantic features in the fused data representation, a candidate entity set E is identified and extracted. cand ; S62. Represent the fused data set V fused Same candidate entity set E cand As input, a relation recognition mechanism is used to detect semantic associations between candidate entities, identify and extract a candidate relation set R. cand ; S63. Based on candidate entity set E cand With candidate relation set R cand Constructing a preliminary knowledge graph G init =(E init ,R init ),in For the initially determined set of entities, This is a preliminary set of relationships, and all entities and their relationships are constructed based on the fused data representation.

7. The knowledge graph optimization method based on the fusion of large models and multimodal data as described in claim 1, characterized in that, The final optimized knowledge graph construction satisfies: Reserved nodes: semantic liveness ≥Update threshold δ upd Furthermore, it has structural connectivity with the core subgraph of the graph; Remove nodes: semantic activity <Update threshold δ upd And there are no high-confidence related edges; Edge retention: Both connected nodes meet the retention condition, and the semantic confidence of the edge is greater than or equal to the set edge confidence threshold; Merging edges or attributes: Multiple semantically repetitive edges pointing to the same entity, or multiple redundant attribute values ​​are merged into a unified attribute representation after structural reduction.

Images