Multi-modal knowledge graph construction method and device based on large model and program product

Abstract

The invention discloses a multi-modal knowledge graph construction method and device based on a large model and a program product, belongs to the technical field of artificial intelligence and knowledge engineering crossing, and particularly relates to a knowledge graph dynamic construction and evolution method based on a large language model technology and multi-modal data processing. The problems of symbol grounding and semantic understanding obstacle in the prior art are solved. According to the method, innovation and breakthrough are realized in three dimensions of knowledge acquisition, representation and reasoning by fusing deep learning and knowledge engineering technologies. The multi-modal knowledge graph construction method and device based on the large model and the program product are applied to the field of multi-modal knowledge graph construction and are suitable for specific task scenes such as intelligent question and answer, decision support and semantic search.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of artificial intelligence and knowledge engineering, and particularly to a method for dynamically constructing and evolving knowledge graphs based on large language model technology and multi - modal data processing. Background Art

[0002] Knowledge graphs were initially represented in a pure symbolic way, that is, entities, concepts, and relationships mainly existed in text form. Although this representation is simple and clear, the single - modal information is difficult to reflect the complexity of the real world and cannot establish the connection between entity concepts and real entities. A multi - modal knowledge graph (MMKG) is a knowledge representation method that combines symbolic knowledge with multimedia data by introducing various modal information such as images, videos, and audios on the basis of traditional knowledge graphs. It can describe entities and relationships more comprehensively and accurately, and has characteristics such as diversity, complementarity, and interpretability. The multi - modal knowledge graph further introduces the way humans understand the world into machine learning and deep learning. It can not only capture the semantic relationships between entities and concepts, but also use non - text data to provide richer and more intuitive information expressions, which is an important means to enhance machine intelligence.

[0003] The multi - modalization of knowledge in a multi - modal knowledge graph can be carried out in two main ways: one is to regard multi - modal data as special attribute values of entities or concepts; the other is to directly treat multi - modal data as entities in the knowledge graph. Specifically:

[0004] (1) A - MMKG: In this representation, multi - modal data is treated as an attribute value of an entity. For example, an image, sound, or video file becomes part of the entity's "hasImage", "hasSound", etc. attributes. Such a design enables multi - modal data to directly participate in attribute triples. For example, in (s, "hasImage", o), o is a link pointing to an image resource;

[0005] (2) N - MMKG: In this way, multi - modal data itself is regarded as an entity and co - exists with other text - type entities. This allows for the establishment of more complex structures. For example, an image can have its own attributes and relationships with other entities, thus forming a more comprehensive knowledge network.

[0006] The current situation of multi - modal knowledge graph construction technology reflects the high integration of interdisciplinary research and technological innovation. Currently, the construction of multi - modal knowledge graphs not only relies on traditional symbolic representation methods, but also combines advanced technologies such as deep learning, computer vision, and natural language processing to effectively integrate various modal data such as text and images.

[0007] First, in terms of the understanding of multimodal data, researchers have developed a variety of self-supervised pre-training tasks to enhance the learning ability of models, such as masked language models, sentence-image alignment, region label classification, region feature regression, and object prediction. These tasks help improve fine-grained cross-modal understanding, enabling the model to more accurately capture semantic associations between different modalities. In addition, to improve the effect of cross-modal alignment, some works have introduced cross-modal object alignment mechanisms, further strengthening the mutual understanding and expression between text and images. For example, the Vision-and-Language Transformer (ViLT) model does not rely on convolutional operations or region supervision, but effectively encodes visual and language information through self-attention mechanisms.

[0008] Second, regarding the construction efficiency problem of large-scale multimodal knowledge graphs, existing solutions face scalability challenges. Due to the complexity of multimedia data processing, the time cost required to build a multimodal knowledge graph containing billions of instances is huge. For example, the NEIL project took approximately 350K CPU hours to collect 400,000 visual instances. To address this issue, researchers are exploring more efficient algorithms and technologies, including but not limited to the application of distributed computing frameworks and special optimization strategies for complex media types such as videos.

[0009] Furthermore, in terms of quality control, it is necessary to consider how to ensure that the knowledge incorporated in the construction process of multimodal knowledge graphs is accurate and representative. This involves multiple aspects such as entity recognition, relation extraction, and event classification, and problems such as the misuse of famous entity pictures and the difficulty of visualizing abstract concepts need to be solved. To this end, many systems have adopted a series of means such as semi-supervised classification, object detection, fine-grained classification, and heuristic rules to ensure the quality of knowledge.

[0010] Currently, the construction technology of multimodal knowledge graphs mainly faces the above three bottlenecks. In addition, there are also detailed problems such as high manual dependence and lagging updates. It can be foreseen that future research will continue to focus on three core directions: improving construction efficiency, enhancing cross-modal understanding ability, and ensuring knowledge accuracy. Summary of the Invention

[0011] The object of the present invention is to solve the symbol grounding problem and semantic understanding obstacles in the construction of multimodal knowledge graphs, establish an accurate and reliable connection between abstract knowledge symbols and specific multimodal data, and solve the problems of low construction efficiency, poor cross-modal understanding ability, low knowledge accuracy, high manual dependence, and lagging updates in the prior art, further improving the precision and efficiency of multimodal knowledge graph construction, and providing a multimodal knowledge graph construction method, device, and program product based on large models.

[0012] The multi-modal knowledge graph construction method based on a large model according to the present invention, the method comprising the following steps:

[0013] Step S1: Collect multi-modal data and perform preprocessing operations on it; the preprocessing operations include cleaning, deduplication, filtering, classification, entity alignment, and annotation;

[0014] Step S2: Train a cross-modal encoder based on the Transformer architecture and build a multi-modal perception layer; use the multi-modal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the preprocessed multi-modal data to obtain an aligned multi-modal embedding matrix;

[0015] Step S3: Implement knowledge extraction guided by a hybrid attention mechanism and build a knowledge distillation layer; use the knowledge distillation layer to perform dual-channel knowledge extraction and knowledge fusion and verification on the aligned multi-modal embedding matrix to obtain a set of knowledge cells;

[0016] Step S4: Build a dynamic knowledge graph layer through incremental dynamic update and closed-loop feedback optimization of the inference module; the dynamic knowledge graph layer obtains a dynamically updated multi-modal knowledge graph according to the input set of knowledge cells.

[0017] Further, a preferred implementation is provided. In step S1, the multi-modal data is collected as follows:

[0018] Collect multi-modal data from open-source multi-modal datasets.

[0019] Further, a preferred implementation is provided. In step S1, the annotation of the preprocessing operation is as follows:

[0020] For incomplete entities lacking multi-modal information in the multi-modal data, the Flamingo vision-language large model is used for annotation.

[0021] Further, a preferred implementation is provided. In step S2, using the multi-modal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the preprocessed multi-modal data to obtain an aligned multi-modal embedding matrix includes:

[0022] Step S2.1: Perform modality-specific encoding on the preprocessed multi-modal data to obtain feature vectors of different modality data:

[0023] For text data, hierarchical BERT model encoding is used to obtain dynamic word vectors as the feature vectors of the text data;

[0024] For image data, the spatial attention network of the CLIP large model is used to extract regional features as the feature vectors of the image data;

[0025] For video data, TimeSformer is used to capture action semantics as the feature vector of the video data;

[0026] Step S2.2: Project the feature vectors of different modalities into a unified feature vector space through optimal transport projection;

[0027] Step S2.3: Adopt contrastive learning with the InfoNCE loss function to perform multi-modal semantic alignment in the unified feature vector space, and obtain the aligned multi-modal embedding matrix.

[0028] Furthermore, a preferred implementation is provided. In step S3, the knowledge distillation layer is used to perform dual-channel knowledge extraction, knowledge fusion and verification on the aligned multi-modal embedding matrix, and obtain a knowledge cell set including:

[0029] Step S3.1: The knowledge distillation layer includes a dual-channel collaborative extraction network; the dual-channel collaborative extraction network includes channel A and channel B, where channel A is a neural extraction channel and channel B is a symbolic reasoning channel;

[0030] Use channel A and channel B to perform knowledge extraction on the aligned multi-modal embedding matrix respectively;

[0031] Channel A uses dynamically generated entity recognition Prompts to call a large model to extract knowledge;

[0032] Channel B uses an expert model to build a rule trigger matrix and perform relational graph inference propagation on a graph neural network;

[0033] Step S3.2: Perform fusion and verification on the knowledge obtained after knowledge extraction by channel A and channel B:

[0034] A knowledge distillation bridge is set between channel A and channel B;

[0035] By calculating the prediction confidence, introduce an uncertainty calibration module and add Gaussian noise to the gradient to introduce differential privacy protection measures;

[0036] Perform causal counterfactual argumentation to ensure the rationality of the dual-channel collaborative extraction network, and obtain a knowledge cell set.

[0037] Furthermore, a preferred implementation is provided. In step S4, through the incremental dynamic update and closed-loop feedback optimization of the inference module, a dynamic knowledge graph layer is built, including:

[0038] Step S4.1: Incremental dynamic update:

[0039] Input the knowledge cell set into the dynamic knowledge graph layer, calculate the survival weight of the knowledge unit and construct a dynamic adjacency matrix;

[0040] Reinforcement learning through importance-aware replay;

[0041] Perform online knowledge distillation;

[0042] Step S4.2: Construct a closed-loop feedback optimization channel:

[0043] Collect feedback signals in the real-time data stream, calculate the feedback loss and backpropagate it along the computational graph to the cross-modal encoder of the multi-modal perception layer to update the weights of the cross-modal encoder.

[0044] The present invention also proposes a multi-modal knowledge graph construction device based on a large model, and the device includes the following modules:

[0045] Module S1: Collect multi-modal data and perform preprocessing operations on it; the preprocessing operations include cleaning, deduplication, filtering, classification, entity alignment, and annotation;

[0046] Module S2: Train a cross-modal encoder based on the Transformer architecture and build a multi-modal perception layer; use the multi-modal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the multi-modal data after the preprocessing operation to obtain an aligned multi-modal embedding matrix;

[0047] Module S3: Implement knowledge extraction guided by a hybrid attention mechanism and build a knowledge distillation layer; use the knowledge distillation layer to perform two-channel knowledge extraction and knowledge fusion and verification on the aligned multi-modal embedding matrix to obtain a knowledge cell set;

[0048] Module S4: Build a dynamic knowledge graph layer through incremental dynamic update and closed-loop feedback optimization of the inference module; the dynamic knowledge graph layer obtains a dynamically updated multi-modal knowledge graph according to the input knowledge cell set.

[0049] The present invention also proposes a computer device, including: a processor and a memory, the memory is used to store the executable instructions of the processor, and the processor is configured to execute the above-mentioned multi-modal knowledge graph construction method based on a large model by executing the executable instructions.

[0050] The present invention also proposes a computer storage medium, in which a computer program is stored, and when the computer program runs, it executes the above-mentioned multi-modal knowledge graph construction method based on a large model.

[0051] The present invention also proposes a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above-mentioned multi-modal knowledge graph construction method based on a large model are implemented.

[0052] The present invention has the following beneficial effects:

[0053] 1. The method for constructing a multimodal knowledge graph based on a large model according to the present invention effectively solves the semantic gap problem of multimodal data and significantly improves the cross-modal semantic understanding ability through an optimal transport projection and contrastive learning joint optimization framework.

[0054] 2. The method for constructing a multimodal knowledge graph based on a large model according to the present invention constructs a neural-symbol collaborative dual-channel extraction architecture, reduces the incidence of knowledge conflicts through a dual-channel uncertainty calibration module, improves the knowledge acquisition process, and breaks through the artificial dependence bottleneck of traditional methods.

[0055] 3. The method for constructing a multimodal knowledge graph based on a large model according to the present invention builds a spatio-temporal coupled dynamic update model for the knowledge graph, solves the problem of lagging knowledge update through incremental update, realizes the dynamic evolution of the knowledge graph, and controls the incidence of catastrophic forgetting through an importance replay mechanism to ensure the stability of continuous learning of the system.

[0056] 4. The method for constructing a multimodal knowledge graph based on a large model according to the present invention introduces a large model into the multimodal knowledge graph construction task, uses the large model to provide symbolic information about unseen objects, or establishes semantic relationships between known and unknown objects to help the model better understand and interpret new visual content. The rich knowledge, powerful reasoning ability, and versatility in dealing with multiple natural language processing tasks of the large model enable the multimodal knowledge graph in the construction process to more easily learn implicit knowledge represented by background knowledge, long-tail knowledge, and fine-grained knowledge, which are lacking in the prior art. For example, using the external symbolic knowledge provided by the large model to enhance the matching ability between images and texts enables the system to not only rely on statistical patterns but also understand and process rare or unseen objects. This is particularly important for learning knowledge related to long-tail entities that are rare in the training data.

[0057] The method, device, and program product for constructing a multimodal knowledge graph based on a large model according to the present invention are applied to the field of multimodal knowledge graph construction and are suitable for specific task scenarios such as intelligent question answering, decision support, and semantic search. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0059] Figure 1In an embodiment of the present invention, it is an architecture flowchart of a multi-modal knowledge graph construction method based on a large model. Detailed implementation manners

[0060] To make the technical solutions and advantages of the present invention more clearly expressed, the following will further describe in detail and completely the specific implementation manners of the present invention in conjunction with the accompanying drawings. The following described implementation manners are only some preferred solutions of the present invention, rather than all implementation solutions; the following described implementation manners are intended to explain the present invention and should not be construed as a limitation to the present invention; the reasonable combination of the technical features defined in each implementation manner of the present invention, and all other implementation manners obtained by those of ordinary skill in the art based on the implementation manners of the present invention without creative efforts belong to the scope of protection of the present invention.

[0061] In one embodiment, a multi-modal knowledge graph construction method based on a large model is provided, and the method includes the following steps:

[0062] Step S1: Collect multi-modal data and perform preprocessing operations on it; the preprocessing operations include cleaning, deduplication, filtering, classification, entity alignment, and annotation;

[0063] Step S2: Train a cross-modal encoder based on the Transformer architecture and build a multi-modal perception layer; use the multi-modal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the multi-modal data after preprocessing operations to obtain an aligned multi-modal embedding matrix;

[0064] Step S3: Implement knowledge extraction guided by a hybrid attention mechanism and build a knowledge distillation layer; use the knowledge distillation layer to perform dual-channel knowledge extraction and knowledge fusion and verification on the aligned multi-modal embedding matrix to obtain a set of knowledge cells;

[0065] Step S4: Build a dynamic knowledge graph layer through incremental dynamic update and closed-loop feedback optimization of the inference module; the dynamic knowledge graph layer obtains a dynamically updated multi-modal knowledge graph according to the input set of knowledge cells.

[0066] In this embodiment, a multi-modal knowledge graph construction method based on a large model is proposed to solve the symbol grounding problem in multi-modal knowledge graph construction, that is, to establish an accurate and reliable connection between abstract knowledge symbols and specific multi-modal data, and the semantic understanding obstacle of multi-modal data.

[0067] In this embodiment, the multi-modal knowledge graph construction method based on a large model has achieved innovative breakthroughs in three dimensions of knowledge acquisition, representation, and reasoning by integrating deep learning and knowledge engineering technologies.

[0068] In this embodiment, the preprocessing operations include:

[0069] Cleaning: Removing noise, errors, or redundant data. For example, deleting low-resolution images, correcting spelling mistakes in text, and filtering out invalid frames in videos.

[0070] Deduplication: Eliminating duplicate data. For example, when the same entity appears multiple times in different datasets, only retaining the unique instance.

[0071] Filtering: Removing irrelevant or low-quality data. For example, excluding text unrelated to the target domain or blurry pictures.

[0072] Classification: Organizing data into categories. For example, classifying medical images and natural scene images into different categories.

[0073] Entity Alignment: Identifying and linking records that point to the same real-world entity in different datasets, that is, aligning entities with the same meaning but different references in different datasets (such as "name", "first name", and "person's name"). For example, "Diabetes" in MedTrinity-25M and "Diabetes" in LAION may refer to the same medical concept and need to be merged into a single entity. Entity alignment is completed by performing cross-dataset entity recognition and linking tasks.

[0074] Annotation: That is, annotating incomplete data and supplementing labels for data lacking some information. Some datasets may lack multimodal information. For example, pure image or video datasets may have little annotated text content, so text modality information needs to be supplemented; or for pure text data, pictures or videos, etc. need to be supplemented.

[0075] In this embodiment, the method decomposes the multimodal knowledge graph construction task into several steps: data acquisition and preprocessing, multimodal fusion and knowledge extraction, and constructing a dynamic multimodal knowledge graph for implementation. During the multimodal fusion and knowledge extraction process, a neural-symbolic joint reasoning engine and a knowledge conflict self-checking and correction mechanism are introduced, and a two-way information flow control mechanism of forward knowledge distillation and reverse semantic correction is constructed throughout the process. Through these innovations, the first multimodal, fully automatic, and real-time evolving knowledge graph construction framework based on large models is provided.

[0076] In this embodiment, the set of knowledge cells output by the knowledge distillation layer is processed by the dynamic knowledge graph layer K, and then a dynamically updated multimodal knowledge graph G is output. Due to the incremental update strategy and the closed-loop feedback optimization mechanism, the graph G can be dynamically updated.

[0077] In this embodiment, all parameters in the entire model of the method are optimized through closed-loop feedback optimization, that is, the model parameters are updated through feedback information transmitted by real-time data streams in reverse transmission to obtain optimized model parameters θ.

[0078] In this embodiment, in actual applications, the graph G is submitted to the upper application layer in the output of the dynamic graph layer, and the model parameter θ is fed back to the model components such as the downstream encoder to optimize the model effect.

[0079] In addition, in one embodiment, in step S1, the multimodal data is collected as follows:

[0080] Collect multimodal data from open source multimodal datasets.

[0081] In addition, in one embodiment, the open source multimodal datasets include MedTrinity-25M, OpenVid-1M, Touch100k, Video-MME, MultiOOD, LAION, OmniCorpus, and ShareGPT4V.

[0082] In this embodiment, the MedTrinity-25M, OpenVid-1M, Touch100k, Video-MME, MultiOOD, LAION, OmniCorpus and ShareGPT4V are all high-quality open source multimodal datasets. These datasets are of high quality, large scale, and cover a wide range of fields, and are suitable for multimodal knowledge graph construction tasks.

[0083] In this implementation, different open source multimodal datasets need to be cleaned, deduplicated, filtered, and other operations performed separately. Because different datasets have different defined standards, they usually have unique structures and quality issues and need to be processed in a targeted manner:

[0084] Format differences: For example, a text dataset may be stored in CSV format, while an image dataset may be stored in JPEG format.

[0085] Quality issues: Some datasets may be noisier and need to be cleaned separately.

[0086] Domain differences: For example, a medical dataset (MedTrinity-25M) requires specialized terminology processing, while a general dataset (LAION) requires general cleaning.

[0087] Uniformity of objectives: Individual processing ensures the internal consistency of each dataset, laying the foundation for subsequent cross-dataset classification, entity alignment, and labeling.

[0088] In addition, in one embodiment, in the step S1, the preprocessing operation is labeled as follows:

[0089] For incomplete entities lacking multimodal information in multimodal data, the Flamingo vision-language large model is used for annotation.

[0090] In this embodiment, incomplete entities include, for example, text data lacking picture descriptions.

[0091] In addition, in one embodiment, the Flamingo vision-language large model uses an interleaved text and image method for annotation.

[0092] In this embodiment, the interleaved text and image method alternates visual data and text data, making it easier for the model (Flamingo vision-language large model) to learn the association between entities and the real world, thereby achieving a high-quality annotation effect.

[0093] In this embodiment, the architecture design of the Flamingo vision-language large model integrates a pre-trained visual encoder and a large language model (LLM, or large-scale language model, or simply large model), and introduces a Perceiver Resampler and Gated Cross-Attention Layers to achieve effective integration of multimodal data (or multimodal information).

[0094] Each Gated Cross-Attention Layer contains a Cross-Attention Mechanism, a gating structure, and a Feed-Forward Network.

[0095] The Cross-Attention Mechanism allows the LLM to query the visual tokens generated by the Perceiver Resampler to obtain specific details about the input image.

[0096] The gating structure controls the degree of influence of the newly added layer on the behavior of the original LLM, ensuring that it does not deviate too far from the original LLM during the initial training stage of the model.

[0097] The Feed-Forward Network performs a non-linear transformation on the features output by the Cross-Attention Mechanism, enhancing the expressive power of the model, and adjusting the representation spaces of visual and text features to adapt to the generation requirements of the language model.

[0098] As the model training progresses, the gating value gradually increases, allowing more visual semantics to be incorporated into the text generation process.

[0099] The Flamingo vision-language large model cleverly combines high-quality visual encoding technology, innovative architecture design, and powerful few-shot learning capabilities, making it very effective in multi-modal annotation tasks.

[0100] The annotation data of the Flamingo vision-language large model is achieved through Few-shot Prompt.

[0101] In addition, in one embodiment, in step S2, the multi-modal perception layer is used to perform modality-specific encoding and cross-modal semantic alignment on the preprocessed multi-modal data to obtain the aligned multi-modal embedding matrix, including:

[0102] Step S2.1: Perform modality-specific encoding on the preprocessed multi-modal data to obtain feature vectors of different modality data:

[0103] For text data, the hierarchical BERT model is used for encoding to obtain dynamic word vectors as the feature vectors of the text data;

[0104] For image data, the spatial attention network of the CLIP large model is used to extract regional features as the feature vectors of the image data;

[0105] For video data, TimeSformer is used to capture action semantics as the feature vectors of the video data;

[0106] Step S2.2: Project the feature vectors of different modalities into a unified feature vector space through optimal transport projection;

[0107] Step S2.3: Use contrastive learning with the InfoNCE loss function to perform multi-modal semantic alignment in the unified feature vector space to obtain the aligned multi-modal embedding matrix.

[0108] In this embodiment, in the contrastive learning using the InfoNCE loss function, the multi-modal perception layer and the knowledge distillation layer share the contrastive learning memory bank.

[0109] In this embodiment, let the preprocessed multi-modal data be:

[0110] D = {T, I, V};

[0111] where T is text data; I is image data; V is video data.

[0112] In addition, in one embodiment, for text data, the hierarchical BERT model is used for encoding to obtain dynamic word vectors, which is described by the following formula:

[0113] f t [i]= BERT(T)[CLS].

[0114] Among them, t (text) represents the text modality, i represents the hierarchical index (i.e., i = 1 represents word-level features, i = 2 represents sentence-level features, i = 3 represents discourse-level features), and CLS represents the classification token.

[0115] CLS is a placeholder of the BERT model, located at the head of the input sequence, and its output vector captures global semantic information.

[0116] In addition, in one embodiment, the hierarchical BERT model is an improvement of the BERT model. The improvement lies in: first encoding the local text separately, and then combining the encodings of the local text.

[0117] It should be noted that due to the input length limitation, the traditional BERT model has limited ability in processing long texts. In this embodiment, the hierarchical BERT model that encodes the local text separately and then combines it improves the long text processing ability, improves the computing efficiency, enhances the context understanding, and can capture more multi-level structure information.

[0118] In addition, in one embodiment, for image data, the spatial attention network of the CLIP large model is used to extract region features, which is described by the following formula:

[0119]

[0120] Among them, K represents the total number of ROI regions, ROI k represents the k-th ROI region, i (image) represents the image modality, ROI (Region of Interest) represents the region of interest, and α represents the attention weight corresponding to the ROI, which reflects the importance of this region for the current task.

[0121] In this embodiment, the CLIP large model is trained on a large number of image-text pairs, so it can understand and associate visual and language information. Compared with traditional neural networks such as CNN, using the spatial attention network of the CLIP large model can more effectively identify and extract the region in the image that is most relevant to a specific text description, and enhance the understanding of the image content.

[0122] In addition, in one embodiment, for video data, the temporal slice TimeSformer is used to capture action semantics, which is described by the following formula:

[0123]

[0124] Among them, v (video) represents the video modality, V tDenote the t-th frame of the video, and T represents the total number of frames of the video.

[0125] In this embodiment, the TimeSformer, i.e., the Temporal Slice Transformer, is used to capture the action semantics in the video.

[0126] In this embodiment, the TimeSformer innovatively applies the Transformer architecture to video understanding tasks. By simultaneously considering information in both the temporal and spatial dimensions (i.e., spatio-temporal joint modeling), it can more accurately capture the details and context of actions in the video. Using the TimeSformer to capture the action semantics in the video allows for the effective identification of temporal changes and action progress without losing spatial information. Using the TimeSformer for spatio-temporal joint modeling enables the model to learn long-range dependencies, improving its flexibility and scalability.

[0127] In addition, in one embodiment, the feature vectors of different modalities are projected into a unified feature vector space through optimal transport projection, which is described by the following formula:

[0128]

[0129] where, represents the feature vector after being projected into the unified feature vector space (or called the unified semantic space), W m represents the projection weight, f m represents the feature vector of the original modality, b m represents the projection bias, and m ∈ represents the type of the original modality (i.e., whether the modality of the original feature vector is text T, image I, or video V).

[0130] In this embodiment, let the (semantically) aligned multi-modal embedding matrix be E, then E ∈ R N×d .

[0131] In this embodiment, by mapping data of different modalities into the same vector space, information from different modalities can be directly compared and correlated, thereby enhancing the model's ability to understand cross-modal content.

[0132] In this embodiment, the InfoNCE loss function is used to maximize the similarity between positive sample pairs while minimizing the similarity between negative sample pairs. This helps the model learn more accurate cross-modal representations, improving accuracy and robustness.

[0133] In this embodiment, the formula of the InfoNCE loss function is as follows:

[0134]

[0135] Among them, s represents the similarity coefficient, and τ represents the temperature coefficient. and represent positive sample pairs from different modalities, and represent negative sample pairs from different modalities.

[0136] The temperature coefficient is used to control the sharpness of the similarity distribution.

[0137] It should be noted that semantic alignment and entity alignment have different meanings. Semantic alignment is to map the feature representations of different modalities (such as text, image, video) to a unified vector space, so that the multi-modal data of the same entity is semantically consistent, that is, to align the feature vectors with the same meaning in different modalities in the unified projection space. For example, the vector representations of the text "tiger" and the tiger picture should be close.

[0138] In addition, in one embodiment, in step S3, the knowledge distillation layer is used to perform two-channel knowledge extraction, knowledge fusion and verification on the aligned multi-modal embedding matrix, and the obtained knowledge cell set includes:

[0139] Step S3.1: The knowledge distillation layer includes a two-channel collaborative extraction network; the two-channel collaborative extraction network includes channel A and channel B, where channel A is a neural extraction channel and channel B is a symbolic reasoning channel;

[0140] Use channel A and channel B to perform knowledge extraction on the aligned multi-modal embedding matrix respectively;

[0141] Channel A uses a dynamically generated entity recognition Prompt to call a large model to extract knowledge;

[0142] Channel B uses an expert model to build a rule trigger matrix and perform relational graph reasoning propagation on the graph neural network;

[0143] Step S3.2: Perform fusion and verification on the knowledge obtained by performing knowledge extraction on channel A and channel B:

[0144] There is a knowledge distillation bridge between channel A and channel B;

[0145] By calculating the prediction confidence, introduce an uncertainty calibration module, and add Gaussian noise to the gradient to introduce differential privacy protection measures;

[0146] Perform causal counterfactual argumentation to ensure the rationality of the two-channel collaborative extraction network and obtain the knowledge cell set.

[0147] In this embodiment, the knowledge distillation bridge is essentially a part of the two-channel collaborative mechanism and is used to fuse the two-channel reasoning results.

[0148] In this embodiment, calculating the prediction confidence is for subsequent dual-channel information fusion.

[0149] In this embodiment, differential privacy protection perturbs the gradient information through a noise injection mechanism to ensure that the training process meets the DP constraint. The reason for performing this step is that in knowledge distillation, the gradient may disclose sensitive information of the training data.

[0150] It should be noted that the causal effect (CE) is the core indicator in causal inference for quantifying the impact intensity of a certain intervention (or cause) on the outcome variable. Performing causal counterfactual reasoning is to determine whether the result given by the model (here refers to the dual-channel collaborative extraction network) is a correlation or a causal relationship.

[0151] In this embodiment, by designing a dual-channel collaborative extraction network and introducing differential privacy protection measures, an innovative upgrade of the knowledge extraction mechanism is achieved.

[0152] In this embodiment, the dual-channel collaborative extraction network is a neural-symbolic collaborative dual-channel knowledge extraction architecture, which realizes the organic combination of generative extraction and structured reasoning.

[0153] In this embodiment, Channel A (neural channel) performs dynamic entity recognition, and Channel B (symbolic channel) performs relational graph reasoning. The knowledge complementarity of the dual channels enables the model to better adapt to different modalities and achieve an efficiency balance at the same time.

[0154] In this embodiment, Prompt, that is, the prompt word or command, is used to control the (language) large model to execute tasks.

[0155] In this embodiment, in the prediction confidence, the knowledge distillation layer and the dynamic knowledge graph layer jointly optimize the knowledge confidence.

[0156] In this embodiment, let the obtained knowledge cell set be:

[0157] K = {(e i , r k , e j , c)}

[0158] Among them, e i represents the background event or initial condition, r k represents the k-th applied intervention or cause variable, e j represents the result event, and c (Causal Effect, CE) represents the calculated value of the causal effect.

[0159] The above parameters are related to the formulas in the causal counterfactual reasoning part.

[0160] In addition, in one embodiment, Channel A utilizes dynamically generated entity recognition Prompts to call a large model to extract knowledge:

[0161] The dynamic Prompt generation mechanism of Channel A automatically generates an optimal Prompt template according to the characteristics of the input data:

[0162] For the input multi-modal feature vector h m , context-aware encoding is performed, key features are extracted through the attention mechanism, and it is described by the following formula:

[0163]

[0164] c = ∑α i H p [i]

[0165] where H p is a predefined Prompt prototype library; α i represents the attention weight of the i-th Prompt template, W q and W k represent projection matrices with learnable parameters, h m represents the current input multi-modal feature vector, c represents the generated feature context vector; the T on h m is the transpose symbol.

[0166] Prompt dynamic synthesis, generating a Prompt according to the feature context c:

[0167] prompt = [CLS] ⊕ φ1 ⊕ [ENT] ⊕ φ2 ⊕ [SEP]

[0168] where φ1 represents the entity type description, φ2 represents the constraint condition; CLS represents the classification token, ENT represents the entity placeholder, and SEP represents the separator token;

[0169] After obtaining the generated Prompt, knowledge extraction is performed through a large model.

[0170] It should be noted that CLS, ENT, and SEP themselves are all placeholder types of things, which are the unified understanding of existing technologies in the field of natural language processing and are not model parameters that need to be learned.

[0171] In this embodiment, the dynamic Prompt generation mechanism of Channel A automatically generates an optimal Prompt template according to the characteristics of the input data, solving the domain limitation of the traditional fixed template.

[0172] In this embodiment, after obtaining the Prompt, a language large model such as Qwen can be used for knowledge extraction.

[0173] In addition, in one embodiment, Channel B constructs a rule trigger matrix using an expert model and performs relational graph inference propagation on a graph neural network;

[0174] Channel B constructs a rule trigger matrix using an expert model:

[0175] Let the set of rules given by the expert model be R = {r1, r2,..., r R}, where each rule r k describes the trigger conditions for a specific relationship type;

[0176] For each rule r k , construct a binary adjacency matrix A (k) ∈ {0, 1} N×N , and stipulate that when entities e i and e j satisfy the trigger conditions of rule r k , is 1, otherwise it is 0;

[0177] Perform relational graph inference propagation on a graph neural network, which is described by the following formula:

[0178]

[0179] Among them, H (l) represents the node feature matrix of the l-th layer, σ represents the non-linear activation function, R represents the total number of predefined relationship types, A (k) represents the adjacency matrix of relationship k, and W (k,l) represents the learnable parameter matrix of relationship k at the l-th layer.

[0180] In addition, in one embodiment, a knowledge distillation bridge is set between Channel A and Channel B, which is described by the following formula:

[0181]

[0182] Among them, c(x) represents the confidence score (i.e., the calculated prediction confidence), T represents the number of sampling times, p t (x) represents the probability distribution of the t-th sampling, represents the average prediction probability distribution.

[0183] T: Repeat the experiment multiple times to improve the prediction stability.

[0184] In addition, in one embodiment, by calculating the prediction confidence, an uncertainty calibration module is introduced, which is described by the following formula:

[0185]

[0186] Among them, c ARepresents the confidence of channel A, y A Represents the output result of channel A; c B Represents the confidence of channel B, y B Represents the output result of channel B.

[0187] In addition, in one embodiment, Gaussian noise is added to the gradient to introduce differential privacy protection measures, which are described by the following formula:

[0188]

[0189] Wherein, Represents the gradient after adding noise, B (Batch size) represents the batch size, Represents the gradient of the i-th sample, N represents Gaussian noise with a mean of 0 and a variance of σ 2 Δ 2 And σ represents the noise scaling factor, and Δ represents the sensitivity of the gradient.

[0190] In addition, in one embodiment, causal counterfactual reasoning is performed to ensure the rationality of the dual-channel collaborative extraction network, which is described by the following formula:

[0191]

[0192] Wherein, CE (Causal Effect) represents the causal effect, e i Represents the background event or initial condition, r represents the applied intervention or causal variable, e j Represents the result event, P(e j |do(r)) represents the probability of e i occurring under the intervention r, Represents the probability of e j occurring without the intervention r.

[0193] In this embodiment, here is to calculate the causal relationship CE, that is, c (Causal Effect, CE, that is, the value of the calculated causal effect) in the set of knowledge cells K output by the dual-channel collaborative extraction network.

[0194] In this embodiment, CE is the core index used to quantify the influence intensity of a certain intervention or cause on the result variable in causal inference.

[0195] In addition, in one embodiment, in step S4, through the incremental dynamic update and closed-loop feedback optimization of the inference module, building the dynamic knowledge graph layer includes:

[0196] Step S4.1: Incremental dynamic update:

[0197] Input the set of knowledge cells into the dynamic knowledge graph layer, calculate the survival weights of knowledge units, and construct a dynamic adjacency matrix;

[0198] Perform reinforcement learning through importance-aware replay;

[0199] Execute online knowledge distillation;

[0200] Step S4.2: Construct a closed-loop feedback optimization channel:

[0201] Collect feedback signals in the real-time data stream, calculate the feedback loss, and backpropagate it along the computational graph to the cross-modal encoder in the multi-modal perception layer to update the weights of the cross-modal encoder.

[0202] In this embodiment, the feedback signals in the real-time data stream, such as annotating or rejecting a certain inference result. According to the feedback signals, calculate the feedback loss and backpropagate it along the computational graph to the feature encoder (cross-modal encoder) to update the weights of the encoder.

[0203] In this embodiment, the dynamic knowledge graph layer is a spatio-temporal coupled knowledge evolution framework, which solves the problem of updating graph knowledge.

[0204] In this embodiment, the cross-modal encoder in the multi-modal perception layer is used for modality-specific encoding, and through closed-loop feedback optimization, the feedback drives the fine-tuning of the encoder.

[0205] In addition, in one embodiment, the survival weight of the knowledge unit is calculated and described by the following formula:

[0206]

[0207] where λ t is the time decay coefficient, used to control the speed of old knowledge elimination; N(k) is the set of direct neighbors of knowledge unit k in the knowledge graph; Δt represents the time difference between the current timestamp and the last update time of knowledge unit k; sim(k,j) represents the similarity between knowledge units k and j.

[0208] In addition, in one embodiment, construct a dynamic adjacency matrix:

[0209] If sim(e i , e j ) ≤ 0, define A i,j = 0, otherwise define A i,j = w i w j sim(e i , e j ).

[0210] where e represents an entity, such as e i represents entity i, ej j represents the entity; w represents the weight; A is the constructed dynamic adjacency matrix, and i, j refer to the corresponding positions.

[0211] In addition, in one embodiment, reinforcement learning is performed through importance-aware replay, and key samples are retained in the replay buffer according to the importance ratio. The formula for sample importance is as follows:

[0212]

[0213] where x represents the newly input sample, represents taking the gradient, and loss(x) represents the loss function.

[0214] The meaning of this formula is: for the newly input sample x, the importance score p replay (x) and the loss function loss(x) are positively correlated with the two-norm of the gradient of the model parameters in terms of.

[0215] In addition, in one embodiment, online knowledge distillation is performed:

[0216]

[0217] where T is the temperature coefficient; θ new represents the updated model parameters, L distill represents the loss function of online distillation, represents taking the gradient; α represents the weight; KL represents the divergence, which is used to measure the difference between two probability distributions; z T represents the logits vector output by the Teacher model, and z S represents the logits vector output by the Student model;

[0218] The updated gradient is backpropagated to the projection layer in a closed-loop manner to achieve closed-loop feedback optimization.

[0219] In addition, in one embodiment, the construction of the closed-loop feedback optimization channel: collect the feedback signal in the real-time data stream, calculate the feedback loss and backpropagate it along the computational graph to the feature encoder to update the weights of the encoder, which is described by the following formula:

[0220]

[0221] where g m is the modal confidence gating; m(modal) represents the modality, and L feedback represents the feedback loss function, and W m represents the feature encoder parameters of modality m, and confidence represents the confidence score of modality m (i.e., the aforementioned c(x))

[0222] In one embodiment, a multi-modal knowledge graph construction device based on a large model is provided. The device includes the following modules:

[0223] Module S1: Collect multi-modal data and perform preprocessing operations on it. The preprocessing operations include cleaning, deduplication, filtering, classification, entity alignment, and annotation.

[0224] Module S2: Train a cross-modal encoder based on the Transformer architecture and build a multi-modal perception layer. Use the multi-modal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the multi-modal data after the preprocessing operations to obtain an aligned multi-modal embedding matrix.

[0225] Module S3: Implement knowledge extraction guided by a hybrid attention mechanism and build a knowledge distillation layer. Use the knowledge distillation layer to perform two-channel knowledge extraction and knowledge fusion and verification on the aligned multi-modal embedding matrix to obtain a set of knowledge cells.

[0226] Module S4: Build a dynamic knowledge graph layer through incremental dynamic update and closed-loop feedback optimization of the inference module. The dynamic knowledge graph layer obtains a dynamically updated multi-modal knowledge graph based on the input set of knowledge cells.

[0227] In one embodiment, a computer device is provided, including: a processor and a memory. The memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned multi-modal knowledge graph construction method based on a large model via executing the executable instructions.

[0228] In one embodiment, a computer storage medium is provided. A computer program is stored in the storage medium, and when the computer program runs, it executes the above-mentioned multi-modal knowledge graph construction method based on a large model.

[0229] In one embodiment, a computer program product is provided, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the above-mentioned multi-modal knowledge graph construction method based on a large model are implemented.

[0230] In addition, in one embodiment, specific embodiments are used to further illustrate the above-mentioned multi-modal knowledge graph construction method based on a large model:

[0231] The above-mentioned multi-modal knowledge graph construction method based on a large model will have different implementation methods according to different selected encoder baseline models and data domains.

[0232] The following are specific embodiments of the present invention.

[0233] Example 1: Construction of a Multimodal Knowledge Graph in the Medical Field

[0234] A certain hospital needs to build an intelligent diagnosis and treatment system, which requires integrating multimodal data such as patients' electronic medical records (text), CT images (images), and surgical videos (videos), and updating the knowledge graph in real time to reflect the latest medical guidelines.

[0235] Traditional methods cannot meet the requirements due to problems such as high cost of manual annotation, difficulty in cross-modal alignment, and large update delays.

[0236] This solution realizes efficient and automated knowledge graph construction through the following steps:

[0237] 1. Data Integration and Cleaning

[0238] The input data includes:

[0239] Text type: Electronic medical records, etc., such as "Patient A has a right lung nodule, with a diameter of 12 mm and a spiculated margin".

[0240] Image type: CT images, etc.;

[0241] Video type: Surgical process videos, etc.

[0242] Preprocess the input data, remove duplicates, align entities (for example, associate the "right lung nodule" area in the CT image with the description in the medical record text), and annotate according to the UMLS medical terminology library.

[0243] 2. Cross-modal Semantic Alignment

[0244] First, extract features from multimodal data respectively:

[0245] Use the hierarchical BERT model to extract entity features such as "nodule size" and "location";

[0246] Use the CLIP model to extract the texture and shape features of nodules in CT images;

[0247] Use the TimeSformer model to extract the temporal action features of the "resection operation" in surgical videos.

[0248] Secondly, perform semantic alignment:

[0249] Project the multimodal vectors into a unified semantic feature space;

[0250] Use the contrastive learning method to closely associate the "CT nodule features" and "medical record text descriptions" of the same patient in the unified semantic space. For example, let the system automatically learn that the "spiculation sign" in the CT image corresponds to the description of "irregular margin" in the text.

[0251] 3. Dual-channel Knowledge Extraction

[0252] Channel A generates customized Prompts based on data content. For example, for a CT report, it generates a prompt: "Identify tumor-related entities, including location, size, and morphology."

[0253] Channel B performs graph reasoning according to predefined logical rules. For example, for the rule "Spiculation + diameter > 10mm → suspected malignancy", when the system detects the "spiculation" feature, it automatically associates the "malignancy risk" label and links to the treatment guidelines.

[0254] Perform confidence fusion on the results of the two channels. If the prediction of Channel A is "malignancy probability 85%", and the rule reasoning result of Channel B is "high risk", then it is fused into the final conclusion.

[0255] 4. Dynamic update and feedback optimization

[0256] According to the specific medical situation requirements, the "malignant nodule threshold" is adjusted from 10mm to 8mm. The system will automatically retrieve the nodule cases with a size of 8 - 10mm in the historical data and update the "risk level" labels of relevant entities in the knowledge graph.

[0257] When the doctor marks a wrong judgment of "benign → malignant" for a certain case, the system will automatically trace the features that led to the error (such as misjudged image texture), adjust the model weights, and update the threshold conditions in the rule base to avoid similar errors.

[0258] Example 2: Construction of a multi-modal knowledge graph in the industrial manufacturing field:

[0259] An automobile manufacturing plant needs to build an intelligent operation and maintenance system, which requires integrating multi-modal data such as photos taken after equipment damage, maintenance records, and production line monitoring videos to optimize equipment maintenance strategies in real time. Traditional methods have problems such as isolated data, lag in fault mode recognition, and maintenance decisions relying on experience, resulting in serious equipment downtime losses.

[0260] This solution realizes efficient and automated knowledge graph construction through the following steps:

[0261] 1. Data integration and cleaning:

[0262] The input data includes:

[0263] Text type: Text describes the fault phenomenon and treatment measures (such as "motor overheating, replace the bearing");

[0264] Image type: Photos taken after equipment damage, etc.;

[0265] Video type: Videos of abnormal production line periods (such as the movement of the robotic arm when there is abnormal noise in the equipment).

[0266] Preprocess the input data, remove duplicates, perform entity alignment (such as precisely synchronizing the sensor data timestamp with the video frame), and establish entity links (such as associating "bearing damage" in the maintenance work order with the equipment BOM number).

[0267] 2. Cross-modal semantic alignment:

[0268] First, perform feature extraction on multi-modal data separately:

[0269] Use the hierarchical BERT model to identify the fault type and component name in the maintenance text (such as "motor bearing");

[0270] Use the CLIP model to extract local features of the damaged equipment image;

[0271] Use the TimeSformer model to extract the action semantics in production anomaly videos such as the deviation of the robotic arm movement trajectory.

[0272] Secondly, perform semantic alignment:

[0273] Project the multi-modal vectors into a unified semantic feature space;

[0274] Use the contrastive learning method to closely associate the maintenance text of equipment failures with data in other modalities such as on-site photos and surveillance videos in the semantic space. For example, let the system automatically learn that "abnormal noise from the gearbox" in the surveillance video can correspond to "transmission system failure" in the maintenance text.

[0275] 3. Dual-channel knowledge extraction:

[0276] Channel A generates a dynamic Prompt based on the data content, such as "Identify the current equipment anomaly type", and uses a large model to extract and analyze "transmission system failure, gear damage degree 88%" and the corresponding image and video data from the data.

[0277] Channel B performs graph reasoning according to predefined rules. For example, for the rule "vibration amplitude > 7mm / s → need to stop for maintenance", when the system detects three consecutive vibrations exceeding the threshold, it automatically triggers an early warning and pushes the maintenance plan.

[0278] Perform confidence fusion on the results of the two channels. If Channel A predicts "remaining bearing life is 2 days" and the rule of Channel B requires "must be replaced within 72 hours", then a shutdown maintenance work order is automatically generated. In case of conflicts, the rule is given priority and the engineer is notified for review.

[0279] 4. Dynamic update and feedback optimization:

[0280] When new equipment is introduced in the actual production scenario, the system automatically collects the operation data of the new equipment and constructs a supplementary training set to improve the model accuracy and update the dynamic graph.

[0281] When maintenance personnel mark error correction information (such as "false alarm of belt breakage"), the system automatically traces the cause and updates the threshold judgment logic in the rule base.

[0282] Example 3: Construction of a multi-modal knowledge graph in the education field:

[0283] An online education platform needs to build a personalized learning system, which requires integrating multi-modal data such as course PDF lecture notes, teaching videos, and student behavior logs (clicks, answers, discussions), and recommending learning paths. Traditional recommendation systems have poor recommendation effects because they ignore multi-modal associations and cannot dynamically adapt to the learning progress.

[0284] This solution realizes efficient and automated knowledge graph construction through the following steps:

[0285] 1. Data integration and cleaning:

[0286] The input data includes:

[0287] Textual: Student behavior logs, question banks, etc.;

[0288] Image-based: Course PDF lecture notes, recorded photos of teachers correcting homework, etc.;

[0289] Video-based: Teaching videos.

[0290] Preprocess the input data, denoise (such as filtering invalid clicks and screening out video plays shorter than 5s from the data), and entity alignment (such as associating the "trigonometric function explanation" segment in the video with Chapter 3 of the textbook).

[0291] 2. Cross-modal semantic alignment:

[0292] First, perform feature extraction on multi-modal data respectively:

[0293] Use the hierarchical BERT model to identify keywords of learning difficulties, such as "not understanding derivative operations";

[0294] Use the CLIP model to extract local image features of the course PDF;

[0295] Use the TimeSformer model to extract knowledge point labels in the teaching video (such as "derivation steps of the sine theorem").

[0296] Secondly, perform semantic alignment:

[0297] Project the multi-modal vectors into a unified semantic feature space;

[0298] The contrastive learning method is used to achieve semantic correspondence of multimodal feature vectors. For example, the "video viewing interruption point" is associated with the "corresponding exercise error rate". The system automatically learns that when students pause the "solid geometry" video, they often make mistakes related to spatial imagination.

[0299] 3. Dual-channel knowledge extraction:

[0300] Channel A generates dynamic prompts based on student behavior data, such as "check the accuracy of questions in the derivative application part", and extracts "the accuracy of questions in the derivative application part is 45%" through the (language) big model, thereby recommending to review basic knowledge first.

[0301] Channel B performs graph reasoning according to predefined rules. For example, for the rule "Mastery < 60% → Need to supplement basic exercises", when the system detects that a student's knowledge mastery is low through error rate, discussion forum comments, etc., it automatically pushes the corresponding teaching video to him.

[0302] The results of the two channels are confidently integrated. If channel A recommends "studying advanced mathematics topics" and the rule of channel B restricts "need to complete the required chapters first", a step-by-step learning plan is generated.

[0303] 4. Dynamic update and feedback optimization:

[0304] If the new examination syllabus adds a "data analysis" test point, the system will automatically capture the analysis videos of the new test point, automatically update the knowledge graph, and push teaching videos to the corresponding learning based on student behavior analysis.

[0305] When a student marks that the video solution to a question does not match the question stem, the system will automatically trace back and discover errors in the video chapter labels, correct the labels, recalculate the knowledge point associations, and adjust the learning path recommendations accordingly.

[0306] A computer device or system is provided in this embodiment. The hardware device of this part is a general model and is not shown in the form of a diagram. The system includes a processor and a memory, wherein the processor and the memory can be connected via a bus or other means. The memory is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, and corresponding program instructions / modules. The processor executes various functional applications and data processing of the processor by running the non-volatile software programs, instructions and modules stored in the memory, so as to realize the data space entity resolution data quality enhancement method in the above method embodiment.

[0307] The memory may include a program storage area and a data storage area. Among them, the program storage area can store the operating system and application programs required for at least one function; the data storage area can store data created by the processor and the like. In addition, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely provided relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, an enterprise internal network, a mobile communication network, and combinations thereof.

[0308] One or more modules are stored in the memory. When the processor executes, the method steps in the embodiments are performed. In this way, through the method, device, and process of the present invention, the inventive purpose of the present invention can be achieved. The specific details of the above computer device can be understood by referring to the corresponding relevant descriptions and effects in the embodiments, and will not be elaborated here.

[0309] Those skilled in the art can understand that to implement all or part of the processes in the above embodiment methods, it can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, an optical disc, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memories.

[0310] The technical solutions provided by the present invention are further described in detail through several specific embodiments to highlight the advantages and beneficial effects of the technical solutions provided by the present invention. However, the above-mentioned several specific embodiments are not used as a limitation to the present invention. Any reasonable modification and improvement of the present invention, reasonable combination of implementation manners, and equivalent replacement within the spirit and principle scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a multi-modal knowledge graph based on a large model, characterized in that The method includes the following steps: Step S1: Collect multimodal data and perform preprocessing operations on it; the preprocessing operations include cleaning, deduplication, filtering, classification, entity alignment, and annotation; Step S2: Train a cross-modal encoder based on the Transformer architecture and build a multimodal perception layer; use the multimodal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the preprocessed multimodal data to obtain an aligned multimodal embedding matrix; Step S3: Implement knowledge extraction guided by a hybrid attention mechanism and build a knowledge distillation layer; use the knowledge distillation layer to perform two-channel knowledge extraction and knowledge fusion and verification on the aligned multimodal embedding matrix to obtain a set of knowledge cells; Step S4: Build a dynamic knowledge graph layer through incremental dynamic update and closed-loop feedback optimization of the inference module; the dynamic knowledge graph layer obtains a dynamically updated multimodal knowledge graph according to the input set of knowledge cells.

2. The method for constructing a multi-modal knowledge graph based on a large model according to claim 1, wherein In the said Step S1, the multimodal data is collected as follows: Collect multimodal data from open-source multimodal datasets.

3. The method for constructing a multi-modal knowledge graph based on a large model according to claim 1, wherein In the said Step S1, the annotation of the preprocessing operation is as follows: For incomplete entities lacking multimodal information in the multimodal data, use the Flamingo vision-language large model for annotation.

4. The method for constructing a multi-modal knowledge graph based on a large model according to claim 1, wherein In the said Step S2, using the multimodal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the preprocessed multimodal data to obtain an aligned multimodal embedding matrix includes: Step S2.1: Perform modality-specific encoding on the preprocessed multimodal data to obtain feature vectors of different modality data: For text data, use the hierarchical BERT model to encode to obtain dynamic word vectors as the feature vectors of the text data; For image data, use the spatial attention network of the CLIP large model to extract regional features as the feature vectors of the image data; For video data, use TimeSformer to capture action semantics as the feature vectors of the video data; Step S2.2: Project the feature vectors of different modalities into a unified feature vector space through optimal transport projection; Step S2.3: Use contrastive learning with the InfoNCE loss function to perform cross-modal semantic alignment in the unified feature vector space to obtain an aligned multimodal embedding matrix.

5. The method for constructing a multimodal knowledge graph based on a large model according to claim 1, wherein In the said Step S3, using the knowledge distillation layer to perform two-channel knowledge extraction and knowledge fusion and verification on the aligned multimodal embedding matrix to obtain a set of knowledge cells includes: Step S3.1: The knowledge distillation layer includes a two-channel collaborative extraction network; the two-channel collaborative extraction network includes Channel A and Channel B, where Channel A is a neural extraction channel and Channel B is a symbolic reasoning channel; Use Channel A and Channel B to perform knowledge extraction on the aligned multimodal embedding matrix respectively; Channel A uses a dynamically generated entity recognition Prompt to call the large model to extract knowledge; Channel B uses an expert model to build a rule trigger matrix and perform relational graph inference propagation on the graph neural network; Step S3.2: Perform knowledge fusion and verification on the knowledge obtained after knowledge extraction by Channel A and Channel B: A knowledge distillation bridge is set between Channel A and Channel B; By calculating the prediction confidence, an uncertainty calibration module is introduced, and Gaussian noise is added to the gradient to introduce differential privacy protection measures; Causal counterfactual argumentation is performed to ensure the rationality of the dual-channel collaborative extraction network, and a knowledge cell set is obtained.

6. The method for constructing a multi-modal knowledge graph based on a large model according to claim 1, wherein In step S4, through the incremental dynamic update and closed-loop feedback optimization of the inference module, the dynamic knowledge graph layer is built, including: Step S4.1: Incremental dynamic update: The knowledge cell set is input into the dynamic knowledge graph layer, the survival weights of knowledge units are calculated, and a dynamic adjacency matrix is constructed; Reinforcement learning is performed through importance-aware replay; Online knowledge distillation is performed; Step S4.2: Construct a closed-loop feedback optimization channel: Feedback signals in the real-time data stream are collected, the feedback loss is calculated, and it is propagated backward along the computational graph to the cross-modal encoder of the multi-modal perception layer to update the weights of the cross-modal encoder.

7. The multimodal knowledge graph construction device based on a large model is characterized in that The device includes the following modules: Module S1: Collect multi-modal data and perform preprocessing operations on it; the preprocessing operations include cleaning, deduplication, filtering, classification, entity alignment, and annotation; Module S2: Train a cross-modal encoder based on the Transformer architecture to build a multi-modal perception layer; use the multi-modal perception layer to perform modality-specific encoding and cross-modal semantic alignment on the preprocessed multi-modal data to obtain an aligned multi-modal embedding matrix; Module S3: Implement knowledge extraction guided by a hybrid attention mechanism to build a knowledge distillation layer; use the knowledge distillation layer to perform dual-channel knowledge extraction and knowledge fusion and verification on the aligned multi-modal embedding matrix to obtain a knowledge cell set; Module S4: Through the incremental dynamic update and closed-loop feedback optimization of the inference module, build a dynamic knowledge graph layer; the dynamic knowledge graph layer obtains a dynamically updated multi-modal knowledge graph according to the input knowledge cell set.

8. A computer device, comprising: A processor and a memory, characterized in that the memory is used to store executable instructions of the processor, and the processor is configured to execute the multi-modal knowledge graph construction method according to any one of claims 1-6 by executing the executable instructions.

9. A computer storage medium, characterized in that, A computer program is stored in the storage medium, and when the computer program runs, it executes the multi-modal knowledge graph construction method according to any one of claims 1-6.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the multi-modal knowledge graph construction method according to any one of claims 1-6 are implemented.

Images