A knowledge base construction method based on multimodal fusion

Abstract

This invention relates to the field of data processing technology, specifically to a method for constructing a knowledge base based on multimodal fusion. The method includes: evaluating the information entropy of image and text features at the input layer to determine mutual information between different modalities, and performing dynamic feature pruning. Subsequently, a bidirectional cross-attention mechanism is used in the intermediate layer for deep feature fusion to form collaborative features. These collaborative features are then input to a knowledge extraction unit to extract knowledge triples, and a semantic consistency score is calculated using a cross-modal similarity verification mechanism. Finally, the knowledge triples are filtered based on the score, and knowledge with high reliability is written into a multimodal knowledge graph. This invention, through optimized data preprocessing and structured knowledge filtering logic, ensures the accuracy of the knowledge base content and the stability of the system operation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for constructing a knowledge base based on multimodal fusion. Background Technology

[0002] The rapid development of the current technological environment has led to an exponential increase in the generation of multimodal data such as images, text, audio, and video. This demands that next-generation artificial intelligence systems possess the ability to comprehensively understand, structure, and efficiently retrieve heterogeneous information. Traditional knowledge bases primarily rely on a single text modality. This limitation of modal isolation makes it difficult for systems to fully utilize multi-source information for comprehensive knowledge retrieval and reasoning. Therefore, constructing multimodal knowledge graphs has become an infrastructure requirement, aiming to build a unified knowledge representation through cross-modal information supplementation, serving as a reliable anchor point for large-scale models.

[0003] However, the construction of existing multimodal knowledge bases faces several severe challenges. Regarding data organization and fusion, early fusion strategies typically concatenate low-level feature vectors from different modalities. Due to the dramatic increase in the dimensionality of the feature space, this easily leads to the curse of dimensionality, severely impacting the efficiency of subsequent knowledge extraction and data storage. In terms of knowledge reliability, successful fusion highly depends on accurate alignment. Poor alignment, such as semantic mismatch between image and text descriptions, will inevitably lead to pseudo-knowledge or semantic noise in the fusion results, impairing the stability and robustness of the stored knowledge. If the construction method cannot effectively filter this noise, its function as an external knowledge source for cognitive models will be significantly reduced, thus affecting downstream knowledge retrieval and applications.

[0004] Therefore, the current problem to be solved is: how to effectively overcome the dimensionality curse and noise sensitivity issues in the early stages of fusion in a large-scale heterogeneous data input environment, and introduce explicit knowledge verification and noise reduction mechanisms to achieve highly robust and semantically consistent knowledge extraction, construct a highly reliable multimodal knowledge graph, and ensure the structured storage and reliable retrieval of knowledge.

[0005] To address this, a knowledge base construction method based on multimodal fusion is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a knowledge base construction method based on multimodal fusion. By evaluating the information entropy and dynamically pruning the original feature vectors of images and text, a dimensionality-reduced and pre-aligned feature representation is obtained. Then, a bidirectional cross-attention mechanism is used to achieve deep collaborative fusion. Finally, a cross-modal similarity verification mechanism is used to calculate the knowledge credibility score and select highly reliable knowledge triples to be written into the multimodal knowledge graph.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for constructing a knowledge base based on multimodal fusion includes:

[0009] In the input layer, information entropy is evaluated on the original feature vectors of image features and text features to determine the mutual information between edge representations of different modalities, and dynamic feature pruning is performed based on the mutual information to obtain the pre-aligned features after dimensionality reduction.

[0010] The pre-aligned features are input into the intermediate feature representation layer, and a bidirectional cross-attention mechanism is used to achieve mutual retrieval of text features and image features. The attention weights in the two directions are weighted and averaged to form attention synergy features.

[0011] The attention collaboration features are input into the knowledge extraction unit to extract knowledge triples. A cross-modal similarity verification mechanism is used to calculate the semantic consistency score between the knowledge triples and the attention collaboration features to generate a knowledge credibility score. Based on the knowledge credibility score, the knowledge triples are screened, and knowledge triples with a knowledge credibility score higher than a preset threshold are written into the multimodal knowledge graph.

[0012] Preferably, the information entropy evaluation of the original feature vectors of image features and text features includes: constructing a self-supervised contrastive learning structure, treating the edge representations of the image features and the text features as positive sample pairs; using a contrastive learning loss function to calculate the similarity difference between the positive sample pairs and randomly sampled negative sample pairs; and maximizing the mutual information between the edge representations of different modalities by minimizing the contrastive learning loss function.

[0013] Preferably, the step of performing dynamic feature trimming based on the mutual information includes:

[0014] In each training batch of mutual information maximization learning, the contribution of the edge representations of the image features and the text features to the feature dimensions is monitored; an information entropy threshold is defined, the information content of each feature dimension is calculated, and dimensions with information content lower than the information entropy threshold are determined as low information entropy dimensions; the low information entropy dimensions are dynamically cropped and removed to generate pre-aligned features.

[0015] Preferably, the method of employing a bidirectional cross-attention mechanism to achieve mutual retrieval of text features and image features, and weighting the attention weights in both directions, includes:

[0016] Using text features as query vectors and image features as key and value vectors, a text retrieval attention weight matrix for the image direction is calculated and generated. Using image features as query vectors and text features as key and value vectors, an image retrieval attention weight matrix for the text direction is calculated and generated. The text retrieval attention weight matrix and the image retrieval attention weight matrix are then weighted and averaged to form a balanced attention co-weight. Based on the attention co-weight, pre-aligned features are aggregated to obtain attention co-features containing deep interaction information.

[0017] Preferably, the attentional collaboration features are input into the knowledge extraction unit to extract knowledge triples, which includes: using the attentional collaboration features as input to construct a graph neural network structure; the graph neural network models and learns with entities and relationships as targets, and analyzes the potential structural information in the attentional collaboration features; and through the inference operation of the graph neural network, identifies the entity pairs and their semantic relationships contained in the features to form knowledge triples.

[0018] Preferably, the step of calculating the semantic consistency score between knowledge triples and attention co-features using a cross-modal similarity verification mechanism includes:

[0019] The knowledge triples and the attention co-features are mapped to a unified semantic embedding space; a similarity metric function is defined to calculate the distance between the semantic embedding representation of the knowledge triples and the semantic embedding representation of the attention co-features; the distance is used as the output of the semantic consistency score.

[0020] Preferably, the knowledge credibility score is generated based on the semantic consistency score and is used to represent the degree of semantic consistency between the knowledge triple and the corresponding attention co-feature.

[0021] The knowledge credibility score reflects the reliability level of the knowledge triple under cross-modal verification. When the knowledge credibility score is higher than the preset threshold, the knowledge triple will be written into the multimodal knowledge graph.

[0022] Preferably, the calculation method of the preset threshold includes: determining the required system performance target based on the knowledge accuracy index and knowledge recall index of the multimodal knowledge graph in different application scenarios;

[0023] The performance of the knowledge accuracy and knowledge recall metrics is evaluated using receiver operating characteristic curves and precision-recall curves; the knowledge credibility score threshold corresponding to the optimal balance between knowledge accuracy and knowledge recall is defined as the preset threshold.

[0024] Preferably, when the multimodal knowledge graph receives input of new knowledge triples, the knowledge credibility score of the existing knowledge triples is recalculated;

[0025] If the relative decrease in the knowledge credibility score of an existing knowledge triple exceeds a preset decrease threshold, it indicates a conflict between the existing knowledge triple and the new knowledge triple, and the existing knowledge triple is marked as knowledge to be corrected.

[0026] The knowledge to be corrected and the new knowledge triples are used together as training data to perform regional fine-tuning of the graph neural network and the cross-modal similarity verification mechanism.

[0027] Preferably, when the knowledge triple is written into a multimodal knowledge graph, the knowledge element structure stored in the graph includes:

[0028] The main structural information of the knowledge triple, the unique identifier of the original image feature vector corresponding to the knowledge triple, and the unique identifier of the original text feature vector serve as multimodal evidence anchors, and the knowledge credibility score of the knowledge triple serves as the reliability weight of the knowledge element in downstream reasoning applications.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. This invention effectively solves the curse of dimensionality problem caused by simple concatenation of feature vectors in traditional multimodal early fusion methods by introducing information entropy evaluation and dynamic feature pruning mechanism based on mutual information maximization in the input layer. It avoids interference from low information entropy noise features, realizes feature dimensionality reduction and lossless pre-alignment, and improves the data processing efficiency and feature fusion quality of subsequent fusion.

[0031] 2. This invention proposes a highly reliable screening mechanism based on knowledge credibility scoring. It utilizes cross-modal similarity verification to quantify the semantic consistency between extracted knowledge triples and deeply fused features. By setting a preset threshold, this method can proactively and objectively filter out semantically inconsistent pseudo-knowledge and noise, fundamentally overcoming the visual-knowledge conflict problem in existing technologies and ensuring that the knowledge written into the multimodal knowledge graph has high robustness and high credibility.

[0032] 3. This invention designs an incremental learning and updating mode based on knowledge credibility scoring. When receiving new knowledge input, it only performs regional fine-tuning on existing knowledge whose scores have changed significantly. This mechanism avoids the complete reconstruction of the knowledge graph, ensures the stability and protection of core knowledge, and improves the efficiency and timeliness of knowledge updates, enabling the multimodal knowledge graph to continuously and reliably support downstream reasoning applications. Attached Figure Description

[0033] Figure 1 This is a flowchart of a knowledge base construction method based on multimodal fusion proposed in this invention;

[0034] Figure 2 This is a flowchart illustrating a knowledge base construction method based on multimodal fusion proposed in this invention.

[0035] Figure 3 This is a flowchart illustrating the verification process for cross-modal knowledge credibility proposed in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1

[0038] Please see Figures 1 to 3 This invention provides a knowledge base construction method based on multimodal fusion, the technical solution of which is as follows:

[0039] A knowledge base construction method based on multimodal fusion, such as Figure 1 - Figure 2 As shown, it includes:

[0040] In the input layer, information entropy is evaluated on the original feature vectors of image features and text features to determine the mutual information between edge representations of different modalities, and dynamic feature pruning is performed based on the mutual information to obtain the pre-aligned features after dimensionality reduction.

[0041] The pre-aligned features are input into the intermediate feature representation layer, and a bidirectional cross-attention mechanism is used to achieve mutual retrieval of text features and image features. The attention weights in the two directions are weighted and averaged to form attention synergy features.

[0042] The attention collaboration features are input into the knowledge extraction unit to extract knowledge triples. A cross-modal similarity verification mechanism is used to calculate the semantic consistency score between the knowledge triples and the attention collaboration features to generate a knowledge credibility score. Based on the knowledge credibility score, the knowledge triples are screened, and knowledge triples with a knowledge credibility score higher than a preset threshold are written into the multimodal knowledge graph.

[0043] Furthermore, the information entropy evaluation of the original feature vectors of image features and text features includes: constructing a self-supervised contrastive learning structure, treating the edge representations of the image features and the text features as positive sample pairs; using a contrastive learning loss function to calculate the similarity difference between the positive sample pairs and randomly sampled negative sample pairs; and maximizing the mutual information between the edge representations of different modalities by minimizing the contrastive learning loss function.

[0044] The raw feature vector refers to the initial representation extracted from raw multimodal data using a pre-trained deep neural network. For example, for image modalities, ResNet-50 can be used as a feature extractor to extract feature maps, followed by global average pooling to obtain a 2048-dimensional floating-point vector; for text modalities, the BERT model can be used as an encoder to extract the 768-dimensional floating-point vector corresponding to the [CLS] tag. These vectors constitute the input for subsequent processing.

[0045] The contrastive learning loss function specifically employs the InfoNCE loss function. The calculation of the InfoNCE loss function involves an important temperature parameter. This parameter is typically set within the range of [0.05, 0.2], for example, a value of 0.07, to control the scale of similarity calculation. This loss function ensures that during encoder training, the similarity between positive sample pairs is maximized, while the similarity with negative sample pairs is minimized.

[0046] The process of maximizing mutual information employs the Adam optimizer for parameter updates, with the initial learning rate set to [value missing]. During training, the number of samples in each batch was set to 256 pairs of multimodal features to ensure that the contrastive learning loss function could obtain sufficient negative samples for contrastive training.

[0047] The edge representation refers to the independent feature distribution representation of the image modality and the text modality before cross-modal interaction fusion. Specifically, in this invention, the image edge representation is the visual feature vector extracted by ResNet-50, and the text edge representation is the semantic feature vector extracted by BERT.

[0048] This embodiment effectively ensures the semantic alignment of the original feature vectors of images and text by constructing a self-supervised contrastive learning structure and minimizing the InfoNCE loss. This maximizes the mutual information between edge representations of different modalities, resulting in higher discriminative power and semantic consistency of the initial features. It not only solves the curse of dimensionality problem caused by the rapid expansion of feature dimensions in early multimodal fusion but also directly reduces the computational complexity and memory consumption of subsequent modules, improving the operational efficiency and robustness of the entire knowledge base construction process.

[0049] Furthermore, the step of performing dynamic feature trimming based on the mutual information includes:

[0050] In each training batch of mutual information maximization learning, the contribution of the edge representations of the image features and the text features to the feature dimensions is monitored; an information entropy threshold is defined, the information content of each feature dimension is calculated, and dimensions with information content lower than the information entropy threshold are determined as low information entropy dimensions; the low information entropy dimensions are dynamically cropped and removed to generate pre-aligned features.

[0051] The information entropy threshold This is a preset hyperparameter, for example, set to 0.1, used to distinguish the information contribution of each feature dimension. For each feature dimension... Its information content This is obtained by calculating the Shannon entropy of the empirical probability distribution of the feature values ​​in that dimension. Specifically, since the original feature values ​​are continuous floating-point numbers, it is necessary to perform a certain calculation on each feature dimension before calculating the empirical probability distribution. eigenvalues Perform equal-frequency binning discretization. Distribute all samples in the training batch across dimensions... The eigenvalues ​​on the surface are uniformly divided into Discretized intervals This ensures that the number of samples within each interval is approximately equal. Then, statistical characteristic values ​​are calculated. Within the discretization range empirical probability Information content The calculation formula is: calculate the probability of each interval. Its logarithm The product of the products is then summed negatively over all intervals. By dynamically pruning and removing dimensions with less information than this threshold, dimensionality reduction and noise reduction can be effectively achieved.

[0052] In the dynamic feature pruning process, feature dimension correlation analysis is introduced. Specifically, the Pearson correlation coefficient matrix between the remaining feature dimensions after pruning is calculated. A correlation threshold is set, for example, 0.85. For feature dimension pairs with correlation coefficients higher than the threshold, the dimension with higher information content (Shannon entropy) is retained, and the dimension with lower information content is removed. Through the feature dimension correlation analysis, redundant linear correlations between features after pruning are further eliminated, generating pre-aligned features with lower dimensionality and stronger independence between features. This scheme effectively solves the problem of redundant information residues between features that may be caused by evaluating based on only single-dimensional information entropy in the mutual information maximization learning process. It further improves the quality of pre-aligned features and the robustness of subsequent fusion, avoiding the waste of computational resources and potential model overfitting risks caused by feature redundancy.

[0053] This embodiment utilizes Shannon entropy calculation to identify and remove low-information dimensions in the high-dimensional feature space. This removes redundant information and dimensional noise from the features, avoiding the curse of dimensionality problem caused by the rapid increase in feature dimensions in traditional early-stage fusion. This significantly reduces the computational burden on subsequent models and the demand for storage resources, thereby improving the system's operating efficiency.

[0054] Furthermore, the method employs a bidirectional cross-attention mechanism to achieve mutual retrieval of text features and image features, and performs a weighted average of the attention weights in the two directions, including:

[0055] Using text features as query vectors and image features as key and value vectors, a text retrieval attention weight matrix for the image direction is calculated and generated. Using image features as query vectors and text features as key and value vectors, an image retrieval attention weight matrix for the text direction is calculated and generated. The text retrieval attention weight matrix and the image retrieval attention weight matrix are then weighted and averaged to form a balanced attention co-weight. Based on the attention co-weight, pre-aligned features are aggregated to obtain attention co-features containing deep interaction information.

[0056] The weighted average fusion is used to combine attention weight matrices from two directions to achieve balanced interaction. This invention can employ fixed weighting, i.e., a text retrieval attention weight matrix. and image retrieval attention weight matrix Each component is assigned a weight of 0.5 and then merged. Alternatively, dynamic weighting can be used: text weights are predicted from pre-aligned features using a small fully connected network (MLP). Image weights This ensures that the sum of the two weights is one, achieving dynamic balance.

[0057] Before calculating the text retrieval attention weight matrix and the image retrieval attention weight matrix, intra-modal feature normalization is performed on the pre-aligned features. Specifically, L2 norm normalization is performed on the text feature vector and the image feature vector respectively. Through the intra-modal feature normalization, the scale differences of feature vectors from different modalities are eliminated. After generating the text retrieval attention weight matrix and the image retrieval attention weight matrix, attention weight smoothing technology is applied. Specifically, a Gaussian kernel function is used to perform a convolution operation on the attention weight matrix. Through the intra-modal feature normalization and attention weight smoothing technology, the stability of the bidirectional cross-attention mechanism is enhanced when facing changes in the scale of intra-modal features or local noise interference, ensuring that the generated attention co-weights are smoother and more robust, reducing attention bias caused by individual abnormal feature values, and improving the accuracy and reliability of feature deep fusion.

[0058] The dynamically weighted MLP network structure consists of an input layer, hidden layers, and an output layer. Its input is a concatenated vector of image feature vectors and text feature vectors after global average pooling. The hidden layer contains a fully connected layer with 128 neurons and uses the ReLU activation function. The output layer contains a fully connected layer with one neuron, connected to a sigmoid activation function at its end, mapping the output value to the (0, 1) interval.

[0059] This embodiment employs a bidirectional cross-attention mechanism, allowing each modality to actively capture and focus on the most relevant semantic information in the other modality. This effectively overcomes the problems of large information loss and shallow interaction in traditional unidirectional fusion, greatly improving the comprehensiveness and accuracy of feature fusion. By weighted averaging the bidirectional attention weight matrices, a balanced attentional collaborative weight is formed. This ensures that the fusion process does not favor any particular modality, but rather aggregates pre-aligned features in an optimal balance, generating collaborative features containing deep interaction information. These collaborative features have higher semantic representation capabilities and cross-modal consistency, providing high-quality and robust input for subsequent knowledge extraction units, significantly improving the semantic accuracy of the final knowledge triples.

[0060] Furthermore, the attentional collaboration features are input into the knowledge extraction unit to extract knowledge triples, including: using the attentional collaboration features as input to construct a graph neural network structure; the graph neural network models and learns with entities and relationships as targets, and analyzes the potential structural information in the attentional collaboration features; through the reasoning operation of the graph neural network, the entity pairs and their semantic relationships contained in the features are identified to form knowledge triples.

[0061] Before constructing the graph neural network structure, both entity embeddings and relation embeddings are randomly initialized using a Glorot uniform distribution. The attention-cooperative features are used as the initial feature vectors of the entities and input into the first layer of the graph neural network.

[0062] The transformation process from the attention collaboration features to the graph structure is as follows:

[0063] First, each feature vector of length D in the attention collaboration feature matrix (corresponding to a text term or image region in the original multimodal data) is considered a candidate node in the graph. Second, the cosine similarity between nodes is calculated to evaluate their association strength. If the similarity between two node features exceeds a preset correlation threshold (e.g., 0.75), an edge is established between the two nodes, using the similarity value as the initial weight of the edge. Finally, through the connection relationships between the nodes and edges, an adjacency matrix is ​​generated for the graph neural network input, thereby transforming the linearly arranged collaboration features into a graph structure with topological relationships.

[0064] The graph neural network structure specifically employs a relational graph convolutional network, which is optimized for multi-relation modeling in knowledge graphs. The graph neural network contains two R-GCN layers and uses ReLU as the activation function to increase the model's non-linear representation capability. The input nodes are entity embeddings from attention-coordinated features, and the output layer uses Softmax activation for probabilistic classification of entity and relation types. During training, a cross-entropy loss function is used for optimization.

[0065] The modeling and learning process of the graph neural network is performed using the AdamW optimizer, with a learning rate set to... The learning rate is scheduled using a warm-up strategy. During the training phase of the graph neural network, the batch size of the knowledge extraction module is set to 32 knowledge graph subgraphs to balance training efficiency and model stability.

[0066] After the knowledge credibility score is passed, the knowledge triples will be written into the multimodal knowledge graph. The storage structure of the multimodal knowledge graph adopts a three-layer indexing mechanism: the first layer stores the knowledge triples (S, P, O) and their low-dimensional embeddings; the second layer indexes the corresponding high-dimensional attention collaborative features through the unique identifier of the knowledge triples; the third layer indexes the original image and text data through the feature identifiers, realizing efficient cross-modal retrieval and tracking.

[0067] This embodiment utilizes a relational graph convolutional network to perform deep analysis and reasoning on the latent structural information in attention collaboration features, significantly improving the extraction accuracy and reliability of knowledge triples. Simultaneously, the knowledge graph employs a three-layer indexing mechanism for storage, enabling efficient cross-modal retrieval and tracking of knowledge and original image / text data, enhancing the practicality and traceability of the knowledge base.

[0068] Furthermore, such as Figure 3 As shown, the semantic consistency score between knowledge triples and attention co-features calculated using the cross-modal similarity verification mechanism includes:

[0069] The knowledge triples and the attention co-features are mapped to a unified semantic embedding space; a similarity metric function is defined to calculate the distance between the semantic embedding representation of the knowledge triples and the semantic embedding representation of the attention co-features; the distance is used as the output of the semantic consistency score.

[0070] The unified semantic embedding space is set to 128 dimensions. Knowledge triples and attention-coordinated features are mapped to this unified space through two independent fully connected layers with softmax activation. The similarity metric function is defined as the semantic embedding representation of the knowledge triples. Semantic embedding representation of attention-coordinated features The cosine similarity between them is calculated using the following formula: The similarity value is then used as the output of the semantic consistency score.

[0071] This embodiment maps the structured information of knowledge triples and the deep fusion information of attention-coordinated features into a unified semantic embedding space. By calculating the cosine similarity between the two as a semantic consistency score, an objective and quantitative evaluation of the extracted knowledge triples is achieved. This mechanism directly verifies whether the knowledge triples faithfully and accurately reflect the semantics of the original multimodal data, effectively filtering out pseudo-knowledge or semantic noise that may be generated during the knowledge extraction process. Therefore, it significantly improves the reliability and robustness of the knowledge written into the knowledge graph, ensuring the high quality of the final content of the knowledge base.

[0072] Furthermore, the knowledge credibility score is generated based on the semantic consistency score and is used to represent the degree of semantic consistency between the knowledge triple and the corresponding attention co-feature.

[0073] The knowledge credibility score reflects the reliability level of the knowledge triple under cross-modal verification. When the knowledge credibility score is higher than the preset threshold, the knowledge triple will be written into the multimodal knowledge graph.

[0074] This embodiment generates a knowledge credibility score based on semantic consistency scoring and uses it as the final criterion for knowledge selection. This quantitative scoring based on cross-modal semantic fit effectively ensures that only knowledge triples with high reliability are written into the knowledge graph, fundamentally improving the quality, purity, and robustness of the knowledge base.

[0075] Furthermore, the calculation method for the preset threshold includes: determining the required system performance target based on the knowledge accuracy and knowledge recall indicators of the multimodal knowledge graph in different application scenarios;

[0076] The performance of the knowledge accuracy and knowledge recall metrics is evaluated using receiver operating characteristic curves and precision-recall curves; the knowledge credibility score threshold corresponding to the optimal balance between knowledge accuracy and knowledge recall is defined as the preset threshold.

[0077] The definition of the optimal balance point is to achieve the highest accuracy of knowledge (…). ) and knowledge recall ( The point that achieves the optimal compromise. Specifically, this point corresponds to... The critical value at which the maximum score is located. The formula for calculating fractions is: Select this The knowledge credibility score threshold corresponding to the highest score is used as the final preset threshold.

[0078] This embodiment uses knowledge accuracy and knowledge recall as core indicators, employing receiver operating characteristic curves and precision-recall curves for performance evaluation. This ensures that the threshold setting is based on objective, quantifiable system performance goals, rather than arbitrary empirical values. Through calculation... The critical value of the knowledge credibility score corresponding to the maximum score is used as the final threshold, achieving the best balance and optimal trade-off between knowledge accuracy and recall. This ensures that the knowledge screening mechanism can both maximize the inclusion of reliable knowledge (high recall) and effectively eliminate pseudo-knowledge and noise (high accuracy), thereby significantly improving the overall construction quality and robustness of the multimodal knowledge graph.

[0079] Furthermore, when the multimodal knowledge graph receives input of new knowledge triples, the knowledge credibility score of the existing knowledge triples is recalculated.

[0080] If the relative decrease in the knowledge credibility score of an existing knowledge triple exceeds a preset decrease threshold, it indicates a conflict between the existing knowledge triple and the new knowledge triple, and the existing knowledge triple is marked as knowledge to be corrected.

[0081] The knowledge to be corrected and the new knowledge triples are used together as training data to perform regional fine-tuning of the graph neural network and the cross-modal similarity verification mechanism.

[0082] The trigger condition for the relative decline of the knowledge credibility score exceeding the preset decline threshold is: the relative decline of the knowledge credibility score exceeds the predetermined threshold, for example, 10%.

[0083] The regional fine-tuning aims to make small updates only to the model parameters most affected by conflicts, in order to quickly adapt to new knowledge and maintain the stability of existing knowledge. Specifically, during the fine-tuning process, only the parameters of the last relation classification layer of the graph neural network and the semantic embedding mapping layer in the cross-modal similarity verification mechanism are unfrozen, while the parameters of other layers (such as the parameters of the attention mechanism and the first few layers of the graph neural network) remain unchanged.

[0084] The fine-tuning process uses the same cross-entropy loss function as the knowledge extraction unit in the initial training, but trains with a very small learning rate (e.g., one-tenth of the initial learning rate).

[0085] This embodiment recalculates the credibility score of existing knowledge in real time upon receiving new knowledge, enabling rapid detection of knowledge conflicts. When the score of existing knowledge drops significantly, the system immediately identifies a semantic conflict between it and the new knowledge and marks it as knowledge to be corrected. By using conflicting knowledge as training data and performing regional fine-tuning on the model, this method achieves rapid adaptation to and efficient resolution of conflicts. This fine-tuning only updates the local parameters of the model most affected by the conflict (such as the classification layer and mapping layer), avoiding catastrophic forgetting and global retraining of the entire knowledge graph and model parameters, significantly improving the online update efficiency of the knowledge graph and the stability of knowledge maintenance.

[0086] Furthermore, when the knowledge triple is written into the multimodal knowledge graph, the knowledge element structure stored in the graph includes:

[0087] The main structural information of the knowledge triple, the unique identifier of the original image feature vector corresponding to the knowledge triple, and the unique identifier of the original text feature vector serve as multimodal evidence anchors, and the knowledge credibility score of the knowledge triple serves as the reliability weight of the knowledge element in downstream reasoning applications.

[0088] In this embodiment, the knowledge graph explicitly records the unique identifiers of the original feature vectors as anchor points for multimodal evidence, ensuring the integrity and traceability of knowledge. Simultaneously, the knowledge credibility score is directly used as a reliability weight in downstream reasoning applications, significantly improving the accuracy of reasoning and the robustness of decision-making.

[0089] This method effectively addresses the curse of dimensionality and noise sensitivity issues faced by traditional early-stage multimodal fusion through information entropy evaluation and dynamic feature pruning at the input layer. It achieves efficient feature reduction and redundancy removal, significantly improving data processing efficiency and robustness. In the intermediate layer, a bidirectional cross-attention mechanism enables deep, balanced interactive fusion of image and text features, generating collaborative features with higher semantic representation capabilities. This provides a high-quality input foundation for subsequent knowledge extraction. In the knowledge extraction unit, a graph neural network model extracts knowledge triples from the collaborative features, and a cross-modal similarity verification mechanism is used to quantify the semantic consistency of the extracted knowledge. Finally, knowledge is filtered based on credibility scores, ensuring that only high-reliability knowledge is written into the multimodal knowledge graph, fundamentally guaranteeing the accuracy, stability, and high purity of the knowledge base content.

[0090] Example 2

[0091] This embodiment applies the method of the present invention to a specific application scenario—the construction and maintenance of a context-aware knowledge graph in an intelligent transportation system. This knowledge graph aims to extract structured knowledge from onboard sensor data, such as camera vision and text information processed by LiDAR, thereby providing highly reliable environmental understanding for autonomous driving decisions.

[0092] The original multimodal data in this embodiment is:

[0093] Image features: Real-time environmental image frames captured by the vehicle-mounted camera.

[0094] Text features: Environmental descriptions and obstacle attribute texts generated from preprocessed and summarized LiDAR and millimeter-wave radar data.

[0095] For the image modality, this embodiment uses a pre-trained EfficientNet-B7 model as the feature extractor. After feature extraction, a 1280-dimensional visual feature vector is obtained through global average pooling. For the text modality, a T5 model is used as the encoder to extract a 768-dimensional text feature vector corresponding to the text summary. These vectors constitute the original input for subsequent processing. The information entropy evaluation of the original feature vectors includes constructing a self-supervised contrastive learning structure. Edge representations of image features and text features collected at the same timestamp and geographical location are considered as positive sample pairs, and the InfoNCE loss function is used for training. The temperature parameter of the loss function... The value is set to 0.07. By minimizing the loss function, the mutual information between visual features and sensor summary features is maximized.

[0096] The dynamic feature trimming based on mutual information is defined by defining an information entropy threshold. The value is 0.10. For each feature dimension, equal-frequency binning discretization is used, uniformly dividing the feature values ​​of all samples in the training batch into N=10 discretization intervals for that dimension. After calculating the Shannon entropy information content, dynamic pruning and removal of information content below a certain threshold are performed. =0.10 dimension. After pruning, the original 2048-dimensional feature vector is reduced to, for example, a 1536-dimensional pre-aligned feature vector, effectively removing low information entropy features caused by weather noise, redundant road textures, or unimportant sensor data descriptions.

[0097] The 1536-dimensional pre-aligned features are input into the intermediate feature representation layer, and a bidirectional cross-attention mechanism is used to achieve mutual retrieval of visual features and sensor text features. Through cross-attention calculation in two directions, attention weight matrices for image retrieval in the text direction and text retrieval in the image direction are generated, respectively.

[0098] This embodiment employs a dynamic weighting method to perform a weighted average fusion of the attention weight matrices for the two directions. A small fully connected network (MLP) predicts one weight from the pre-aligned features, and the other weight is obtained by subtracting this weight from the first, achieving a dynamic balance between the two directions. This allows the system to automatically increase the weight of sensor text information in specific scenarios such as low visibility.

[0099] The attention-coordinated features are input into the knowledge extraction unit to extract knowledge triples, such as vehicle A, distance, 20 meters, or traffic light, status, and red. A graph neural network structure is constructed, specifically using a relational graph convolutional network (R-GCN), which contains two R-GCN layers and uses ReLU as the activation function. The attention-coordinated features, after parsing, are used as the initial feature vector input for the entities. Through the inference operation of R-GCN, entity pairs and their semantic relationships contained in the features are identified, forming knowledge triples. The training batch size of the R-GCN is set to 32 knowledge graph subgraphs, and the learning rate is set to... The method utilizes a cross-modal similarity verification mechanism to map knowledge triples and collaborative features into a unified 128-dimensional semantic embedding space. Cosine similarity is used as the similarity metric function to calculate the distance between the knowledge triple embedding representation and the collaborative feature embedding representation, and this similarity value is used as the semantic consistency score. The preset threshold is calculated as follows: based on the high requirements of accuracy and recall in autonomous driving scenarios, performance is evaluated using receiver operating characteristic curves and precision-recall curves, and a selection is made... The knowledge credibility score threshold corresponding to the maximum score is defined as the preset threshold.

[0100] When the knowledge triples are written into the multimodal knowledge graph, in addition to the main structural information of the triples, the knowledge element structure also stores the unique identifiers of the original EfficientNet-B7 visual feature vectors and the original T5 text feature vectors as multimodal evidence anchors, as well as the knowledge credibility score of the knowledge triples, which serves as the reliability weight of the knowledge element in risk assessment during downstream path planning and decision-making models. The incremental learning update mode is as follows: when the knowledge graph receives new scene knowledge input, the knowledge credibility score of the existing knowledge triples is recalculated in real time. If the relative decrease in the existing knowledge credibility score exceeds a predetermined threshold of 10%, it indicates a semantic conflict between the old and new knowledge, such as a traffic light state conflict, and the existing knowledge is marked as knowledge to be corrected. The conflicting knowledge and the new knowledge triples are used together as training data to perform regional fine-tuning on the last relation classification layer of R-GCN and the semantic embedding mapping layer in the cross-modal similarity verification mechanism. The fine-tuning process uses a very small learning rate. This is to enable rapid adaptation to constantly changing traffic situations.

[0101] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a knowledge base based on multimodal fusion, characterized in that, include: In the input layer, information entropy is evaluated on the original feature vectors of image features and text features to determine the mutual information between edge representations of different modalities, and dynamic feature pruning is performed based on the mutual information value to obtain the pre-aligned features after dimensionality reduction. The information entropy evaluation of the original feature vectors of image features and text features includes: constructing a self-supervised contrastive learning structure, treating the edge representations of the image features and the text features as positive sample pairs; using a contrastive learning loss function to calculate the similarity difference between the positive sample pairs and randomly sampled negative sample pairs; and maximizing the mutual information between the edge representations of different modalities by minimizing the contrastive learning loss function. The step of performing dynamic feature clipping based on the mutual information value includes: In each training batch of mutual information maximization learning, the contribution of the edge representations of the image features and the text features to the feature dimensions is monitored; an information entropy threshold is defined, and the information content based on Shannon entropy is calculated for each feature dimension. Dimensions with information content lower than the information entropy threshold are judged as low information entropy dimensions; the low information entropy dimensions are dynamically cropped and removed to generate pre-aligned features. The pre-aligned features are input into the intermediate feature representation layer, and a bidirectional cross-attention mechanism is used to achieve mutual retrieval of text features and image features. The attention weights in the two directions are weighted and averaged to form attention synergy features. The attention collaboration features are input into the knowledge extraction unit to extract knowledge triples. A cross-modal similarity verification mechanism is used to calculate the semantic consistency score between the knowledge triples and the attention collaboration features to generate a knowledge credibility score. Based on the knowledge credibility score, the knowledge triples are screened, and knowledge triples with a knowledge credibility score higher than a preset threshold are written into the multimodal knowledge graph.

2. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that, The method employs a bidirectional cross-attention mechanism to achieve mutual retrieval of text features and image features, and performs a weighted average of the attention weights in both directions, including: Using text features as query vectors and image features as key and value vectors, a text retrieval attention weight matrix for the image direction is calculated and generated. Using image features as query vectors and text features as key and value vectors, an image retrieval attention weight matrix for the text direction is calculated and generated. The text retrieval attention weight matrix and the image retrieval attention weight matrix are then weighted and averaged to form a balanced attention co-weight. Based on the attention co-weight, pre-aligned features are aggregated to obtain attention co-features containing deep interaction information.

3. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that, The attentional collaboration features are input into the knowledge extraction unit to extract knowledge triples, including: using the attentional collaboration features as input to construct a graph neural network structure; the graph neural network models and learns with entities and relationships as targets, and analyzes the potential structural information in the attentional collaboration features; through the inference operation of the graph neural network, the entity pairs and their semantic relationships contained in the features are identified to form knowledge triples.

4. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that, The method of calculating the semantic consistency score between knowledge triples and attention co-features using a cross-modal similarity verification mechanism includes: The knowledge triples and the attention co-features are mapped to a unified semantic embedding space; a similarity metric function is defined to calculate the distance between the semantic embedding representation of the knowledge triples and the semantic embedding representation of the attention co-features; the distance is used as the output of the semantic consistency score.

5. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that: The knowledge credibility score is generated based on the semantic consistency score and is used to represent the degree of semantic consistency between the knowledge triple and the corresponding attention co-feature. The knowledge credibility score reflects the reliability level of the knowledge triple under cross-modal verification. When the knowledge credibility score is higher than the preset threshold, the knowledge triple will be written into the multimodal knowledge graph.

6. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that: The calculation method of the preset threshold includes: determining the required system performance target based on the knowledge accuracy index and knowledge recall index of the multimodal knowledge graph in different application scenarios; The performance of the knowledge accuracy and knowledge recall metrics is evaluated using receiver operating characteristic curves and precision-recall curves; the knowledge credibility score threshold corresponding to the optimal balance between knowledge accuracy and knowledge recall is defined as the preset threshold.

7. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that, include: When a multimodal knowledge graph receives new knowledge triples as input, the knowledge credibility score of the existing knowledge triples is recalculated. If the relative decrease in the knowledge credibility score of an existing knowledge triple exceeds a preset decrease threshold, it indicates a conflict between the existing knowledge triple and the new knowledge triple, and the existing knowledge triple is marked as knowledge to be corrected. The knowledge to be corrected and the new knowledge triples are used together as training data to perform regional fine-tuning of the graph neural network and the cross-modal similarity verification mechanism.

8. The knowledge base construction method based on multimodal fusion according to claim 1, characterized in that, When the knowledge triple is written into a multimodal knowledge graph, the knowledge element structure stored in the graph includes: The main structural information of the knowledge triple, the unique identifier of the original image feature vector corresponding to the knowledge triple, and the unique identifier of the original text feature vector serve as multimodal evidence anchors, and the knowledge credibility score of the knowledge triple serves as the reliability weight of the knowledge element in downstream reasoning applications.

Images