A multimodal data semantic fusion method, system and device

Abstract

The application discloses a multimodal data semantic fusion method, system and device, which first extracts independent modal features and cross-modal public features, performs cross attention on the public features as queries and the independent features as key values, suppresses noise, outputs enhanced features containing public semantics and private information, and realizes fine alignment; then, stacks the features to construct a multimodal graph, dynamically updates an adjacency matrix after graph convolution of each layer, learns interdependence by using multi-head graph attention, and outputs graph enhanced features with rich semantics after 2-layer propagation; finally, the quality of the modes is evaluated from clarity, integrity and consistency, weights are obtained through temperature softmax after average scoring, the weights of low-quality modes are reset to zero, and weighted fusion is realized to achieve robust fusion.

Description

Technical Field

[0001] This invention relates to the field of multimodal data, and in particular to a method, system, and apparatus for multimodal data semantic fusion. Background Technology

[0002] With the rapid development of multimodal large-scale models and knowledge-intensive applications, the deep integration of multimodal data (text, images, audio, etc.) with graph databases (knowledge bases) has become crucial for overcoming the dual bottlenecks of semantic understanding and relational reasoning in intelligent systems. Traditional multimodal data processing faces two fundamental contradictions: firstly, the storage methods for different modalities vary significantly—unstructured data is stored as vector embeddings, while structured data is stored as graph nodes and edges, making it difficult to efficiently query semantic similarities and relational associations within a unified framework; secondly, while graph databases excel at relational traversal and path reasoning, their native support for multimodal data is insufficient, leading to difficulties in semantic alignment between modalities and low efficiency in fusion reasoning. Therefore, there is an urgent need for a unified representation method that can transform multimodal data into a storable, queryable, and reasonable representation for graph databases. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal data semantic fusion method, system, and apparatus, which solves the aforementioned technical problems pointed out in the prior art.

[0004] This invention provides a multimodal data semantic fusion method, comprising the following steps:

[0005] Modal independent features and cross-modal common features are extracted from the input multimodal data respectively. The co-encoding and decoding of common features and independent features are achieved through multi-head attention interaction processing to obtain the enhanced multimodal feature vector.

[0006] The enhanced multimodal feature vectors are stacked to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node. The implicit semantic associations between modalities are mined through a dynamic graph convolutional network to obtain graph-enhanced feature vectors.

[0007] Based on the modal quality assessment branch, the input quality of each modality is evaluated in real time. The fusion weights of each modality are assigned according to the evaluation results, and the enhanced feature vectors of each graph are weighted and fused to obtain the final feature vector.

[0008] Preferably, modal-independent features and cross-modal common features are extracted from the input multi-modal data, and the common features and independent features are co-encoded and decoded through a multi-head attention interaction processing method to obtain an enhanced multi-modal feature vector, including the following steps:

[0009] The input modal data is preprocessed to obtain preprocessed modal data;

[0010] A modal-independent encoder is used to extract modal-independent feature vectors from the preprocessed modal data;

[0011] A cross-modal encoder with shared parameters maps the preprocessed modal data to a unified common semantic space and extracts cross-modal common feature vectors.

[0012] After concatenating the modal-independent feature vectors and cross-modal common feature vectors of each modal data, the enhanced multimodal feature vectors are obtained through multi-head attention interaction processing.

[0013] Preferably, the multimodal data includes raw text data, raw image data, and raw audio data; the preprocessed modal data includes preprocessed text data, preprocessed image data, and preprocessed audio data.

[0014] The modality-independent feature vectors include text modality-independent feature vectors, image modality-independent feature vectors, and audio modality-independent feature vectors; the cross-modal encoder with shared parameters is a 2-layer Transformer structure, and the parameters are completely shared between different modalities;

[0015] The cross-modal common feature vector includes text cross-modal common feature vector, image cross-modal common feature vector, and audio cross-modal common feature vector;

[0016] The enhanced multimodal feature vectors include text-enhanced multimodal feature vectors, image-enhanced multimodal feature vectors, and audio-enhanced multimodal feature vectors.

[0017] Preferably, the enhanced multimodal feature vectors are stacked to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node, and a dynamic graph convolutional network is used to mine the implicit semantic associations between modalities to obtain graph-enhanced feature vectors. The process includes the following steps:

[0018] The enhanced multimodal feature vectors are mapped to the same dimension to obtain the initial feature vectors of each modality; the initial feature vectors of each modality are stacked row by row to form a multimodal complementary feature matrix.

[0019] Multimodal complementary feature matrix The row vectors are the node features. The cosine similarity between any two node features is calculated to form the initial adjacency matrix.

[0020] A two-layer dynamic graph convolutional network is used in conjunction with an initial adjacency matrix to perform convolutional propagation on the multimodal complementary feature matrix to obtain the target multimodal complementary feature matrix;

[0021] Based on the target multimodal complementary feature matrix, a multi-head graph attention processing method is applied to learn independent attention weights for intermodal dependencies, thereby obtaining graph-enhanced feature vectors for each modality node.

[0022] Preferably, the two-layer dynamic graph convolutional network includes a first graph convolutional layer and a second graph convolutional layer.

[0023] Preferably, a two-layer dynamic graph convolutional network is used in conjunction with an initial adjacency matrix to perform convolutional propagation on the multimodal complementary feature matrix to obtain the target multimodal complementary feature matrix, including the following steps:

[0024] The first graph convolutional layer is used in conjunction with the initial adjacency matrix to process the multimodal complementary feature matrix. Propagation is performed to obtain the second multimodal complementary feature matrix;

[0025] The second adjacency matrix is ​​calculated based on the second multimodal complementary feature matrix.

[0026] The second graph convolutional layer is used in conjunction with the second adjacency matrix to apply the second multimodal complementary feature matrix. Propagation is performed to obtain the target multimodal complementary feature matrix.

[0027] Preferably, based on the modal quality assessment branch, the input quality of each modality is evaluated in real time, and the fusion weights of each modality are assigned according to the evaluation results. The enhanced feature vectors of each graph are then weighted and fused to obtain the final feature vector, including the following steps:

[0028] The quality of preprocessed text data, preprocessed image data, and preprocessed audio data is evaluated, and quality scores for each modality are output.

[0029] The fusion weights are calculated based on modal quality scores combined with a softmax function with a temperature coefficient.

[0030] The enhanced feature vectors of each image are weighted and fused according to their corresponding fusion weights to obtain the final feature vector.

[0031] Preferably, the quality assessment includes a sharpness assessment, an integrity assessment, and a consistency assessment;

[0032] The sharpness assessment includes calculating the Laplacian variance of the preprocessed image data and then normalizing it to obtain an image sharpness score; calculating the signal-to-noise ratio of the preprocessed audio data and then normalizing it to obtain an audio sharpness score; the default sharpness score for the text modality is 1.0.

[0033] Integrity assessment includes a text integrity score of 1 - number of truncated characters / total number of characters; an image integrity score of 1 - occluded area / total area; and an audio integrity score of 1 - number of silent frames / total frames.

[0034] Consistency assessment includes a default consistency score of 0.8 for non-strictly registered multimodal data; a default consistency score of 0.8 for strictly registered multimodal data, calculated based on the temporal offset; and a consistency score calculated based on the temporal offset for strictly registered multimodal data.

[0035] The present invention also provides a multimodal data semantic fusion system, including a feature enhancement module, a graph enhancement module, and a fusion module;

[0036] The feature enhancement module is used to extract modal independent features and cross-modal common features from the input multimodal data respectively, and to achieve collaborative encoding and decoding of common features and independent features through multi-head attention interaction processing to obtain the enhanced multimodal feature vector;

[0037] The graph augmentation module is used to stack the augmented multimodal feature vectors to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node. The implicit semantic associations between modalities are mined through a dynamic graph convolutional network to obtain graph augmented feature vectors.

[0038] The fusion module is used to evaluate the quality of each modal input in real time based on the modal quality assessment branch, allocate fusion weights for each modality according to the evaluation results, and weight and fuse the enhanced feature vectors of each graph to obtain the final feature vector.

[0039] In another aspect, the present invention also provides an apparatus comprising:

[0040] A memory that stores computer programs that can run on a processor;

[0041] A processor that, when executing the program, implements the steps of the method described above;

[0042] The processor is equipped with a graphics processing unit or a neural network processing unit to accelerate multi-head attention calculation and graph convolution operation.

[0043] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of a multimodal data semantic fusion method.

[0044] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0045] Analysis of the multimodal data semantic fusion method, system, and apparatus provided by this invention reveals that, in practical applications, modal-independent features and cross-modal common features are extracted simultaneously, preserving modal-specific information while extracting shared semantics between modalities. Subsequently, multi-head cross-attention calculation is performed using common features as queries and independent features as keys / values, allowing common semantics to guide the enhancement of independent features, while attention weights suppress noise or irrelevant information in independent features. Residual connections and layer normalization preserve the original common features, avoiding overfitting. The final output enhanced feature vector contains both cross-modal aligned common semantics and retains the discriminative information specific to each modality. Compared to direct concatenation or simple weighting, this achieves refined alignment of semantics between modalities, reduces alignment errors, and provides higher-quality node features for subsequent graph structure modeling.

[0046] Next, the enhanced feature vectors of each modality are stacked into a matrix, with each modality serving as a graph node, constructing a multimodal semantic graph. The relationships between modalities are explicitly modeled as edges. Traditional graph convolution uses a fixed adjacency matrix, which cannot reflect the dynamic changes in semantics as the network deepens. In this embodiment, after each graph convolution layer, the cosine similarity is recalculated based on the updated node features, and the adjacency matrix is ​​updated, allowing the graph structure to adjust with semantic evolution. For example, modal pairs with low initial similarity may become similar in the deeper semantic space, thus automatically establishing connections. Simultaneously, a multi-head graph attention processing method is adopted to learn independent weights for intermodal dependencies (capturing semantic interactions between different modalities), automatically focusing on cross-modal relationships that contribute more to the task. After two layers of dynamic graph convolution, the features of each node incorporate the semantic information of its neighboring nodes, achieving intermodal information complementarity and high-order association modeling. The output graph-enhanced feature vector is more discriminative and semantically richer than the original features.

[0047] Finally, the modality quality assessment branch is used to quantify the input quality of each modality from three dimensions: clarity, completeness, and consistency. The scores of the three dimensions are averaged to obtain a comprehensive quality score, which represents the modality's credibility. Softmax with a temperature coefficient is used to convert the score into fusion weights. The temperature coefficient amplifies the weight difference between high-quality and low-quality modalities, giving high-quality modalities higher weights and suppressing low-quality modalities. For modalities with scores below a threshold (e.g., 0.3), their weights are directly set to zero and renormalized, effectively discarding unusable modalities. During the final weighted fusion, high-weight graph augmentation features contribute more to the final features. This allows the model to extract effective information from other high-quality modalities even when faced with single-modality quality degradation (e.g., camera occlusion, audio noise, text truncation). Compared to fixed-weight or simple average fusion, dynamic fusion of modality quality is achieved, improving the system's robustness in real-world complex scenarios. Attached Figure Description

[0048] Figure 1This is a schematic diagram of the main process of a multimodal data semantic fusion method. Detailed Implementation

[0049] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0051] Example 1

[0052] like Figure 1 As shown, Embodiment 1 of the present invention provides a multimodal data semantic fusion method, including the following operation steps:

[0053] Step S10: Extract modal independent features and cross-modal common features from the input multi-modal data respectively, and realize the collaborative encoding and decoding (i.e., enhancement processing) of common features and independent features through multi-head attention interaction processing to obtain the enhanced multi-modal feature vector (one vector for each modality).

[0054] Step S20: Stack the enhanced multimodal feature vectors to form a multimodal complementary feature matrix (i.e., a matrix formed by stacking the feature vectors of each modality, where rows correspond to modalities and columns correspond to feature dimensions, and the complementarity of modalities is represented by the similarity between rows). Use each row of the multimodal complementary feature matrix (i.e., the feature vector of each modality) as a graph node, and mine the implicit semantic associations (i.e., higher-order feature correlations) between modalities through a dynamic graph convolutional network (i.e., recalculate the adjacency matrix based on the current node features after each graph convolution layer, so that the graph structure evolves dynamically with semantics) to obtain graph-enhanced feature vectors (one vector is output for each modality).

[0055] The adjacency matrix of the dynamic graph convolutional network is dynamically updated according to the node features;

[0056] Step S30: Based on the modal quality evaluation branch (i.e., a lightweight evaluation module independent of the main network, which calculates the input quality score of each modality from three dimensions: clarity, completeness, and consistency), the quality of each modality input is evaluated in real time. According to the evaluation results, the fusion weights of each modality are assigned, and the enhanced feature vectors of each graph are weighted and fused to obtain the final feature vector. .

[0057] In subsequent processing, Inputting the data into a fully connected classifier (reducing dimensionality from 256 to 2) yields sentiment labels, while simultaneously... It serves as an attribute vector storage for entities in a graph database, supporting cross-modal semantic retrieval.

[0058] It should be noted that the embodiments of this application simultaneously extract modal-independent features (using the optimal encoder for each of text, image, and audio) and cross-modal common features (shared parameter encoders mapped to a unified semantic space), preserving modal-specific information (such as text syntax and image texture) as well as semantics shared between modalities (such as sentiment). Through multi-head attention interaction processing, cross-attention calculation is performed with common features as queries and independent features as keys / values, enabling common semantics to guide the enhancement of independent features, while noise or irrelevant information in independent features is suppressed by attention weights. Residual connections and layer normalization preserve the original common features, avoiding overfitting. The final output enhanced feature vector contains both cross-modal aligned common semantics and retains the discriminative information specific to each modality. Compared to direct concatenation or simple weighted fusion, this achieves fine-grained alignment of semantics between modalities, reduces alignment errors, and provides higher-quality node features for subsequent graph structure modeling.

[0059] Next, the enhanced feature vectors of each modality are stacked into a matrix, with each modality serving as a graph node, constructing a multimodal semantic graph. This graph structure explicitly models the relationships between modalities as edges, facilitating relationship propagation using graph convolution. Traditional graph convolution uses a fixed adjacency matrix (e.g., based on initial similarity), which cannot reflect semantic changes as the network depth increases. In this embodiment, after each graph convolution layer, the cosine similarity between nodes is recalculated based on the updated node features, and the adjacency matrix is ​​updated, allowing the graph structure to adjust with semantic evolution. For example, modality pairs with low initial similarity can... It can become similar in deep semantic space, thus automatically establishing connections; it adopts a multi-head graph attention processing method, which learns independent attention weights for inter-modal dependencies, and captures the semantic interactions between different modalities (such as text dogs and dogs in images); the attention processing method can automatically focus on cross-modal relationships that contribute more to the current task; after two layers of dynamic graph convolution, the features of each node are integrated with the semantic information of its neighboring nodes (other modalities), realizing information complementarity and high-order association modeling between modalities; the output graph-enhanced feature vector is more discriminative and semantically richer than the original features;

[0060] Finally, the modal quality assessment branch is used to quantify the quality of each modal input from three dimensions (sharpness, integrity, and consistency). Sharpness assessment (Laplacian variance, signal-to-noise ratio) reflects whether the image / audio is blurry or noisy; integrity assessment (missing percentage) detects whether the data is truncated or occluded; and consistency assessment detects whether the modalities are aligned. These metrics do not require additional training and are computationally efficient.

[0061] The average score across the three dimensions yields a comprehensive quality score, which characterizes the credibility of the modal input. A softmax algorithm with a temperature coefficient is used to convert the quality score into fusion weights. The temperature coefficient amplifies the weight difference between high-quality and low-quality modalities, meaning high-quality modalities receive higher weights while low-quality modalities have lower weights. For modalities with quality scores below a threshold (e.g., 0.3), their weights are reset to 0 and renormalized, effectively discarding these completely unusable modalities to prevent them from interfering with the fusion results. During the final weighted fusion, high-weight graph-enhanced feature vectors contribute more to the final features, while low-weight vectors contribute less. This allows the model to extract effective information from other high-quality modalities even when faced with single-modal quality degradation (e.g., camera obstruction, high audio background noise, or truncated text), maintaining the accuracy of the overall semantic representation. Compared to fixed-weight or simple average fusion, this achieves dynamic fusion of modal quality, improving the system's robustness in real-world complex scenarios.

[0062] Specifically, in step S10, modal independent features and cross-modal common features are extracted from the input multi-modal data, and the co-encoding and decoding of common features and independent features are achieved through multi-head attention interaction processing to obtain the enhanced multi-modal feature vector, including the following operation steps:

[0063] Step S11: Preprocess the input modal data to obtain preprocessed modal data;

[0064] The multimodal data includes raw text data, raw image data, and raw audio data; the preprocessed modal data includes preprocessed text data, preprocessed image data, and preprocessed audio data.

[0065] It should be noted that, in the above embodiments of this application, preprocessing the input multimodal data refers to preprocessing the input original text data, original image data, and original audio data respectively to obtain preprocessed text data, preprocessed image data, and preprocessed audio data. Specifically, for the original text data, special characters are removed and words are segmented to obtain preprocessed text data (lexical sequence); for the original image data, it is scaled to 224×224 pixels and the pixel values ​​are normalized to the [0,1] interval to obtain preprocessed image data (224×224×3 tensor); for the original audio data, it is resampled to 16kHz and the Melspectrogram is extracted to obtain preprocessed audio data (time-frequency tensor).

[0066] Step S12: Use a modal independent encoder to extract modal independent feature vectors from the preprocessed modal data;

[0067] The modality-independent feature vectors include text modality-independent feature vectors, image modality-independent feature vectors, and audio modality-independent feature vectors;

[0068] It should be noted that in the above embodiments of this application, for the preprocessed text data, a BERT-base encoder (12 layers, 768 dimensions) is used to extract and output text modality-independent feature vectors; for the preprocessed image data, a ResNet50 encoder (pre-trained on ImageNet) is used to extract and output image modality-independent feature vectors; for the preprocessed audio data, a VGGish encoder (pre-trained on AudioSet) is used to extract and output audio modality-independent feature vectors.

[0069] Step S13: Using a cross-modal encoder with shared parameters, the preprocessed modal data are mapped to a unified common semantic space, and cross-modal common feature vectors are extracted.

[0070] The cross-modal encoder with shared parameters is a 2-layer Transformer structure (4 attention heads, 1024 dimensions of the feedforward network), and the parameters are completely shared between different modalities.

[0071] The cross-modal common feature vector includes text cross-modal common feature vector, image cross-modal common feature vector, and audio cross-modal common feature vector;

[0072] Step S14: After concatenating the modal independent feature vectors and cross-modal common feature vectors of each modal data respectively, the enhanced multimodal feature vectors are obtained through multi-head attention interaction processing (using a cross-attention structure);

[0073] The enhanced multimodal feature vectors include text-enhanced multimodal feature vectors, image-enhanced multimodal feature vectors, and audio-enhanced multimodal feature vectors.

[0074] It should be noted that in the above embodiments of this application, the text modality is used as an example for processing. First, concatenation is performed. ;in, It is a concatenated text vector obtained by concatenating the text modal independent feature vectors and the text cross-modal common feature vectors; These are text modality-independent feature vectors. It is the common feature vector across modalities of the text;

[0075] Then, cross-attention calculation is performed to obtain the text cross-modal common feature vector. As a query, it uses text modality-independent feature vectors. As key and value,

[0076] The formula is: ;

[0077] in The dimensions of the projection matrix are all (here) The multi-head algorithm has 4 heads. After each head independently calculates its attention, the outputs of all heads are concatenated along the feature dimension to obtain Concat( ,..., )∈ Then, through the output projection matrix ∈ (Here, 768 is the target dimension, consistent with the dimension of the modality-independent feature vector.) A linear transformation is performed, residual connections are added, and finally, layer normalization is applied to obtain the enhanced multimodal feature vector; where... In this embodiment, the dimension of the key vector K is... (Consistent with the dimension of the cross-modal common feature vector); , and All are learnable linear projection matrices, and the parameters are not shared between different modalities (each modality has three independent sets of projection matrices) to preserve modality specificity. The dimensions of the query, key, and value after projection are all d=512.

[0078] Residual connections and layer normalization processing, i.e., attention output and original... The vectors are added together and then normalized to obtain the text-enhanced multimodal feature vector, i.e. (Dimensionality restored to be the same as that of the modality-independent feature vectors);

[0079] Similarly, the same operation is performed on the image modality and the audio modality to obtain the multimodal feature vector after image enhancement. and audio-enhanced multimodal feature vectors .

[0080] Specifically, in step S20, the enhanced multimodal feature vectors are stacked to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node. The implicit semantic associations between modalities are mined through a dynamic graph convolutional network to obtain graph-enhanced feature vectors. This includes the following steps:

[0081] Step S21: Map each enhanced multimodal feature vector to the same dimension to obtain the initial feature vector of each modality; stack the initial feature vectors of each modality row by row to form a multimodal complementary feature matrix;

[0082] It should be noted that in the above embodiments of this application, the enhanced multimodal feature vectors output in step S10 have different dimensions, i.e. , , Therefore, the three vectors should be mapped to a unified dimension, i.e., d=256, through independent linear projection layers.

[0083] Specifically, its mapping methods include text projection, image projection, and audio projection;

[0084] Specifically, text projection is ,in Output That is, the initial feature vector of the text modality;

[0085] Image projection is ,in Output That is, the initial feature vector of the image modality;

[0086] Audio projection is ,in Output That is, the initial feature vector of the audio modality;

[0087] Stack the three projected vectors row by row to form a multimodal complementary feature matrix. , Each row of the matrix corresponds to the initial eigenvector of a modal node.

[0088] Step S22: Using multimodal complementary feature matrices The row vectors are the node features. The cosine similarity between any two node features is calculated to form the initial adjacency matrix. ;

[0089] The elements of the initial adjacency matrix ,for ;for ;

[0090] It should be noted that in the above embodiments of this application, a similarity threshold is set. (A threshold of 0.5 is an empirical value used to filter weak connections, preserve strong intermodal semantic relationships, and reduce noise propagation in graph convolution. This threshold can be adjusted within the range of 0.3 to 0.7 depending on the task. A higher threshold results in a sparser graph structure, which may lose weak but still meaningful cross-modal relationships; a lower threshold results in a denser graph structure, which may introduce noise. This example uses 0.5, but in practical applications, it can be adjusted within the range of [0.3, 0.7] according to the task.) The value is set to 0, and the rest are retained; in this example, the similarity between text node features and image node features is 0.62, the similarity between text node features and audio node features is 0.31, and the similarity between image node features and audio node features is 0.28, therefore... In the text-image boundary, only the edge weight is 0.62, while the rest are 0.

[0091] In the above embodiments of this application, although the number of graph nodes is small (3), the feature vector dimension of each node is 256, which is a high-dimensional representation. The role of graph convolution is to learn the relational weights between modalities in the high-dimensional feature space, rather than to reduce the dimension. The combination of high-dimensional features and low graph size will not introduce inconsistency in dimensions, because in the operation of graph convolution, the adjacency matrix is ​​a dimensionless similarity, the feature H is a normalized vector, and the weight W is a learnable transformation matrix, and the overall operation maintains mathematical consistency.

[0092] Step S23: Use a 2-layer dynamic graph convolutional network combined with the initial adjacency matrix to perform convolutional propagation on the multimodal complementary feature matrix to obtain the target multimodal complementary feature matrix. ;

[0093] The two-layer dynamic graph convolutional network includes a first graph convolutional layer and a second graph convolutional layer;

[0094] It should be noted that in the above embodiments of this application, the number of graph nodes is relatively small (only 3 modal nodes). The design of 2-layer graph convolution is intended to allow each node to aggregate the semantic information of its second-order neighbors (i.e., the indirect relationship transmitted through intermediate nodes). To avoid losing modal uniqueness due to excessive smoothing of node features caused by too many layers, this scheme uses only 2 layers of convolution, resulting in a limited receptive field. The adjacency matrix is ​​dynamically updated after each layer to allow the graph structure to evolve rather than remain fixed. In the graph attention processing method, differentiated weights are learned for the relationships between different modalities to preserve modal specificity. Experiments show that the 2-layer structure has the best performance when the number of nodes is ≤5.

[0095] Step S24: In the target multimodal complementary feature matrix Based on this, a multi-head graph attention processing method is applied to learn independent attention weights for inter-modal dependencies, thereby obtaining graph-enhanced feature vectors for each modality node;

[0096] Specifically, assuming a total of Each attention head, single-head output dimension For the first Each attention head has learnable parameters including a weight matrix. Attention vector ;

[0097] The specific calculation steps are as follows:

[0098] Calculate the node under the k-th head. Its neighboring nodes Attention coefficient

[0099] Neighbors set Includes all other modal nodes;

[0100] Calculate attention score ;

[0101] here, and It is a node and nodes eigenvectors, symbols Indicates a splicing operation;

[0102] Calculate the normalized attention coefficient ;

[0103] (2) Calculate the nodes output by this head. New features

[0104] The calculated attention coefficients are used to weight the features of neighboring nodes. ;

[0105] (3) Each node All The output of the size is concatenated along the feature dimension.

[0106] The final feature representation of a node is obtained by concatenating the outputs of all attention heads. ;

[0107] (4) By outputting the linear transformation matrix The final graph-enhanced feature vector is obtained;

[0108] (5) Finally, perform a linear transformation on the concatenated feature vectors. ;

[0109] Through these steps, nodes can be obtained. The final feature representation This will be used for subsequent tasks, such as node classification or graph representation learning;

[0110] For the three nodes of text, image, and audio, the calculations are as follows: ;

[0111] For a graph node i, its neighbor set N(i) contains all other modal nodes except itself (because in this scheme each modality corresponds to a node, and the nodes are fully connected), and the attention weight is calculated only for the relationship between modalities.

[0112] Specifically, in step S23, a two-layer dynamic graph convolutional network is used in conjunction with the initial adjacency matrix to perform convolutional propagation on the multimodal complementary feature matrix to obtain the target multimodal complementary feature matrix. The operation includes the following steps:

[0113] Step S231: Use the first graph convolutional layer combined with the initial adjacency matrix to process the multimodal complementary feature matrix. Propagation is performed to obtain the second multimodal complementary feature matrix;

[0114] Specifically, this refers to using the first graph convolution formula of the first graph convolutional layer, combined with the initial adjacency matrix and the multimodal complementary feature matrix, to calculate the second multimodal complementary feature matrix, i.e. ;

[0115] in, It is the identity matrix. This is the initial adjacency matrix; for The first degree matrix, its diagonal elements ; This is the first learnable weight matrix; Here, it is the ReLU activation function; Represents the degree matrix Each diagonal element is raised to the power of negative 1 / 2, i.e. (If the diagonal element is 0, keep it 0), and all off-diagonal elements are 0.

[0116] Step S232: Calculate the second adjacency matrix based on the second multimodal complementary feature matrix (the analysis and calculation process is the same as that in S22 above, and will not be repeated here).

[0117] Step S233: Use the second graph convolutional layer combined with the second adjacency matrix to apply the second multimodal complementary feature matrix. Propagation is performed to obtain the target multimodal complementary feature matrix;

[0118] Specifically, this refers to using the second graph convolution formula of the second graph convolutional layer, combined with the second adjacency matrix and the second multimodal complementary feature matrix, to calculate the target multimodal complementary feature matrix. (i.e., the aforementioned target multimodal complementary feature matrix) ),Right now ;

[0119] in, ; for The second degree matrix; The second learnable weight matrix; output As the node feature matrix after graph convolution propagation;

[0120] Specifically, in step S30, based on the modal quality assessment branch, the input quality of each modality is evaluated in real time, fusion weights for each modality are assigned according to the evaluation results, and the enhanced feature vectors of each graph are weighted and fused to obtain the final feature vector, including the following operation steps:

[0121] Step S31: Evaluate the quality of the preprocessed text data, preprocessed image data, and preprocessed audio data output from step S11, and output the quality score for each modality. ;

[0122] It should be noted that, in the above embodiments of this application, the quality assessment includes sharpness assessment, integrity assessment, and consistency assessment, and the specific calculation methods are as follows:

[0123] The sharpness assessment includes calculating the Laplacian variance of the preprocessed image data and normalizing it to obtain an image sharpness score. (Calculate the Laplacian variance of the preprocessed image data) Use the predefined reference maximum value (For example, the maximum variance in the experiment) is linearly normalized. Calculate the signal-to-noise ratio of the preprocessed audio data. (unit The following normalization is adopted, namely ,in If the value exceeds the upper or lower bound, it is truncated; if it cannot be pre-calculated, the Sigmoid function can be used for normalization, i.e. Map values ​​to ,in (For scale parameters); calculate the signal-to-noise ratio (SNR) of the preprocessed audio data, and normalize it to obtain the audio clarity score. The default clarity for text modality is 1.0.

[0124] Integrity assessment includes calculating the text missing percentage = number of truncated characters / total number of characters, and a text integrity score. Text missing percentage; calculate image occlusion percentage = occlusion area / total area, image integrity score. Image occlusion ratio; calculate audio silence ratio = number of silent frames / total frames, audio integrity score. Audio mute ratio;

[0125] Consistency assessment includes a default consistency score of 0.8 for non-strictly registered multimodal data; a default consistency score of 0.8 for strictly registered multimodal data, calculated based on the temporal offset; and a default consistency score of 0.8 for strictly registered multimodal data. The tolerance threshold is a preset maximum allowable offset, set according to the data sampling rate. For example, when the video frame rate is 25fps, the tolerance threshold can be set to 0.04 seconds (corresponding to 1 frame); when the audio sampling rate is 16kHz, the tolerance threshold can be 0.0625 seconds (corresponding to 1000 sampling points). If the offset is 0 (i.e., if there is no timing information), the score is 1; if the offset exceeds the tolerance threshold, the score is 0.

[0126] Modal quality score Take the arithmetic mean of the sharpness score, integrity score, and consistency score. ;

[0127] It should be noted that modal registration is not strictly performed in this example, and the default consistency score is used. Final modal quality scores ;

[0128] In this example, text quality scoring Image quality score Audio quality rating ;

[0129] Step S32: Calculate the fusion weights based on the modal quality score combined with the softmax function with a temperature coefficient, i.e.

[0130] Where α is a temperature coefficient, with a value greater than 0; the temperature coefficient α is used to control the degree of influence of quality score on fusion weight. When α>1, the weight of high-quality modes is further amplified, and the weight of low-quality modes is suppressed; when α=1, it is the standard softmax; when α<1, the weight tends to be uniform. In this embodiment, α=2 is taken as an empirical value, which can be adjusted in the range of [1,5] according to the actual modal quality fluctuation. If the modal quality difference is significant, α can be increased; if the quality of each modality is stable, α can be decreased.

[0131] If there is an arbitrary modal quality score If the quality score is below a preset low-quality threshold (e.g., 0.3, an empirical value indicating that the modality information is severely distorted or almost unusable; when the modality quality score is below this value, the modality information is considered severely distorted or contains almost no effective semantics and should be completely discarded; in this example, it is set to 0.3, which can be set according to the distribution of quality scores in the dataset, generally ranging from [0.2, 0.4]), then the fusion weight of that modality is... Set to 0 and renormalize the remaining modal weights; in this example, there are no low-quality modes, and the calculated values ​​are... ;

[0132] Step S33: Weight and fuse the enhanced feature vectors of each image according to their corresponding fusion weights to obtain the final feature vector. ;

[0133] ;

[0134] Should This is the final feature vector;

[0135] In summary, the multimodal data semantic fusion method, system, and apparatus proposed in this invention first achieve cross-modal semantic alignment through common and independent encoding and decoding, then model multimodal features as a graph structure for relation propagation through dynamic graph convolution, and finally use quality assessment for weighted fusion to output a unified feature vector. This vector can be stored in a graph database to support semantic similarity retrieval, and can also be used as an entity attribute to participate in graph relation traversal. This solves the defect that semantic similarity and relation association cannot be queried in a unified manner, and at the same time makes up for the problem that graph databases are not capable of semantic alignment of multimodal data.

[0136] In further processing, the original data of different modalities are standardized, scaled, and resampled to eliminate differences in the original data dimensions and formats, providing comparable input for subsequent unified feature extraction and indirectly supporting cross-modal alignment. Dedicated encoders such as BERT, ResNet50, and VGGish are used to extract independent feature vectors for text, images, and audio respectively, preserving the unique discriminative information of each modality (such as text syntax, image texture, and audio timbre), avoiding the loss of modality-specific details from simple common features, thereby improving the accuracy of semantic alignment. A Transformer encoder with shared parameters maps each modality to a unified 512-dimensional common semantic space, making the data of different modalities mathematically comparable and solving the alignment difficulty of graph databases' inability to natively understand multimodal semantics. Cross-attention is performed using common features as queries and independent features as keys / values, followed by residual connections and layer normalization, allowing common semantics to guide the enhancement of independent features while suppressing noise in independent features, outputting enhanced features that contain both common semantics and retain private information, providing high-quality nodes for subsequent graph modeling.

[0137] On the other hand, enhanced features of different dimensions are mapped to a unified 256-dimensional matrix through linear projection, and then stacked row by row to form a multimodal complementary feature matrix. This allows text, image, and audio modalities to participate in graph operations as graph nodes, opening up a conversion channel between unstructured vectors and structured graph nodes. Cosine similarity between nodes is calculated, with the diagonal set to 0 and a threshold of 0.5 to filter weak connections. The initial relationship between modalities is explicitly modeled using similarity, providing a propagable edge structure for graph convolution and directly converting semantic similarity into graph edge weights. After each convolution layer, the cosine similarity is recalculated based on the updated node features, and the adjacency matrix is ​​updated, so that the graph structure is adjusted with semantic evolution. Modal pairs with low initial similarity may become similar in deeper layers and automatically establish connections, thereby capturing higher-order implicit associations and solving the defect that fixed graph structures cannot reflect dynamic semantic changes. Independent attention weights are learned for intermodal dependencies, automatically focusing on cross-modal relationships that contribute more to the current task, enhancing the semantic richness and discriminative power of graph enhanced feature vectors, and thus improving the efficiency of graph databases in relational reasoning.

[0138] Furthermore, each node further integrates the evolved neighbor semantics to achieve higher-order association modeling. The output graph-enhanced features are better able to represent the implicit semantic associations between modalities than the original features, thereby improving the graph database's reasoning ability for multimodal data.

[0139] On the other hand, the quality scores of each modality are calculated from three dimensions: clarity (Laplacian variance, signal-to-noise ratio), completeness (missing ratio), and consistency (temporal offset or default value). This quantifies the credibility of modal inputs without additional training, providing a basis for dynamic fusion and solving the problem of fusion failure caused by single-modal quality fluctuations (such as camera obstruction or high audio noise) in real-world scenarios. The quality scores are converted into fusion weights using a softmax function with a temperature coefficient. Low-quality modal weights are reset to zero, amplifying the contribution of high-quality modalities and suppressing or even discarding unusable modalities. This ensures that the final fused features are not affected by inferior inputs, improving the system's robustness. The enhanced feature vectors of each graph are weighted and summed to output a final fused feature vector with a fixed dimension (256 dimensions). This vector can be directly stored in a graph database as entity attributes and supports downstream tasks such as sentiment analysis using a fully connected classifier, truly achieving efficient querying of semantic similarity and relational associations within a unified framework.

[0140] This solution also provides a multimodal data semantic fusion system, including a feature enhancement module, a graph enhancement module, and a fusion module;

[0141] The feature enhancement module is used to extract modal independent features and cross-modal common features from the input multimodal data. It achieves collaborative encoding and decoding of common features and independent features through multi-head attention interaction processing to obtain the enhanced multimodal feature vector.

[0142] The graph augmentation module is used to stack the augmented multimodal feature vectors to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node. The implicit semantic associations between modalities are mined through a dynamic graph convolutional network to obtain graph augmented feature vectors.

[0143] The fusion module is used to evaluate the quality of each modal input in real time based on the modal quality assessment branch, assign fusion weights to each modality according to the evaluation results, and weight and fuse the enhanced feature vectors of each graph to obtain the final feature vector.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data semantic fusion method, characterized in that, The following steps are included: Modal independent features and cross-modal common features are extracted from the input multimodal data respectively. The co-encoding and decoding of common features and independent features are achieved through multi-head attention interaction processing to obtain the enhanced multimodal feature vector. The enhanced multimodal feature vectors are stacked to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node. The implicit semantic associations between modalities are mined through a dynamic graph convolutional network to obtain graph-enhanced feature vectors. Based on the modal quality assessment branch, the input quality of each modality is evaluated in real time. The fusion weights of each modality are assigned according to the evaluation results, and the enhanced feature vectors of each graph are weighted and fused to obtain the final feature vector.

2. The multimodal data semantic fusion method according to claim 1, characterized in that, Modal-independent features and cross-modal common features are extracted from the input multi-modal data. A multi-head attention interaction processing method is used to achieve collaborative encoding and decoding of common and independent features, resulting in an enhanced multi-modal feature vector. The process includes the following steps: The input modal data is preprocessed to obtain preprocessed modal data; A modal-independent encoder is used to extract modal-independent feature vectors from the preprocessed modal data; A cross-modal encoder with shared parameters maps the preprocessed modal data to a unified common semantic space and extracts cross-modal common feature vectors. After concatenating the modal-independent feature vectors and cross-modal common feature vectors of each modal data, the enhanced multimodal feature vectors are obtained through multi-head attention interaction processing.

3. The multimodal data semantic fusion method according to claim 2, characterized in that, The multimodal data includes raw text data, raw image data, and raw audio data; the preprocessed modal data includes preprocessed text data, preprocessed image data, and preprocessed audio data. The modality-independent feature vectors include text modality-independent feature vectors, image modality-independent feature vectors, and audio modality-independent feature vectors; the cross-modal encoder with shared parameters is a 2-layer Transformer structure, and the parameters are completely shared between different modalities; The cross-modal common feature vector includes text cross-modal common feature vector, image cross-modal common feature vector, and audio cross-modal common feature vector; The enhanced multimodal feature vectors include text-enhanced multimodal feature vectors, image-enhanced multimodal feature vectors, and audio-enhanced multimodal feature vectors.

4. The multimodal data semantic fusion method according to claim 3, characterized in that, The enhanced multimodal feature vectors are stacked to form a multimodal complementary feature matrix. Each row of the multimodal complementary feature matrix is ​​used as a graph node. Implicit semantic associations between modalities are mined through a dynamic graph convolutional network to obtain graph-enhanced feature vectors. The process includes the following steps: The enhanced multimodal feature vectors are mapped to the same dimension to obtain the initial feature vectors of each modality; the initial feature vectors of each modality are stacked row by row to form a multimodal complementary feature matrix. Multimodal complementary feature matrix The row vectors are the node features. The cosine similarity between any two node features is calculated to form the initial adjacency matrix. A two-layer dynamic graph convolutional network is used in conjunction with an initial adjacency matrix to perform convolutional propagation on the multimodal complementary feature matrix to obtain the target multimodal complementary feature matrix; Based on the target multimodal complementary feature matrix, a multi-head graph attention processing method is applied to learn independent attention weights for intermodal dependencies, thereby obtaining graph-enhanced feature vectors for each modality node.

5. The multimodal data semantic fusion method according to claim 4, characterized in that, The two-layer dynamic graph convolutional network includes a first graph convolutional layer and a second graph convolutional layer.

6. The multimodal data semantic fusion method according to claim 5, characterized in that, A two-layer dynamic graph convolutional network is used in conjunction with an initial adjacency matrix to perform convolutional propagation on the multimodal complementary feature matrix to obtain the target multimodal complementary feature matrix. The steps include the following: The first graph convolutional layer is used in conjunction with the initial adjacency matrix to process the multimodal complementary feature matrix. Propagation is performed to obtain the second multimodal complementary feature matrix; The second adjacency matrix is ​​calculated based on the second multimodal complementary feature matrix. The second graph convolutional layer is used in conjunction with the second adjacency matrix to apply the second multimodal complementary feature matrix. Propagation is performed to obtain the target multimodal complementary feature matrix.

7. The multimodal data semantic fusion method according to claim 6, characterized in that, Based on the modal quality assessment branch, the input quality of each modality is evaluated in real time. Fusion weights are assigned to each modality according to the evaluation results, and the enhanced feature vectors of each graph are weighted and fused to obtain the final feature vector. The process includes the following steps: The quality of preprocessed text data, preprocessed image data, and preprocessed audio data is evaluated, and quality scores for each modality are output. The fusion weights are calculated based on modal quality scores combined with a softmax function with a temperature coefficient. The enhanced feature vectors of each image are weighted and fused according to their corresponding fusion weights to obtain the final feature vector.

8. The multimodal data semantic fusion method according to claim 7, characterized in that, The quality assessment includes sharpness assessment, integrity assessment, and consistency assessment; wherein, the sharpness assessment includes calculating the Laplacian variance of the preprocessed image data and normalizing it to obtain an image sharpness score; calculating the signal-to-noise ratio of the preprocessed audio data and normalizing it to obtain an audio sharpness score; the default sharpness of the text modality is 1.0; The integrity assessment processing operation includes a text integrity score of 1 - number of truncated characters / total number of characters; an image integrity score of 1 - occlusion area / total area; and an audio integrity score of 1 - number of silent frames / total frames. The consistency assessment processing operation includes setting the consistency score to 0.8 for non-strictly registered multimodal data; setting the consistency score to 0.8 based on the temporal offset for strictly registered multimodal data; and calculating the consistency score based on the temporal offset for strictly registered multimodal data.

9. A multimodal data semantic fusion system, characterized in that, It includes a feature enhancement module 10, a graph enhancement module 20, and a fusion module 30; The feature enhancement module 10 is used to extract modal independent features and cross-modal common features from the input multi-modal data respectively, and to achieve collaborative encoding and decoding of common features and independent features through multi-head attention interaction processing to obtain the enhanced multimodal feature vector; The graph enhancement module 20 is used to stack the enhanced multimodal feature vectors to form a multimodal complementary feature matrix, and use each row of the multimodal complementary feature matrix as a graph node to mine the implicit semantic associations between modalities through a dynamic graph convolutional network to obtain graph-enhanced feature vectors. The fusion module 30 is used to evaluate the quality of each modal input in real time based on the modal quality assessment branch, allocate fusion weights for each modality according to the evaluation results, and weight and fuse the enhanced feature vectors of each graph to obtain the final feature vector.

10. A multimodal data semantic fusion device, characterized in that, The device includes: A memory that stores computer programs that can run on a processor; A processor that, when executing the program, implements the steps of the method according to any one of claims 1 to 7; The processor is equipped with a graphics processing unit or a neural network processing unit to accelerate multi-head attention calculation and graph convolution operation.

Images