Enhanced multi-modal data semantic alignment method and system
By introducing a diffusion mechanism into multimodal contrastive learning, feature maps are constructed and information is propagated, solving the problem that traditional models struggle to capture deep semantic relationships in high-dimensional sparse feature spaces, and achieving higher accuracy and robustness in cross-modal alignment.
Patent Information
- Application Number
- CN202511720428.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-10
AI Technical Summary
Traditional contrastive learning models struggle to effectively capture deep semantic relationships between cross-modal samples, especially in high-dimensional sparse feature spaces, resulting in insufficient model alignment accuracy.
We employ a multimodal contrastive learning method based on diffusion mechanism. By constructing feature maps and simulating the information propagation process, we enhance the ability to capture deep semantic associations between cross-modal samples. We utilize the Laplacian matrix for iterative diffusion and similarity matrix updates, and combine graph theory algorithms to optimize feature relationships.
It significantly improves model alignment accuracy, effectively alleviates the sparsity problem of high-dimensional feature space, enhances the robustness and generalization ability of the model, and can more accurately capture deep semantic relationships, adapting to different datasets and task requirements.
Smart Images
Figure CN121502383A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of multi-modal learning, in particular to an enhanced multi-modal data semantic alignment method and system. BACKGROUND
[0002] The contrastive learning task aims to learn an embedding space in which semantically similar cross-modal samples (such as matched images and texts) are pulled closer, and dissimilar samples are pushed further apart. Since the features of modalities such as images and texts are usually in a high-dimensional and sparse space, the traditional contrastive learning model only learns by directly calculating the similarity (such as cosine similarity) between features, which is difficult to capture the deep and non-direct semantic association between cross-modal samples, thereby limiting the alignment accuracy of the model. Therefore, it is crucial to design a learning method that can go beyond the surface similarity calculation and mine the high-order relationship between features. SUMMARY
[0003] In view of the problems in the prior art, the present application provides an enhanced multi-modal data semantic alignment method and system, which adopts a multi-modal contrastive learning method based on diffusion mechanism semantic association. The learning network constructs a feature map and simulates the propagation process of information on the feature map to enhance the ability to capture deep semantic associations between multi-modal samples, effectively alleviating the sparsity problem of high-dimensional feature space of multi-modal images and texts, and significantly improving the alignment accuracy of the model.
[0004] In an embodiment of the present application, an enhanced multi-modal data semantic alignment method is provided, comprising the following steps: S1, pre-processing and feature extraction are performed on the input image and text data, and after normalization processing, an image feature matrix and a text feature matrix are obtained; S2, matrix inner product operation is performed on the image feature matrix and the text feature matrix to calculate an initial similarity matrix of cross-modal; S3, the initial similarity matrix is regarded as an adjacency matrix of a graph structure, wherein each image feature and text feature is regarded as a node in the graph structure; a degree matrix is calculated based on the adjacency matrix, and a Laplacian matrix is constructed by subtracting the adjacency matrix from the degree matrix; S4, taking the initial similarity matrix as the starting point, the Laplacian matrix is taken as a propagation operator, and iteration calculation of a preset diffusion step number is performed to obtain a final similarity matrix which is propagated and enhanced in deep association relationship; S5, taking the final similarity matrix as the prediction result of the matching relationship between the image features and the text features in the current batch, comparing the prediction result with the known real image-text matching relationship in the batch, and calculating a contrastive learning loss value; S6. Based on the calculated contrastive learning loss value, the parameters of the image and text feature extraction networks are iteratively updated through the backpropagation algorithm until the model converges, ultimately achieving deep semantic alignment of cross-modal features.
[0005] In this embodiment of the invention, an enhanced multimodal data semantic alignment system is also provided. The alignment system is designed to perform the multimodal data semantic alignment method and includes the following functional modules: The feature extraction module is equipped with a pair of parallel deep neural network encoders, which are used to receive the raw image and text data streams and convert them into a normalized image feature matrix and text feature matrix in a unified high-dimensional semantic feature space. The initial similarity calculation module receives the image feature matrix and text feature matrix from the feature extraction module, and calculates the initial similarity matrix representing the direct relationship between samples in real time through matrix operations. The Laplacian matrix construction module uses the initial similarity matrix as the adjacency matrix of the graph structure and constructs a Laplacian matrix that reflects the inherent manifold structure of the feature data through graph theory algorithms. The similarity diffusion module starts with the initial similarity matrix, uses the Laplacian matrix as the propagation operator, and executes a multi-step iterative diffusion algorithm to output a final similarity matrix that has been propagated and enhanced. The contrast optimization module receives the final similarity matrix and, in conjunction with the real intra-batch matching labels, calculates the contrastive learning loss. It then drives an automatic differentiation engine to generate the gradient of the loss with respect to the network parameters, and a co-optimizer guides the encoder parameters in the feature extraction module to perform end-to-end optimization updates.
[0006] Compared with the prior art, the beneficial effects achieved by the present invention specifically include: 1. This invention transforms the traditional point-to-point similarity comparison problem into an information propagation problem on the feature map structure by introducing a diffusion mechanism. This enables the model to not only capture directly related sample pairs, but also to discover indirectly related sample pairs through multi-step propagation, thereby uncovering deeper semantic associations.
[0007] 2. This invention effectively alleviates the sparsity problem of high-dimensional feature spaces through iterative similarity updates and aggregation. First, image and text features are extracted and an initial similarity matrix is calculated. Then, a Laplacian matrix is constructed to describe the connection structure between features, and this matrix is used to iteratively diffuse the initial similarity. Finally, the diffused and enhanced similarity matrix is used for comparative learning. The iterative diffusion process allows similarity information to flow "smoothly" along the feature manifold, enabling similar samples that were originally far apart in the sparse space to establish stronger connections, improving the robustness of learning and significantly enhancing the alignment accuracy of the model.
[0008] 3. The similarity diffusion process proposed in this invention is flexible and controllable. By adjusting the number of diffusion steps, the depth and breadth of the model's exploration of relationships between features can be controlled, enabling it to adapt to data and task requirements of varying complexity and achieving a balance between model performance and computational overhead.
[0009] 4. This invention can be widely applied to scenarios that require multimodal data alignment, such as cross-modal retrieval, image description generation, and visual question answering. By providing more accurate semantic alignment, it comprehensively improves the performance of downstream tasks. Attached Figure Description
[0010] Figure 1 This is a flowchart of a multimodal data semantic alignment method based on diffusion mechanism semantic association in an embodiment of the present invention.
[0011] Figure 2 This is a schematic diagram of the similarity diffusion module framework in an embodiment of the present invention. Detailed Implementation
[0012] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto. Example
[0013] This embodiment provides a semantic alignment method for multimodal data based on semantic association through diffusion mechanisms. It primarily enhances the semantic alignment of multimodal data by simulating the propagation of associations between features, and includes the following steps: S1. Multimodal feature extraction: Preprocess and extract features from the input image and text data, and then normalize them to obtain the image feature matrix and text feature matrix.
[0014] This step is crucial for mapping the original heterogeneous data to a unified embedding space rich in semantic information. A batch of input images and text data are preprocessed and then fed into independent image encoders and text encoders, respectively. This maps the original high-dimensional data to a shared latent embedding space, thereby extracting L2-normalized, dimensionally unified image and text feature matrices. Specifically, the steps include: S11. Data Preprocessing: For the input raw image data First, a series of standard image preprocessing operations are performed, such as random cropping, horizontal flipping, color dithering, and normalization data augmentation techniques, to improve the model's generalization ability and robustness. For the input raw text data T... Then, operations such as text segmentation, conversion to word embedding IDs, and padding are performed to ensure that all text sequences have the same length.
[0015] S12. Feature Encoding: The preprocessed image data is fed in batches into a pre-trained or de novo trained image encoder. In this context, the encoder can be a convolutional neural network architecture or a Transformer-based vision model. The image encoder processes each image... Convert to a high-dimensional image feature vector Simultaneously, the preprocessed text data is batch-fed into a text encoder. In this context, the encoder is typically a Transformer-based language model (such as BERT or RoBERTa). The text encoder processes each segment of text... Convert into a high-dimensional text feature vector .
[0016] S13. Feature Normalization: To ensure that features in the shared embedding space have a uniform scale, all extracted image feature vectors... and text feature vectors All are subjected to L2 norm normalization, that is: ; Thus, the normalized image feature matrix is obtained. and text feature matrix ,in It is the dimension of the shared embedding space. This normalization operation makes subsequent similarity calculations (such as cosine similarity) more direct and efficient.
[0017] S2. Initial similarity calculation: Perform a matrix inner product operation between the image feature matrix and the text feature matrix obtained in step S1 to calculate an initial similarity matrix across modalities.
[0018] This step aims to construct an initial, direct matching strength matrix between image and text features, serving as the basis for subsequent diffusion processes. The image feature matrix extracted in step S1 is multiplied by the text feature matrix to calculate the direct cosine similarity between all image-text pairs, thereby generating an initial, undiffused cross-modal initial similarity matrix. This initial similarity matrix preliminarily reflects the surface correspondence between samples. This step specifically includes: S21. Cross-modal feature projection: To ensure that the dimensions of image features and text features are consistent for similarity calculation, an additional linear projection layer can be introduced; if the dimensions of image features are... Dimensions of text features Not equal, that is Then, a linear projection layer maps image features and text features to a unified shared embedding space dimension. Then, normalization is performed to obtain the normalized image feature matrix and text feature matrix.
[0019] S22, Matrix Multiplication Calculation: Multiplying the normalized image feature matrix... With text feature matrix transpose Perform matrix multiplication. This actually calculates all the values in the batch. The cosine similarity between the image and text yields an initial cross-modal similarity matrix without any propagation or adjustment. : ; in Representing the Image and the first The initial cosine similarity between the text segments. The elements of this initial similarity matrix are between -1 and 1. To ensure the non-negativity assumption of the subsequent Laplace matrix, the initial similarity matrix can be... value range mapped to In this embodiment, the initial similarity matrix is assumed to be... Appropriate adjustments have been made to ensure non-negativity, and higher values represent higher similarity.
[0020] S3. Laplacian Matrix Construction: The initial similarity matrix generated in step S2 is used as the adjacency matrix of the graph structure, where each image feature and text feature is considered a node in the graph structure. Based on this adjacency matrix, the degree matrix is first calculated, and then the Laplacian matrix describing the feature graph structure and connectivity is constructed by subtracting the adjacency matrix from the degree matrix.
[0021] This step forms the basis of the diffusion mechanism. The core of constructing the Laplacian matrix lies in transforming the discrete set of feature points into a structured graph representation. This process aims to utilize the mathematical tools of graph theory to explicitly model higher-order relationships between features. The degree matrix is constructed as a diagonal matrix, where each element on the diagonal is exactly equal to the sum of all elements in the corresponding row or column of the initial similarity matrix. Physically, this represents the centrality or total connectivity strength of each feature node (image or text feature node) within the graph structure.
[0022] In other words, this step abstracts the discrete set of feature points into a graph structure and computes its Laplacian matrix, which encodes the connectivity and structural information of the graph representation.
[0023] To improve the robustness and convergence speed of the diffusion process, this step first performs symmetric normalization on the initial similarity matrix before constructing the Laplacian matrix. This operation effectively adjusts the weights of edges in the graph structure, balancing the influence of nodes with higher degrees (i.e., "center" nodes similar to many other nodes) with nodes with lower degrees, thereby ensuring smoother information propagation in the graph structure and avoiding numerical instability or gradient propagation problems caused by excessive differences in node degrees.
[0024] The detailed construction process of the symmetric normalized Laplace matrix in step S3 includes: S31. Construct the degree matrix: Based on the initial similarity matrix, calculate the degree matrix that describes the total connection strength of each node.
[0025] The initial similarity matrix obtained in step S2 View it as an adjacency matrix of a weighted undirected graph. This weighted undirected graph contains Image nodes and Total text nodes (in total) (a single node), or treat all image features and text features as a unified whole. A set of nodes. For simplicity, assume an initial similarity matrix. It already integrates image-image, text-text, and image-text similarity. Size matrix, where This represents the total number of samples.
[0026] Degree matrix It is a diagonal matrix whose diagonal elements Represents a node The sum of the weights of the edges connecting to all other nodes, i.e.: ; S32, degree matrix Perform each diagonal element The operation yields a degree-normalized factor matrix.
[0027] It can also be based on the degree matrix and the initial similarity matrix Calculate the nonnormalized Laplace matrix : .
[0028] S33. Multiply the degree normalization factor matrix with the initial similarity matrix from the left and right sides respectively, and subtract the product result from the identity matrix to complete the symmetric normalization of the initial similarity matrix. Finally, a symmetric normalized Laplace matrix with better properties is obtained for subsequent iterative diffusion processes.
[0029] In other words, to achieve stable and effective similarity propagation on graph structures, this embodiment employs a symmetric normalized Laplacian matrix. Its construction process is as follows: First, calculate the inverse square root matrix of the degree matrix D. This is a diagonal matrix with diagonal elements of 1. Then, the final symmetric normalized Laplace matrix is calculated using the following formula. : ; in It is a similarity matrix to the initial similarity matrix An identity matrix of the same size. This normalization operation effectively balances the influence of nodes of different degrees in the graph structure, ensuring that similarity information propagates more evenly and stably on the graph structure, and avoiding numerical instability.
[0030] S4. Iterative similarity diffusion: Starting from the initial similarity matrix in step S2, the Laplace matrix constructed in step S3 is used as the propagation operator to perform iterative calculations for a preset number of diffusion steps, resulting in a final similarity matrix that propagates and enhances the deep association relationship.
[0031] In each iteration, the similarity information of the current node is propagated to its neighboring nodes according to the graph structure and weighted and fused, thereby gradually smoothing the similarity distribution and strengthening the connection between sample pairs with potential semantic associations.
[0032] This step is the core mechanism of this invention. Through multiple iterations, it simulates the propagation and aggregation of similarity information on the feature space graph structure, thereby revealing and strengthening the deep and indirect relationships between features. The iterative diffusion process is essentially an information propagation or message passing mechanism on the feature graph structure. This mechanism, through iterative updates, allows the feature representation of each node to not only depend on itself but also aggregate information from neighboring nodes. This enables sample pairs with initially low similarity but indirectly connected through several high-similarity nodes to establish stable associations, effectively overcoming the sparsity problem of high-dimensional feature spaces.
[0033] To ensure the stability and flexibility of the diffusion process, this step introduces an adjustable diffusion coefficient during the iterative diffusion process. This diffusion coefficient is used to precisely control the fusion ratio of newly propagated neighborhood information and the original information of the node itself during each iteration update. It aims to prevent the similarity of all nodes from becoming uniform due to over-diffusion, thereby preserving the uniqueness of the samples and achieving a balance between global association mining and local feature preservation.
[0034] In this embodiment, the iterative diffusion process of the diffusion coefficient is introduced, specifically including the following steps: S41. Initialize diffusion state: Use the initial similarity matrix obtained in step S2. This serves as the initial state for the iterative diffusion process and retains the original information throughout. The initial state, i.e., the state of the 0th iteration, is denoted as... .
[0035] S42, Single-step iterative diffusion: In each iteration, the similarity matrix of the current state is transformed by the Laplacian matrix to obtain the neighborhood aggregation information after one-step propagation; using the preset diffusion coefficient as weight, the obtained neighborhood aggregation information is linearly weighted and fused with the initial similarity matrix to obtain the output result of the current iteration step.
[0036] In each iteration of calculation China (from) arrive ), current similarity matrix Based on the symmetric normalized Laplace matrix constructed in step S3 The update is performed. This embodiment uses the following iterative diffusion formula: ; in, Indicates the first The similarity matrix is updated after the next iteration; It is the identity matrix; It is a key diffusion coefficient, typically ranging from 0.5 to 0.9; It is considered a propagation matrix that performs aggregation of local similarity information on the graph structure; This ensures that the original, direct similarity information is not completely lost during the diffusion process, but is retained and influenced with a certain weight.
[0037] diffusion coefficient Controlling each step of the diffusion process, the newly propagated neighborhood information and the original information The fusion ratio. Smaller. A larger value makes diffusion more conservative, preserving more of the original information; Values allow for the integration of more neighborhood information, enabling wider dissemination.
[0038] S43. Repeated Iteration: The output of the previous iteration is used as the input of the next iteration. The linear weighted fusion process is repeated until the preset number of diffusion steps is reached, and the final similarity matrix is obtained.
[0039] Repeat the single-step iterative diffusion process of step S42, for a total of Number of iterations. Number of diffusion steps. This is an important hyperparameter, typically set based on the complexity of the dataset and the desired association depth (e.g., it can be set to 2 to 8 steps). More steps allow similarity information to propagate further, capturing more distant neighbor relationships.
[0040] When the preset number of diffusion steps is reached After that, the Output of the next iteration This is the final similarity matrix. The final similarity matrix By integrating global contextual information that has undergone multi-step propagation and aggregation, its element values can more accurately reflect the deep, indirect semantic relationships between image and text samples, effectively alleviating the sparsity of the high-dimensional feature space and strengthening potential matching relationships.
[0041] S5. Contrastive Loss Calculation: The final similarity matrix obtained after the multi-step diffusion in step S4, which contains richer global contextual information, is used as the model's prediction result for the image-text matching relationship within the current batch. This prediction result is compared with the known true image-text matching relationships within the batch, and the contrastive learning loss value is calculated accordingly. Here, image-text matching relationship refers to the matching relationship between image features and text features; the known true image-text matching relationships within the batch are specifically represented by the true labels of the known positive sample pairs (matched images and text) and negative sample pairs (non-matched images and text) within the batch.
[0042] This step aims to calculate the difference between the model's predictions and the true labels based on the final similarity matrix after diffusion enhancement, providing a basis for optimizing network parameters. Specifically, it includes: S51. Obtain positive and negative sample pairs: In each batch, for any image feature Text features that match it within the batch Those that do not match are considered positive samples, while other mismatched samples within the batch are considered positive samples. Text features Then it is considered a negative sample. Conversely, it is also considered a negative sample.
[0043] S52. Calculate the contrastive learning loss: Calculate the final similarity matrix... This serves as the model's prediction of the image-text matching relationship within the current batch. This embodiment employs the InfoNCE loss function, which encourages positive sample pairs to have a higher similarity than all negative sample pairs.
[0044] Image-to-text matching loss The calculation formula is as follows: ; in, Indicates the first The similarity between an image and its corresponding positive sample text. It is a learnable or preset temperature parameter used to adjust the smoothness of the similarity distribution and prevent the model from overfitting to easily classified samples. The denominator is the sum of exponential similarities calculated for all text samples, including both positive and negative samples.
[0045] For text-to-image matching loss The calculation formula is similar, specifically: ; in, Indicates the first Similarity between a text segment and its corresponding positive sample image.
[0046] S53. Calculate the total loss: the final total loss. Image-to-text loss and text-to-image loss The average value is used to ensure the balance of the two-way alignment. .
[0047] S6. Network parameter optimization: Based on the contrastive learning loss value calculated in step S5, the parameters of the image and text feature extraction networks are iteratively updated through the backpropagation algorithm until the model converges, ultimately achieving deep semantic alignment of cross-modal features.
[0048] Using the gradient backpropagation algorithm, the gradients of the contrastive learning loss with respect to the network parameters of the image encoder and text encoder are calculated, and the network parameters are iteratively updated using an optimizer (such as Adam). These steps are repeated until the model's performance on the validation set reaches the convergence criterion, ultimately achieving deep semantic alignment of cross-modal features.
[0049] This step aims to continuously improve the performance of feature extraction and diffusion alignment by adjusting the trainable parameters of the model based on the contrastive learning loss value and by optimizing the algorithm.
[0050] S61. Gradient Calculation: Calculate the total loss based on step S5. We calculate the gradient of the contrastive learning loss with respect to the network parameters of the image encoder and text encoder.
[0051] S62. Parameter Update: Optimizers (such as the Adam optimizer, SGD optimizer, or AdamW optimizer) iteratively update the network parameters of the image encoder and text encoder based on the calculated gradients. Learning rate schedulers (such as cosine annealing learning rate schedulers) can be used to dynamically adjust the learning rate for more stable training and better convergence.
[0052] S63. Model Convergence: Repeat steps S1 to S6 until the model's cross-modal retrieval performance on the predefined validation set reaches the convergence criterion (i.e., performance no longer significantly improves) or reaches the preset maximum training period. During this process, the model parameters will be continuously optimized, so that the similarity matrix enhanced by the diffusion mechanism can more accurately reflect the real semantic matching relationship, thereby completing the deep alignment learning of cross-modal features.
[0053] Based on the same inventive concept, this embodiment also provides a multimodal data semantic alignment system based on diffusion mechanism semantic association. This system includes multiple cooperating modules capable of efficiently executing the aforementioned multimodal data semantic alignment method. Specifically, it includes the following modules: Feature extraction module: Internally configured with a pair of parallel deep neural network encoders, namely an image encoder and a text encoder. This module is used to receive raw multimodal data, that is, to receive raw image and text data streams, perform data preprocessing and feature encoding, and transform the raw image and text data streams into normalized feature matrices in a unified-dimensional semantic feature space.
[0054] Initial similarity calculation module: This module receives the normalized feature matrix from the image and text data streams from the feature extraction module, performs matrix multiplication, calculates the initial cosine similarity between all sample pairs in the batch in real time, and generates an initial similarity matrix representing the direct association between samples.
[0055] Laplacian Matrix Construction Module: This module receives the initial similarity matrix, calculates the corresponding degree matrix using graph theory algorithms, and then constructs and symmetrically normalizes it to obtain the Laplacian matrix operator used for diffusion propagation. The Laplacian matrix operator reflects the inherent manifold structure of the feature data.
[0056] Similarity Diffusion Module: As the core innovative unit of the system, this module receives the initial similarity matrix and Laplacian matrix, and executes a multi-step iterative diffusion algorithm to output a final similarity matrix that has undergone information propagation and enhancement. The similarity diffusion module includes a configurable iterative update unit and a step controller. The iterative update unit is responsible for performing single-step similarity propagation and weighted fusion calculations according to the diffusion formula, while the step controller flexibly schedules the execution count of the iterative update unit based on a preset number of diffusion steps, ultimately outputting a final similarity matrix that has undergone multi-step information propagation and enhancement.
[0057] Contrast Optimization Module: This module receives the final similarity matrix and, in conjunction with the real intra-batch matching labels, calculates the contrastive learning loss. Subsequently, it drives an automatic differentiation engine to generate the gradient of the loss with respect to the network parameters, and a co-optimizer guides the encoder parameters in the feature extraction module to perform end-to-end optimization updates.
[0058] In this embodiment, the entire data semantic alignment process is highly efficient and collaborative. First, the feature extraction module transforms heterogeneous data into a unified feature representation; then, the initial similarity calculation module quantifies direct associations; subsequently, the Laplacian matrix construction module abstracts feature relationships into a graph structure; the core similarity diffusion module propagates information on this graph structure, revealing deep associations; finally, the contrast optimization module calculates loss and updates parameters, enabling the entire system to learn high-quality, semantically aligned cross-modal feature representations.
[0059] This invention achieves significant technological advancements in the field of multimodal contrastive learning by introducing a diffusion mechanism. The specific benefits of this invention compared to existing technologies include: 1. Enhanced ability to capture deep semantic associations: This invention innovatively transforms the traditional point-to-point similarity comparison problem into an information propagation problem on the feature map structure. By simulating the iterative diffusion of similarity information in the feature space, the model can effectively capture sample pairs that are not obvious at first but have potential semantic associations, thereby going beyond surface similarity and uncovering deeper semantic matching relationships.
[0060] 2. Effectively alleviates the sparsity problem of the feature space: High-dimensional and sparse image and text feature spaces often make direct similarity calculations insufficiently robust. The diffusion mechanism, through multiple iterations of similarity propagation, makes the connections between similar features stronger, gradually smoothing the similarity distribution in the feature space. This "information sharing" mechanism effectively alleviates the sparsity of the feature space and enhances the model's effective learning of relationships between similar features.
[0061] 3. Improved robustness and generalization ability of alignment: The diffusion process allows the similarity information of each node to aggregate contextual information from multi-hop neighbors, making the model less sensitive to individual noise points or local outliers and improving the robustness of the alignment results. In addition, by learning broader feature associations, the model's generalization ability in the face of unseen new samples is also significantly improved.
[0062] 4. Flexibility and Controllability: The similarity diffusion process proposed in this invention has high flexibility and controllability. This can be achieved by adjusting the diffusion coefficient. The intensity of information propagation can be controlled; by setting the number of diffusion steps K, the depth and breadth of information propagation can be controlled. This allows the model to be finely tuned according to the characteristics of different datasets and task requirements, achieving the optimal balance between model performance, computational cost, and convergence speed.
[0063] 5. Broad Application Prospects: The method and system proposed in this invention can be widely applied to various scenarios requiring efficient and accurate multimodal data alignment. For example, in cross-modal retrieval tasks, it can more accurately retrieve heteromodal content semantically related to a given image or text; in tasks such as image description generation and visual question answering, it can provide more accurate semantic correspondences, thereby comprehensively improving the performance and user experience of downstream tasks.
[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the invention patent is not limited thereto. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the scope of protection of the present invention.
Claims
1. An enhanced multimodal data semantic alignment method, characterized in that, Includes the following steps: S1. Preprocess and extract features from the input image and text data, and then normalize them to obtain the image feature matrix and text feature matrix; S2. Perform a matrix inner product operation on the image feature matrix and the text feature matrix to calculate an initial cross-modal similarity matrix; S3. Treat the initial similarity matrix as an adjacency matrix of a graph structure, where each image feature and text feature is considered a node in the graph structure; The degree matrix is calculated based on the adjacency matrix, and the Laplace matrix is constructed by subtracting the adjacency matrix from the degree matrix. S4. Starting from the initial similarity matrix, the Laplace matrix is used as the propagation operator to perform iterative calculations for a preset number of diffusion steps, resulting in a final similarity matrix that propagates and enhances the deep association relationship. S5. The final similarity matrix is used as the prediction result of the matching relationship between image features and text features in the current batch. The prediction result is compared with the known real image-text matching relationship in the batch, and the contrastive learning loss value is calculated. S6. Based on the calculated contrastive learning loss value, the parameters of the image and text feature extraction networks are iteratively updated through the backpropagation algorithm until the model converges, ultimately achieving deep semantic alignment of cross-modal features.
2. The multimodal data semantic alignment method according to claim 1, characterized in that, Step S2 includes: S21. Cross-modal feature projection: Introduce a linear projection layer. If the dimensions of image features and text features are not equal, the linear projection layer maps the image features and text features to a unified shared embedding space dimension, and then performs normalization processing to obtain normalized image feature matrices and text feature matrices. S22. Perform matrix multiplication on the normalized image feature matrix and the transpose of the text feature matrix to obtain an initial cross-modal similarity matrix without any propagation or adjustment.
3. The multimodal data semantic alignment method according to claim 1, characterized in that, In step S3, the degree matrix is constructed as a diagonal matrix, where each element on the diagonal is equal to the sum of all elements in the corresponding row or column of the initial similarity matrix. Physically, it represents the centrality or total connectivity strength of each image feature node or text feature node in the graph structure.
4. The multimodal data semantic alignment method according to claim 3, characterized in that, Step S3, before constructing the Laplacian matrix, first performs symmetric normalization on the initial similarity matrix, specifically including: S31. Calculate the degree matrix based on the initial similarity matrix. Degree matrix diagonal elements Represents a node The sum of the weights of the edges connected to all other nodes; S32, degree matrix Perform each diagonal element The operation yields a degree-normalized factor matrix; S33. Multiply the degree normalization factor matrix with the initial similarity matrix from the left and right sides respectively, and subtract the product result from the identity matrix to complete the symmetric normalization of the initial similarity matrix, and obtain the symmetric normalized Laplace matrix.
5. The multimodal data semantic alignment method according to claim 1, characterized in that, In step S4, in each iteration, the similarity information of the current node is propagated to its neighboring nodes according to the graph structure and weighted and fused, thereby gradually smoothing the similarity distribution and strengthening the connection between sample pairs with potential semantic associations.
6. The multimodal data semantic alignment method according to claim 5, characterized in that, Step S4 introduces an adjustable diffusion coefficient during the iterative diffusion process to control the fusion ratio of newly propagated neighborhood information and the original information of the node itself during each iteration update.
7. The multimodal data semantic alignment method according to claim 6, characterized in that, The iterative diffusion process in step S4 includes the following steps: S41. Initialize the diffusion state, using the initial similarity matrix as the initial state for the iterative diffusion process; S42. Single-step iterative diffusion: In each iteration, the similarity matrix of the current state is transformed by the Laplacian matrix to obtain the neighborhood aggregation information after one-step propagation. Using a preset diffusion coefficient as a weight, the obtained neighborhood aggregation information is linearly weighted and fused with the initial similarity matrix to obtain the output result of the current iteration step; S43. Repeat the iteration, using the output of the previous iteration as the input of the next iteration, repeat the linear weighted fusion process until the preset number of diffusion steps is reached, and then obtain the final similarity matrix.
8. The multimodal data semantic alignment method according to claim 7, characterized in that, Step S42 calculates in each iteration In the current similarity matrix Based on the symmetric normalized Laplace matrix Update; the iterative diffusion formula is: ； in, Indicates the first The similarity matrix is updated after the next iteration; It is the identity matrix; It is the diffusion coefficient; It is considered a propagation matrix that performs aggregation of local similarity information on the graph structure; This is used to ensure that the original, direct similarity information is not completely lost during the diffusion process.
9. The multimodal data semantic alignment method according to claim 1, characterized in that, Step S5 includes: S51. Obtain positive and negative sample pairs: In each batch, for any image feature Text features that match it within the batch Those that do not match are considered positive samples, while other mismatched samples within the batch are considered positive samples. Text features Then it is considered a negative sample, where ;vice versa; S52. Calculate the contrastive learning loss and convert the final similarity matrix. As the model's prediction result of the image-text matching relationship within the current batch; Image-to-text matching loss The calculation formula is as follows: ； in, Indicates the first The similarity between an image and its corresponding positive sample text; It is a learnable or preset temperature parameter; the denominator is the sum of the exponential similarities calculated for all text samples; For text-to-image matching loss The calculation formula is as follows: ； in, Indicates the first Similarity between a text segment and its corresponding positive sample image; S53. Calculate the total loss Image-to-text loss and text-to-image loss Average value: 。 10. An enhanced multimodal data semantic alignment system, characterized in that, The alignment system is designed to perform the alignment method according to any one of claims 1-9, and includes the following functional modules: The feature extraction module is equipped with a pair of parallel deep neural network encoders, which are used to receive the raw image and text data streams and convert them into a normalized image feature matrix and text feature matrix in a unified high-dimensional semantic feature space. The initial similarity calculation module receives the image feature matrix and text feature matrix from the feature extraction module, and calculates the initial similarity matrix representing the direct relationship between samples in real time through matrix operations. The Laplacian matrix construction module uses the initial similarity matrix as the adjacency matrix of the graph structure and constructs a Laplacian matrix that reflects the inherent manifold structure of the feature data through graph theory algorithms. The similarity diffusion module starts with the initial similarity matrix, uses the Laplacian matrix as the propagation operator, and executes a multi-step iterative diffusion algorithm to output a final similarity matrix that has been propagated and enhanced. The comparison optimization module receives the final similarity matrix and calculates the comparison learning loss by combining it with the real intra-batch matching labels. This then drives an automatic differentiation engine to generate the gradient of the loss with respect to the network parameters, and a co-optimizer guides the encoder parameters in the feature extraction module to perform end-to-end optimization updates.