A multi-modal feature distribution matching method based on hierarchical covariance alignment

Abstract

This invention proposes a "multimodal feature distribution matching method based on hierarchical covariance alignment," belonging to the field of multimodal machine learning. In the efficient parameter fine-tuning stage of the vision-language large model, this method calculates the covariance matrix only for the visual and text features extracted from the first and twelfth Transformer layers, and achieves distribution-level alignment using a Frobenius norm minimization objective function, while simultaneously optimizing it in conjunction with the classification loss. The method includes a layer selection module, a covariance estimation module, a distribution alignment loss module, and an efficient parameter fine-tuning module, and can be plugged into existing frameworks such as LoRA, CoOp, MapLe, and TCP. Experimental results show that this invention significantly improves cross-modal recognition accuracy in both 16-shot and 4-shot scenarios, with an inference latency increase of only about 1 millisecond and a parameter increment of less than two times. It possesses advantages such as robustness to data scarcity, cross-framework universality, and low computational cost, effectively solving the problems of traditional instance-level alignment methods being sensitive to outliers, ignoring inter-feature correlations, and easily getting trapped in local optima.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal learning technology, and in particular to a multimodal feature distribution matching method based on hierarchical covariance alignment, which is applicable to tasks such as image-text alignment, few-shot learning, and fine-tuning of visual-language models. Background Technology

[0002] With the development of large-scale pre-trained models, Vision-Language Models (VLMs) such as CLIP and ALIGN have achieved significant results in tasks such as image classification, image-text retrieval, and visual question answering by realizing the alignment of images and text in a shared embedding space through contrastive learning.

[0003] However, existing methods primarily rely on instance-level feature contrastive learning, i.e., achieving cross-modal alignment by minimizing the distance between the image and text embeddings. This approach has the following drawbacks: Sensitive to outliers: Instance-level alignment is easily affected by outliers or noise, leading to unstable alignment. Ignoring the covariance structure between features: Failure to model higher-order statistical relationships between modalities limits the depth of semantic understanding; Easily trapped in local optima: Sample-level optimization objectives cannot guarantee global distribution consistency, affecting the model's generalization ability.

[0004] Therefore, there is an urgent need for a method that can achieve intermodal alignment at the distribution level to improve the robustness and accuracy of the model in low-sample, cross-modal tasks. Summary of the Invention

[0005] We describe the entire process of the proposed technique and the updated loss function. Due to the model's sensitivity to outlier data, the covariance structure of the data is often ignored, which frequently leads to local optima and poor generalization under limited sample conditions during model training. To address this, we introduce hierarchical covariance loss to provide alignment of distribution levels, improving robustness and accuracy under limited sample conditions. Figure 2 As shown, specifically, the multimodal feature distribution matching method based on hierarchical covariance alignment mainly includes the following steps: (1) Data acquisition and input. Inputting data into a given original training set. ,use -way- -Shot image-text pair format: random sampling There are several categories, and each category is randomly selected. The images and their corresponding text labels form a support set. Select each category again Zhang (usually) ) as a query set All text uses the template "a photoof {class}", and all images use the model's pre-trained format (e.g., 224*224 images from CLIP).

[0006] (2) Multi-layer feature extraction. This method uses a 'dual-branch' encoder backbone structure, including but not limited to CLIP ViT-B / 16, where the image and text encoders each have 12 Transformer Blocks with an embedding dimension of 512. During parameter updates, the backbone weights are frozen by default; a low-rank decomposition module is inserted into the Query, Value attention head in each Transformer Block. ,in , , Training only The matrix; and the attention is calculated as follows: , in To determine the embedding dimension size, Query, Key, and Value are three attention head matrices.

[0007] (3) Covariance matrix calculation. This is calculated by extracting the covariance matrix from each layer, including but not limited to the low-rank decomposition module with both text and image branches. The covariance of the matrix weights is calculated; the covariances obtained from the text branch and the image branch are respectively: , , For the The first encoded text-image pair Categories In other words, These are their covariances.

[0008] (4) Hierarchical selection mechanism. In large-scale language models, the large parameter size, deep stacking, and the black-box nature of deep learning make it difficult to handle the dynamic distribution of features within the representation. To address this, a layer-by-layer indicator is introduced. The covariance of a specified number of layers is selectively aligned, where This is the total number of encoder layers, if This indicates that the layer was selected for covariance alignment; if If so, it means that the layer was not selected for covariance alignment.

[0009] (5) Covariance Alignment Loss. A method is proposed to enhance fine-tuning performance by explicitly aligning the covariance matrices between modes, thereby preserving higher-order interactions and improving cross-modal feature consistency. The covariance alignment loss is: , in Indicate the squared Frobenius norm, To embed feature dimensions, The first The visual covariance matrix and text covariance matrix of the class layer.

[0010] (6) Joint Loss Optimization. The original optimization objective function of the pre-trained model, including but not limited to the optimized loss of CLIP: , in For the number of categories, Number of samples in each category for Category labels, The final predicted probability is calculated as follows: , in This represents the temperature coefficient of Softmax.

[0011] The final optimization objective function is: , in Hyperparameters are used to control the trade-off between task-specific accuracy and cross-modal feature consistency.

[0012] (7) Downstream task output. The low-rank adaptation module weights and indicators obtained after fine-tuning the pre-trained model on the new dataset are saved for easy deployment. During inference, a test image is input, the encoded visual features are extracted, and the cosine similarity is calculated with the set of text prompts. The test image category is obtained through the argmax function.

[0013] Innovation points: The following are the innovative points of the multimodal feature distribution matching method based on hierarchical covariance alignment. 1. This invention proposes a new paradigm of "distribution-level alignment" for the first time, breaking through the limitation of traditional instance-level contrastive learning that only optimizes single-point distances. By explicitly modeling the high-order statistical dependence of visual and text features on the global distribution through hierarchical covariance alignment, it maintains the systematic consistency of the macroscopic embedding structure. This paradigm can be implemented in scenarios with frozen backbones or low-rank fine-tuning, with an increase of less than 1% in the number of training parameters, opening up a new path for lightweight multimodal alignment.

[0014] 2. The hierarchical covariance alignment multimodal feature distribution matching method achieves plug-and-play collaboration with existing optimal parameter-efficient methods (LoRA, CoOp, MaPLe, TCP), achieving significant performance leaps in both 4-shot and 16-shot scenarios across 8 public datasets. In the extreme small sample scenario, the absolute accuracy of EuroSAT is improved by 31.2%, with an average increase of 0.6 to 8.0 percentage points, verifying the powerful generalization and practical value of distribution alignment under data-scarce conditions.

[0015] 3. Through system ablation experiments, this invention, for the first time, quantitatively demonstrates the crucial impact of "layer selection" on alignment performance: the Hybrid{first layer, last layer} strategy reduces computation by 83% while still improving performance by 1.4 percentage points, revealing the intrinsic mechanism of early layers dominating distributed convergence and deeper layers being responsible for semantic refinement; furthermore, hyperparameters... Sensitivity experiments provide a basis for rapid optimization of tasks with different complexities, laying the theoretical and engineering foundation for subsequent multimodal research.

[0016] Results analysis: Figure 3 The 16-shot results show that after applying the multimodal feature distribution matching method with hierarchical covariance alignment, the average accuracy of LoRA, CoOp, MapLe, and TCP improved by 0.6, 0.1, 0.4, and 0.1 percentage points, respectively, with none showing a decrease. The fine-grained datasets FGVC-Aircraft and Oxford-Flowers showed the highest increases, validating that distribution alignment is more discriminative of texture details. Figure 4 In 4-shot scenarios, the performance gains are further amplified, with average performance improvements of 4.5, 7.9, and 4.1 percentage points for CoOp, MapLe, and TCP, respectively, demonstrating the strong regularization effect of hierarchical covariance alignment-based multimodal feature distribution matching methods when data is scarce. In fine-grained tasks such as EuroSAT, the single-point improvement reaches up to 18.8 percentage points, indicating that covariance constraints effectively compensate for insufficient samples. The method also continues to lead in scenarios with large class distribution differences, such as ImageNet-1K and SUN397, demonstrating the strong correction capability of hierarchical covariance alignment-based multimodal feature distribution matching methods for cross-modal distribution shape differences. Overall, hierarchical covariance alignment-based multimodal feature distribution matching methods achieve stable, significant, and widespread performance improvements under conditions of few samples in a plug-and-play, minimal-parameter, and cross-framework-wide manner. The standard deviation of all methods decreases synchronously after incorporating hierarchical covariance alignment-based multimodal feature distribution matching methods, further proving their robustness. In summary, Figure 3 , Figure 4 We jointly established the multimodal feature distribution matching method of hierarchical covariance alignment as an irreplaceable distribution-level alignment module in data-scarce scenarios and its engineering promotion value.

[0017] Ablation experiment: To evaluate the effectiveness of various methods, we conducted ablation analysis of 1) layer selection strategies, 2) ablation analysis of statistical types, and 3) ablation analysis of the complexity of multimodal alignment matching methods. We evaluated and compared the various methods, and... Figure 5 , Figure 6 and Figure 7 The experimental results are presented and analyzed.

[0018] Figure 5 The results demonstrate that the interlayer selective ablation experiments further validate the technical effectiveness of the interlayer covariance alignment strategy proposed in this invention: among nine layer combination schemes, the traditional "full-layer alignment" method only achieved classification accuracies of 71.45% and 86.28% on the DTD and Food-101 datasets, respectively; while using "single alignment of the last layer" or "alignment of the top four layers" can improve the accuracy to 72.87% and 86.48%, respectively, which is significantly better than the full-layer scheme. Particularly noteworthy is the "hybrid alignment" scheme defined in this invention—which applies covariance matrix constraints only to the first and twelfth layers simultaneously—which can further improve the accuracy to 73.70% and 86.39%, achieving the best performance gain with minimal computational overhead. These results fully demonstrate that the early layer and the last layer correspond to the feature distribution convergence stage and the semantic divergence stage, respectively. Only by aligning the two together can the commonalities of the lower layers be preserved and the semantics of the higher layers be calibrated. Synchronous alignment of all layers is prone to introducing redundant statistical signals, which leads to performance degradation. Interlayer selective covariance alignment is not "the more layers the better", but "the more accurate the layer position, the better", thus establishing the inventiveness and irreplaceability of this invention in terms of structural configuration.

[0019] like Figure 6 As shown, the statistical type ablation experiment further verifies the significant technical effect of the "covariance matrix alignment" used in this invention compared to conventional low-order statistical alignment: under the same 16-shot settings, only mean and variance alignment is performed. The accuracy achieved on the DTD and Food-101 datasets was 72.64% and 85.97% respectively, an improvement of less than 0.1 percentage points compared to the unaligned baseline (72.52% / 85.91%), which is negligible. However, after adopting the hierarchical covariance alignment multimodal feature distribution matching method proposed in this invention, the accuracy jumped to 73.70% and 86.42%, an improvement of 1.2 and 0.5 percentage points respectively, and significantly outperformed any low-order statistical method. This result fully demonstrates that first-order and second-order moments cannot model high-order interdependencies between cross-modal features, resulting in limited alignment effects; the covariance matrix fully preserves the linear correlation between feature dimensions, effectively reducing the structural differences between visual and textual distributions. Therefore, this invention embeds covariance alignment into an inter-layer selective strategy, combining the dual advantages of "high-order statistics + precise layer," effectively overcoming the information loss defects of traditional mean-variance alignment, highlighting its creativity and technical necessity.

[0020] In addition, such as Figure 7 As shown, the computational cost ablation experiment further verifies the controllability and technical rationality of the covariance alignment module introduced in this invention in terms of parameter increment and inference time: Under the condition of batch size of 4, the CoOp baseline only requires 17 ms, has 2.05 K trainable parameters, and an accuracy of 58.6%; after introducing the hierarchical covariance alignment multimodal feature distribution matching method of this invention, the time only increases to 18 ms, the number of parameters increases to 4.71 M, but the accuracy jumps to 71.4%, an improvement of 12.8 percentage points. Similarly, the Maple baseline takes 23 ms, has 3.55 M parameters, and an accuracy of 54.7%; after superimposing the hierarchical covariance alignment multimodal feature distribution matching method, the time is 24 ms, the number of parameters is 8.26 M, and the accuracy rises to 71.1%, an improvement of up to 16.4 percentage points. The above results show that the module of this invention adds only about 1 ms latency on the inference side. The parameter increase mainly comes from the lightweight linear layer required for covariance matrix estimation, and the computational overhead is controllable. Compared with the original method, this invention achieves an absolute accuracy improvement of over 16.4% with less than twice the parameter increment, and the performance-cost ratio is significantly better than conventional dense fine-tuning schemes. Even in throughput tests with a batch size scaled up to 100, this invention still maintains a millisecond-level growth rate similar to the baseline, verifying its good scalable inference capability. In summary, Table 5 confirms that this invention achieves a balance in "high accuracy, low latency, and scalability," thus establishing its practicality and competitive advantage in actual deployment. Attached Figure Description

[0021] (1) Figure 1 This paper compares the CLIP standard contrastive learning paradigm (upper branch) with the hierarchical covariance alignment multimodal feature distribution matching method proposed in this invention (lower branch).

[0022] (2) Figure 2 This is a flowchart of a multimodal feature distribution matching method based on hierarchical covariance alignment.

[0023] (3) Figure 3 This is a comparison of classification accuracy results for various methods after embedding hierarchical covariance aligned multimodal feature distribution matching methods in a 16-shot scenario.

[0024] (4) Figure 4 This data represents the cross-method performance improvement brought about by the multimodal feature distribution matching method with hierarchical covariance alignment in 4-shot scenarios.

[0025] (5) Figure 5 This is a comparison of the impact of different hierarchical covariance alignment multimodal feature distribution matching methods on different datasets under different hierarchical selection strategies.

[0026] (6) Figure 6 This is a comparison of the impact of different statistical measures on performance results.

[0027] (7) Figure 7 This is a comparison of inference latency, parameter count, and accuracy after introducing a multimodal feature distribution matching method with hierarchical covariance alignment.

Claims

1. A multimodal feature distribution matching method based on hierarchical covariance alignment, characterized in that, Includes the following steps: Acquire image-text pairing data and input it into a pre-trained multimodal model; Multi-layer feature representations are extracted using image encoders and text encoders, respectively; For each layer, calculate the covariance matrix between image features and text features; A layer selection mechanism is introduced to determine whether to perform covariance alignment on the current layer; Construct a covariance alignment loss function to minimize the difference between the image and text covariance matrices; The covariance alignment loss and classification loss are jointly optimized to update the model parameters; Output the aligned multimodal model for executing downstream tasks.

2. As described in claim 1, wherein, The multimodal model is the CLIP model, which includes an image encoder and a text encoder, both based on the Transformer architecture.

3. As described in claim 1, the formula for calculating covariance is: For image feature matrix The covariance matrix is: . For the text feature matrix The covariance matrix is: , in, For the sample size, For feature dimensions.

4. As described in claim 1, wherein, The hierarchical selection mechanism introduces hierarchical indicator variables. Determine whether the first Layers are aligned for covariance, where This represents the total number of floors.

5. As described in claim 1, wherein, The covariance alignment loss function is: , in, Denotes the Frobenius norm. and They represent the first Class in Image and text covariance matrix of the layer.

6. As described in claim 1, wherein, The joint optimization objective function is: , in, Cross-entropy classification loss, This refers to the hyperparameters used to control alignment strength.

7. As described in claim 1, wherein, The downstream tasks include image classification, image and text retrieval, and visual question answering.

Images