Large model fine tuning method based on wavelet transform

Abstract

According to the large model fine tuning method based on wavelet transformation, the performance of the visual language model in a small sample learning task is remarkably improved by combining the wavelet transformation and a low-rank matrix adaptation method. The method is suitable for medical image analysis, satellite texture recognition and other scenes needing fine-grained feature adaptation. The core of the method is that model parameters are updated and decomposed into low-frequency components and high-frequency components by using the localization characteristic of wavelet transform, and multi-scale information is fused through inverse wavelet transform, so that efficient adaptation to complex visual tasks is realized. Experimental results show that compared with other methods, the method has the advantage that the accuracy is obviously improved. The method is characterized in that wavelet transform is introduced into a low-rank matrix adaptation method frame for the first time, global-local feature optimization is achieved through multi-resolution decomposition, and the adaptive capacity to complex signals is enhanced. Due to the innovations, the large model fine tuning method based on wavelet transformation has remarkable advantages and application potential in the field of large model fine tuning.

Description

Technical Field

[0001] Large model fine-tuning methods are very important in the practical application of large models. This method has a wide range of uses in applications such as large model customization and local rapid deployment. Because large models have excessive memory requirements during inference and adaptation to specific tasks, these issues affect the local customization of large models. Technical Background

[0002] Large model fine-tuning is an important approach in the field of large computer models, addressing the challenges of deploying and customizing large models. Due to their large number of parameters, large models require excessive computational and memory resources to retrain on new tasks, hindering model customization. For example, in medicine, large models need to be retrained for specific diseases. In this context, large model fine-tuning is used to accelerate model deployment and ensure model customization for specific tasks.

[0003] Currently, large model fine-tuning methods primarily rely on adaptation-based methods, cue-word-based methods, and low-rank matrix adaptation-based methods. Adaptation-based methods primarily fine-tune the model by adding additional network layers to the pre-trained model, but the addition of additional parameters increases inference speed. Cue-word-based methods primarily add learnable tags to text input, but this increases the memory consumption of gradients during backpropagation during training. Low-rank matrix adaptation-based methods advocate exploring the inherent rank problem of large models for specific data distributions and improving fine-tuning speed by learning low-rank matrices.

[0004] To further enhance the ability of large-model fine-tuning methods to capture information at different resolutions, we introduced the discrete wavelet transform into the low-rank matrix adaptation method and proposed a large-model fine-tuning method based on the wavelet transform. This method uses the wavelet transform to capture information at both high and low resolutions, preserving the main information while also focusing on the details of the data distribution. This effectively improves the performance of the fine-tuned model. Summary of the Invention

[0005] We describe the whole process and loss function of the proposed method. Inspired by the Fourier transform-based large model fine-tuning method (FourierFT), we introduce discrete wavelet transform, such as Figure 1 Specifically, the large model fine-tuning method based on wavelet transform mainly includes the following steps:

[0006] (1) Through the two-dimensional wavelet transform technology, the pre-trained model attention head matrix is decomposed into four different resolution matrices, which represent the low-frequency main information and the horizontal, vertical and diagonal high-frequency detail information respectively.

[0007] (2) The four resolution matrices decomposed are regarded as low-rank and obtained by the dot product of two low-rank matrices BA, where the B matrix is initialized to all 0s and the A matrix is initialized to a standard Gaussian distribution.

[0008] (3) The four learned matrices are restored to the supplementary information about the new data in fine-tuning through two-dimensional discrete wavelet inverse transform and added to the weights to finally obtain the fine-tuned weights.

[0009] Large model fine-tuning methods based on wavelet transforms all aim to address the enormous computational and memory requirements required to retrain large models on new datasets. Their goal is to rapidly learn the feature distribution of new datasets and adapt quickly to new features during model fine-tuning. Regarding low-rank matrix adaptation methods, based on the intrinsic rank theory of large models, when processing specific tasks, the rank utilized by the weights is often very small. Therefore, the dot product of a low-rank matrix is used to approximate it. This process can be mathematically modeled as follows:

[0010] W=W0+ΔW=W0+αBA

[0011] in, Represents the weight of the pre-trained model; Here, we take the low-rank matrix B = 0, A∈N(0,1) and r<<min(M,N) trained in fine-tuning. α is a scaling factor. Different optimization strategies are used for different downstream tasks. Here, we take the unsupervised contrastive language-image pre-training (CLIP) model for image vision as an example. The optimization process can be expressed as follows:

[0012]

[0013] in, represents the predicted label of the image, which is obtained by the dot product l of the image and text encoding, Y represents the true label of the image, τ is the temperature coefficient, and L(·) represents the cross entropy loss function used to update the network.

[0014] Step 1. Application of large model fine-tuning method based on wavelet transform

[0015] As a large model fine-tuning method, it adjusts weights by learning a low-rank matrix, reducing computational and storage overhead; this is then applied to the attention heads qkv in the Transformer. Because the attention heads in the Transformer are often responsible for capturing the dependencies between different positions in the input sequence, they directly affect the model's ability to understand context. The attention heads qkv contained in the Transformer are highly sensitive to parameter changes, allowing for more efficient adjustment of the model's feature extraction methods for different tasks. In particular, the text language comparison model has different encoders for processing image data and text data, respectively. Its inference process can be expressed as follows:

[0016] logits=f T t

[0017] Where f∈R Samples×D is the feature obtained after the image passes through the image encoder, t∈R Samples×D is the feature obtained after the text passes through the text encoder, logits is the final prediction of the image, and in the small sample N-way-K-shot setting, Samples = NK, where N represents the number of categories and K represents the number of samples in each category.

[0018] For different encoders, the Transformer architecture is used, and the image encoder adopts the ViTB-16 architecture. The large model fine-tuning method based on wavelet transform is applied to all attention heads of Transformer, such as Figure 2 Compared with full fine-tuning, the large model fine-tuning method based on wavelet transform has a significantly reduced parameter size and can maximize computational efficiency and storage efficiency while maintaining model performance, especially for large models.

[0019] Step 2. Multi-resolution matrix initialization

[0020] In the large model fine-tuning method based on Fourier transform, the Fourier transform technology is used to convert the weights to the frequency domain space, and the main frequency is learned to adapt to the new task. However, it is observed that the Fourier transform is naturally unable to capture small-scale detail changes. Considering that the wavelet transform can effectively capture the mutual information of time and space while converting information to the frequency domain, referring to the role of 2D wavelet transform technology on images, this process can be mathematically expressed as:

[0021]

[0022] in, For image data to be transformed by wavelet, is the low-frequency and high-frequency information of the image obtained after decomposition, where Conv(·) is the convolution operation, For filters of different resolutions of two-dimensional wavelet transform, taking Haar wavelet as an example, they can be numerically expressed in the following form:

[0023]

[0024] Among them, f LL The low-pass filter mainly extracts the low-frequency information of the image, f LH ,f HL ,f HH It is a high-pass filter to capture high-frequency information in the horizontal, vertical and diagonal directions of the image.

[0025] Inspired by this, we introduce wavelet transform into it, so that it can not only capture local features but also retain the original weight main information. Apply it to the weight W0∈R N×M Among them, this process is expressed as:

[0026]

[0027] in, is the low-frequency part of the original weight W0, For the high-frequency part of the original weight, different tasks are assigned by different weights, so that the model can not only focus on low-frequency information during fine-tuning, but also not lose the details of the high-frequency part. The same initialization is performed for the four resolution matrices, and the process is as follows:

[0028]

[0029] in, i={LL,LH,HL,HH}. Initialize B respectively i =0, A i ∈N(0,1) to ensure that the output of the model is consistent with the pre-training results when fine-tuning training begins.

[0030] Compared with the large model fine-tuning method based on Fourier transform, in low-resolution input, high-frequency information is easily misjudged as noise and filtered out, resulting in loss of details; it is difficult to perform fine control for specific tasks. The large model fine-tuning method based on wavelet transform can retain more high-frequency textures in low-resolution tasks, and can allow low-frequency and high-frequency information to be integrated with each other, effectively retaining the model's ability to capture neighborhood-specific details.

[0031] Step 3. Forward propagation process

[0032] (1) 2D wavelet inverse transform

[0033] The frequency domain information is converted into the feature space where the weights are located through 2D inverse wavelet transform. This process can be expressed as:

[0034]

[0035] Among them, TranposedConv(·) is a deconvolution calculation, which is used to restore the frequency domain information to the space where the weights are located.

[0036] (2) Weight merging:

[0037] For the obtained adaptation weight ΔW, it is directly added to the original weight W0 during forward propagation to obtain the updated weight matrix, thereby performing the forward propagation process of the entire model.

[0038] Step 4. Loss function optimization

[0039] We directly use the optimization function of the original CLIP for optimization, and the loss function L is defined as:

[0040]

[0041] where p i,k is the predicted probability, y ik is the true label of the sample. Through a simple and effective cross-entropy loss function, the distance between the predicted value and the true label can be effectively shortened, and the model can be trained efficiently.

[0042] Innovation

[0043] The following are the innovations of the large model fine-tuning method based on wavelet transform

[0044] 1. The proposed wavelet-based large model fine-tuning method optimizes weight updates in the frequency domain, establishing a new paradigm for the dynamic analysis of information propagation and hierarchical feature refinement during model fine-tuning. This spectral perspective reveals a previously unexplored correlation between parameter adjustment and multi-scale feature representation.

[0045] 2. Leveraging the inherent multi-resolution properties of wavelet transforms, this method dynamically adjusts the granularity of weight adjustments in neural network layers. This task-aware adaptive mechanism precisely controls the update amplitude from coarse structural modifications to fine-grained parameter calibration, significantly improving cross-domain generalization while maintaining computational efficiency.

[0046] 3. The successful implementation of a wavelet-based large-model fine-tuning method builds a bridge between classical signal processing theory and contemporary deep learning architectures. This approach provides a new perspective for systematically integrating proven signal processing tools into neural network optimization. This cross-domain collaboration facilitates the encoding of structured domain knowledge in AI systems and provides a signal processing-based approach to enhance model interpretability and generalization.

[0047] Result Analysis

[0048] Quantitative results

[0049] To verify the quantitative performance of our wavelet-based large model fine-tuning method, we conducted experiments on eleven public datasets and compared them with previous state-of-the-art methods. The results are shown in Figure 4 Specifically, we compared 10 previous methods under the same experimental setup. Our large model fine-tuning method based on wavelet transform surpassed all 10 previous methods in average accuracy across 11 datasets, achieving an average improvement of 2.3%. This result demonstrates that our method remains highly competitive while maintaining a relatively low number of parameters and computational complexity.

[0050] Quality results

[0051] In order to verify the quality performance of the large model fine-tuning method based on wavelet transform, we provide a comparison of the large model fine-tuning method based on wavelet transform applied to each layer of CLIP and the large model fine-tuning method based on Fourier transform under the same parameter amount, see Figure 3 and Figure 5 .exist Figure 3 In the detailed comparison, the large model fine-tuning method based on wavelet transform is applied to CLIP in each layer of attention head. For pure text tasks, the performance improves steadily with the increase of parameters, but the marginal benefit decreases significantly. It is recommended to use a smaller parameter scale to save resources. In contrast, for visual tasks, the parameter requirements are higher than those for text tasks, resulting in lower performance. In general, when fine-tuning multimodal models such as CLIP, the impact of different modalities varies. In the comparison under the same parameters in the figure, although the number of parameters increases linearly, the performance of CLIP-LoRA only improves slightly, and even decreases in some cases, which may be due to overfitting or reduced optimization efficiency. In contrast, Figure 5 Under the same parameter scale, the large model fine-tuning method based on wavelet transform shows more stable performance. This result shows that the large model fine-tuning method based on wavelet transform can effectively capture information at different resolutions and effectively compensate for the information loss problem caused by too small a dimension.

[0052] Ablation experiments

[0053] In order to evaluate the effectiveness of each method, including 1) training matrices with different resolutions and 2) wavelet mother functions, we evaluated and compared the use of each method and Figure 6 and Figure 7 The experimental results are shown in .

[0054] Figure 6The results in

[15] show that wavelet transform matrices of varying resolutions have a positive and effective impact on fine-tuning results. Experimental results demonstrate that both high- and low-frequency components can effectively compensate for information loss. When high-frequency information is missing, model performance significantly degrades. This also suggests that low-frequency components serve as the base representation, while high-frequency components enhance detail texture and boundary information to a certain extent, thereby strengthening multi-scale representation capabilities.

[0055] In addition, if Figure 7 As shown in the figure, the simple and effective Haar wavelet function plays a key role in the wavelet-based large model fine-tuning method, significantly outperforming other wavelet functions. The results show that through scaling and translation operations, the Haar wavelet function, due to its simplicity and orthogonality, can effectively decompose features into high- and low-frequency information, ensuring their mutual independence, thus outperforming other wavelet functions. Therefore, using the Haar wavelet function as a mother function can significantly improve the performance of model fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] (1) Figure 1 Compared with FourierFT, our method introduces wavelet transform to capture information at high and low resolutions. This method can effectively compensate for the information loss caused by low dimensions.

[0057] (2) Figure 2 It is a flow chart of the large model fine-tuning method based on wavelet transform.

[0058] (3) Figure 3 It is the impact of the large model fine-tuning method based on wavelet transform applied to different attention heads of different modules of CLIP on the final result.

[0059] (4) Figure 4 This is a comparison of the final experimental results of the large model fine-tuning method based on wavelet transform and the previous 10 advanced methods on 11 public data. The results show that the average experimental results of the large model fine-tuning method based on wavelet transform are improved by 2.3% compared with the previous methods.

[0060] (5) Figure 5 This is a comparison of the experimental results of the large model fine-tuning method based on wavelet transform and different methods under the same parameters.

[0061] (6) Figure 6 This is the impact of the large model fine-tuning method based on wavelet transform on the experimental results of training matrices with different resolutions.

[0062] (7) Figure 7 It is the influence of different wavelet functions on the experimental results.

Claims

1. A large model fine-tuning method based on wavelet transform, characterized by: The following steps are included: Step 1: Apply wavelet transform technology to the Transformer structure of the pre-trained model, decompose its original weights into different resolution matrices, and update the resolution matrices to enable the model to learn the weights of the new task. The high-frequency matrix mainly focuses on the detailed information of the data in the new task, while the low-frequency matrix mainly focuses on the main information of the data; Step 2: Initialize the resolution matrix through Low-Rank Adaptation (LoRA), and decompose it into the dot product of two low-rank matrices BA. Their ranks are much smaller than the rank of the initial weight matrix. Step three: Use the inverse wavelet transform technique to convert the resolution matrix from the frequency domain into the model's weight information to obtain the weight for fine-tuning the model on the new task, thereby achieving fine-tuning of the model on the new task.

2. The large model fine-tuning method based on wavelet transform according to claim 1, characterized in that: The wavelet transform-based large model fine-tuning method is applied to the different attention heads of the model's Transformer structure to enable the model to learn the weights of new tasks, allowing the model to quickly adapt to new datasets.

3. The resolution matrix according to claim 1, wherein: The resolution matrix adaptation is used in different attention heads of the Transformer module in the pre-trained model to adapt the multi-resolution weights of different attention heads.

4. The low-rank matrix adaptation method according to claim 1, wherein: The low-rank matrix is decomposed into the dot product BA of two low-rank matrices by the weight matrix. The model can be fine-tuned by simply updating the two low-rank matrices B and A, thereby accelerating the adaptation of the model to new tasks.

5. The low-rank matrix adaptation method according to claim 1, wherein: The low-rank matrix adaptation is to initialize the low-rank matrix BA to all zeros and to initialize it to obey the normal Gaussian distribution, so as to make the model consistent with the output results of the pre-training model in the initial stage of fine-tuning.

6. The wavelet transform technique according to claim 1, wherein: The wavelet transform technology uses 2D discrete wavelet transform, which converts weights into low-frequency data and high-frequency data in the horizontal, vertical and diagonal directions, where the low-frequency data is used to store main information and the high-frequency data is used to store detail information, so that the model can focus on features at different resolutions and enhance its generalization ability.

7. The wavelet transform technique according to claim 1, wherein: The wavelet transform technology uses Haar wavelet as the mother wavelet for transformation. Due to its simplicity and high efficiency, it can accelerate the calculation of the model and improve the fine-tuning speed.

8. The large model fine-tuning method based on wavelet transform according to claim 1, characterized in that: The wavelet transform-based large model fine-tuning method is trained and tested on multiple public datasets, including but not limited to ImageNet-1K, DTD, UCF101, and Oxford_flowers, to verify its practicality and effectiveness under different data distributions.

9. The large model fine-tuning method based on wavelet transform according to claim 1, characterized in that: This wavelet-based large-model fine-tuning method is applied to various applications, including but not limited to visual language models such as CLIP, to adjust the adaptability of different models to specific data distributions, enabling faster deployment in different workloads.

Images