Large visual model parameter efficient fine tuning method and system and image recognition method

Abstract

The invention provides an efficient fine tuning method and system for parameters of a large visual model, and the method comprises the steps: respectively inserting a parameter fine tuning module in each multi-head attention module and a feedforward neural network module of a pre-trained large visual model, and enabling a part of parameters of the parameter fine tuning module to be globally shared among a plurality of modules of the pre-trained large visual model, the other part is independently adjusted through local characteristics so as to adapt to task requirements of the inserted corresponding modules; inputting the image data into the pre-trained visual large model inserted into the parameter fine tuning module to generate a feature map; based on the feature map, freezing backbone network parameters, and training parameters of a parameter fine tuning module; according to the method, the parameter quantity required by the model is further greatly reduced while relatively high performance of the pre-training large model is ensured, so that more excellent balance between the performance and storage is obtained, and the method can be applied to the fields of migration of the pre-training model to downstream tasks such as image recognition and the like.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, especially transfer learning techniques in deep learning, specifically to an efficient parameter fine-tuning method, system, and image recognition method for large visual models. Background Technology

[0002] With the rapid development of computing power and deep learning, the field of artificial intelligence has witnessed groundbreaking progress. In the field of image processing, the technological evolution exhibits a clear iterative trajectory: Early image processing was limited by image acquisition quality, resulting in lower requirements for feature extraction; therefore, manual feature extraction became the mainstream technique. With improvements in image quality and increased demands for processing precision, the introduction of statistical pattern recognition methods (such as SVM) and signal processing techniques (such as wavelet transform) propelled the advancement of image processing technology. However, manually designed features always have inherent drawbacks: on the one hand, they rely on prior human knowledge, inevitably introducing subjective biases; on the other hand, they require stringent image preprocessing and consistently fall short of human visual perception in terms of performance. The emergence of end-to-end neural networks has broken this limitation. These models, through joint training of feature extraction and classifiers, automatically generate filter banks adapted to the task objectives in a data-driven manner. This eliminates the tediousness of manually designing features and requires no complex preprocessing, demonstrating powerful image processing capabilities. To date, the field of deep learning has developed into a mainstream solution based on large-scale pre-trained models (such as Transformer and BERT). These models have demonstrated outstanding performance in many types of tasks, but at the same time, they are accompanied by huge computational and storage overheads. This problem not only poses a challenge in the training phase, but also becomes a significant performance bottleneck in the model fine-tuning and inference phases (especially on resource-constrained devices). Traditional training methods (especially for large-scale deep learning models) often rely on massive amounts of labeled data and ample computing resources. This "from scratch" training approach not only consumes enormous amounts of computing power but also requires a long training cycle, making it difficult to adapt to scenarios with limited computing resources. For example, pre-trained models such as GPT typically contain hundreds of millions of parameters, requiring significant computing and storage resources even during fine-tuning. Furthermore, these models generally suffer from parameter redundancy: many parameters do not significantly affect the final performance during fine-tuning for a specific task. Therefore, how to reduce the computational and storage costs of large-scale pre-trained models while preserving their core performance in resource-constrained devices (such as mobile devices and edge computing devices) has become a key issue of common concern to both academia and industry. Against this backdrop, the Parameter-Efficient Fine-Tuning (PEFT) method has emerged. Its core idea is to fine-tune only a small number of parameters based on the pre-trained model, thereby reducing computational and storage overhead while maximizing the preservation of the model's original performance—the core objective is to maintain or even improve the model's performance in downstream tasks while reducing the size of trainable parameters. However, traditional PEFT methods have significant limitations: the introduction of parameter sharing or low-rank mappings can introduce approximation errors, leading to homogenization of feature transformations, a phenomenon particularly prominent in multi-layer networks. The core drawback of feature homogenization is that although the parameter size and computational cost are significantly reduced, the model's feature diversity and representational power are also compromised—feature transformations in different layers become too similar during fine-tuning, making it impossible to effectively extract diverse information from different layers, ultimately limiting the model's performance in complex tasks.

[0003] A search revealed that Chinese patent application number 202411602390.X discloses an efficient fine-tuning method for visual base model parameters for dense prediction tasks. This method utilizes training images from dense prediction tasks to train the parameters of the HOM (Hypertext Markup Object) module and the dynamic parameter generation module, effectively reducing the resources required to apply the visual base model to dense prediction tasks, decreasing GPU memory usage and training time. However, this method does not consider the issue of homogenization. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the purpose of this application is to provide an efficient method, system and image recognition method for fine-tuning parameters of large visual models.

[0005] According to a first aspect of this application, an efficient method for fine-tuning the parameters of a large visual model is provided, comprising: Acquire a pre-trained large visual model and image training data for fine-tuning; In each multi-head attention module and feedforward neural network module of the pre-trained large visual model, a parameter fine-tuning module is inserted. Some parameters of the parameter fine-tuning module are shared globally among multiple modules at different levels, while other parameters are set independently for the inserted module to adapt to the task requirements of local features. The image data is input into the pre-trained large visual model inserted into the parameter fine-tuning module to generate feature maps and corresponding predicted values. Based on the predicted values ​​and the true class labels of the image training data, calculate the cross-entropy loss; The gradient is calculated based on the cross-entropy loss, the parameters of the backbone network are fixed, and the parameters of the parameter fine-tuning module are updated based on the gradient descent method. Repeat the above process of image input, feature generation, cross-entropy loss calculation, gradient solving, and parameter update until the cross-entropy loss converges.

[0006] Optionally, the step of inserting parameter fine-tuning modules into each multi-head attention module and feedforward neural network module of the pre-trained large visual model includes: For each of the multi-head attention modules, a learnable parameter matrix for generating the query Q matrix, a learnable parameter matrix for generating the key K matrix, a learnable parameter matrix for generating the value V matrix, and a linear mapping matrix are introduced into a parallel parameter fine-tuning module. Parallel parameter fine-tuning modules are introduced for each feedforward neural network module; Each parameter fine-tuning submodule receives the same input as its parallel structure and merges its output with the output of the parallel network structure element by element.

[0007] Optionally, the parameter fine-tuning module includes an encoder, a hybrid expert submodule, a locally demultiplexed convolutional layer, and a decoder connected in sequence; The encoder maps the input feature map to a low-dimensional space to obtain a dimensionality-reduced feature map; The hybrid expert submodule extracts multi-scale features from the dimensionality-reduced feature map, uses the attention weights corresponding to the feature correlations, and uses the attention weights to perform weighted fusion of the multi-scale features to output hybrid scale features. The local demultiplexed convolutional layer adjusts the mixed-scale features, removes the representation consistency constraint caused by parameter sharing, and generates features that are adapted to the task requirements of each module at its respective level. The decoder maps the features adapted to the task requirements of each module's level back to a high-dimensional space, making its output dimension consistent with the original output feature dimension of the corresponding backbone network module.

[0008] Optionally, the hybrid expert submodule includes an expert unit, a gating unit, and a fusion unit connected in sequence; Multiple expert units configured in parallel are set up with different spatial scales, which can extract multi-scale features by interpolating and convolutionally transforming the reduced feature map. The gating unit calculates expert selection weights based on the input dimensionality-reduced feature map, and selects one or more expert units as target expert units to participate in feature extraction according to the weights; The fusion unit only performs weighted fusion on the multi-scale features output by the target expert unit to generate the final feature representation, namely the hybrid scale feature.

[0009] Optionally, the gating unit includes: A gated linear subnetwork generates the basic weights of each expert unit based on the input dimensionality-reduced feature map; The noise-aware subnetwork generates perturbation weights coupled with standard Gaussian noise based on the input dimensionality-reduced feature map; The output submodule merges the base weights and the perturbation weights to obtain the original expert weights. It selects the top K terms with the largest values ​​from the original expert weights and sets the remaining terms to zero. It then normalizes the non-zero weights and outputs the final expert selection weights.

[0010] Optionally, the encoder, hybrid expert submodule, and decoder in all parameter fine-tuning modules share the same learnable parameters; the locally demultiplexed convolutional layer is configured independently for each module and has non-shared weights.

[0011] A second aspect of this application provides an efficient parameter fine-tuning system for large visual models, comprising: Preliminary preparation module: Acquire pre-trained large visual model and image training data for fine-tuning; Insertion Module: In each multi-head attention module and feedforward neural network module of the pre-trained large visual model, a parameter fine-tuning module is inserted respectively. Some parameters of the parameter fine-tuning module are shared globally among multiple modules at different levels, while other parameters are set independently for the insertion module to adapt to the task requirements of local features. Generation module: Inputs the image data into the pre-trained large visual model inserted into the parameter fine-tuning module to generate feature maps and corresponding predicted values; Loss module: Calculates cross-entropy loss based on the predicted value and the true class labels of the image training data; Fine-tuning module: Calculates the gradient based on the cross-entropy loss, fixes the parameters of the backbone network, and updates the parameters of the parameter fine-tuning module based on the gradient descent method; Loop module: Repeat the above process of image input, feature generation, cross-entropy loss calculation, gradient solving and parameter update until the cross-entropy loss converges.

[0012] A third aspect of this application provides an image recognition method, comprising: Acquire the target image data to be identified; Based on the efficient fine-tuning method for the parameters of the large visual model, the pre-trained large visual model is fine-tuned to obtain the fine-tuned large visual model. The target image data is input into the fine-tuned visual model, and the category prediction result is generated through forward propagation to complete the recognition of the target image.

[0013] According to a fourth aspect of this application, a terminal is provided, including a memory and a processor, wherein the memory stores a program executed by the processor, and the processor, when executing the program, can perform any of the following methods: the above-described efficient fine-tuning method for parameters of a large visual model, or the above-described image recognition method, or the above-described image recognition system.

[0014] According to a fifth aspect of this application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can perform the above-described method for efficient fine-tuning of parameters of a large visual model, or the above-described image recognition method, or the above-described image recognition system.

[0015] This application proposes an efficient fine-tuning method for large visual models. By setting globally shared learnable parameters in the parameter fine-tuning module, it effectively reduces the number of redundant parameters, significantly lowers the model's storage overhead and memory usage, and improves parameter utilization efficiency. This is particularly beneficial for fine-tuning large visual models on edge devices with limited storage resources. By configuring locally demultiplexed convolutional layers to achieve personalized feature modulation at each network layer, this application breaks the representation consistency constraint caused by global parameter sharing. While maintaining parameter efficiency, it enhances the modeling ability of each layer for specific semantic levels, avoiding feature homogenization caused by excessive sharing, and further improving fine-tuning performance. Through the combined effect of the aforementioned globally shared and locally demultiplexed convolutional layers, a balance between performance and storage is achieved, making it applicable to the transfer of pre-trained models to downstream tasks such as image recognition.

[0016] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating an efficient parameter fine-tuning method according to an exemplary embodiment; Figure 2 A flowchart illustrating the insertion of a pre-training module into a parameter fine-tuning module according to an exemplary embodiment; Figure 3 This is a flowchart illustrating a parameter fine-tuning module according to an exemplary embodiment; Figure 4 This is a flowchart illustrating a hybrid expert submodule according to an exemplary embodiment; Figure 5This is a structural diagram illustrating a parameter fine-tuning system according to an exemplary embodiment; Figure 6 This is a comparison chart of the results of different training methods on the VTAB-1K benchmark, based on an exemplary embodiment. Figure 7 This is a comparison chart showing the results of a small-sample image classification embodiment illustrated according to an exemplary embodiment on the FGVC small-sample benchmark test. Detailed Implementation

[0018] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0019] Definitions: Locally demultiplexed convolutional layers are lightweight two-dimensional convolutional modules consisting of a single 3×3 convolutional layer. The number of input and output channels is equal to the low-rank embedding space dimension, and the convolutional kernel parameters are not shared between different network layers.

[0020] This convolutional layer personalizes the hybrid-scale features output by the hybrid expert submodules, transforming the shared feature stream generated by globally shared parameters into a specific representation adapted to the characteristics of each backbone module (such as multi-head attention or feedforward networks in ViT), thereby removing the constraint of cross-module representation consistency. Here, "demultiplexing" specifically refers to the decoupling process from "shared pathways" to "independent paths" through non-shared convolutional operations; the output of the convolutional layer is called "demultiplexed features." "Local" refers to the modules or network structures inserted by the parameter fine-tuning module.

[0021] In existing technologies, traditional PEFT parameter fine-tuning methods suffer from approximation errors due to parameter sharing or low-rank mapping, which can easily lead to homogenization of feature transformations in multi-layer networks. While reducing parameters and computational overhead, this approach sacrifices the model's feature diversity and representational power, limiting its performance on complex tasks. Therefore, embodiments of this application provide an efficient parameter fine-tuning method for large visual models to address the aforementioned problems.

[0022] Figure 1 This is a flowchart of a method for efficient parameter fine-tuning. In one embodiment of this application, a method for efficient parameter fine-tuning of a large visual model includes the following steps: S100, acquires a pre-trained large visual model and image training data for fine-tuning; S200 inserts parameter fine-tuning modules into each multi-head attention module and feedforward neural network module of the pre-trained large visual model. Some parameters of the parameter fine-tuning module are shared globally among multiple modules at different levels, while other parameters are set independently for the inserted module to adapt to the task requirements of local features. S300 inputs image data into the pre-trained large visual model of the insertion parameter fine-tuning module to generate feature maps and corresponding predicted values; S400 calculates the cross-entropy loss based on the predicted values ​​and the true class labels of the image training data; S500 calculates the gradient based on the cross-entropy loss, fixes the parameters of the backbone network, and updates the parameters of the parameter fine-tuning module based on the gradient descent method. S600, repeat the above process of image input, feature generation, cross-entropy loss calculation, gradient solution and parameter update until the cross-entropy loss converges.

[0023] The embodiments described above, by introducing a parameter fine-tuning module, can significantly reduce the number of required parameters while maintaining high performance, achieving a better balance between model performance and storage resource consumption. This method is applicable to fields such as image recognition and can effectively improve the performance of pre-trained models in downstream tasks such as image recognition.

[0024] In order to obtain rich feature representation capabilities, in some specific embodiments of this application, the image data in S100 varies depending on the selected fine-tuning dataset.

[0025] For example: if EuroSAT is selected as the fine-tuning dataset, the image data here represents the 10 classes of satellite image datasets covered by that dataset; if Cifar100 is selected, it represents 100 common natural image datasets; if Clevr-count is selected, it represents structured image data used for counting tasks.

[0026] A large pre-trained model is constructed by loading the weights of a large pre-trained model that has been fully trained on a large dataset. In the field of deep learning, large-scale pre-trained models have become the foundation for various tasks.

[0027] These models typically learn rich feature representations through long-term training on large-scale datasets (such as ImageNet, COCO, and OpenAI's WebGPT). By loading the weights of these pre-trained models, a powerful base model can be quickly built as a starting point for fine-tuning downstream tasks.

[0028] The advantage of pre-trained large models lies in the fact that they have already captured rich low-level features (such as edges, textures, and shapes) and high-level features (such as semantic information and object recognition) through multiple iterations of optimization during training. These features have strong generalization capabilities for most visual tasks. By applying these general features to downstream tasks through transfer learning, the large amount of computational resources and time required to train a model from scratch can be avoided, thus significantly reducing training costs. In some specific embodiments of this application, the pre-trained model can be any deep neural network trained on a large-scale dataset, such as a convolutional neural network (CNN) or a visual Transformer (ViT). These models are widely used in computer vision and can efficiently process multimodal data such as images and videos. By utilizing the feature representation capabilities of these large models, the performance of downstream tasks (such as object detection, image classification, instance segmentation, and video analysis) can be significantly improved.

[0029] In transfer learning, it is usually necessary to fine-tune the pre-trained model to adapt to the new task requirements. Therefore, a parameter fine-tuning module is constructed and inserted into the pre-trained large-scale visual model. In some specific embodiments of this application, S300, inserting the parameter fine-tuning module into each multi-head attention module and feedforward neural network module of the pre-trained large-scale visual model can be achieved by the following steps: S301, for each multi-head attention module, learnable parameter matrices for generating query Q matrix, learnable parameter matrices for generating key K matrix, learnable parameter matrices for generating value V matrix, and linear mapping matrix are introduced into parallel parameter fine-tuning modules respectively. S302 introduces parallel parameter fine-tuning modules for each feedforward neural network module; Each parameter fine-tuning module has the same input as the corresponding parallel network structure, and their outputs are added together.

[0030] For example, such as Figure 2 The diagram shown is a structure diagram of a pre-trained large-scale vision model with parameter fine-tuning inserted. Within this constructed transfer framework, the implementation process is as follows: S101, Image data is subjected to layer normalization to obtain normalized features; S102, multiply the normalized features output from S101 by the pre-trained parameters. Obtain feature Q; S103, input the normalized features output from S101 into the parameter fine-tuning module to obtain the first fine-tuned feature; after fusing the first fine-tuned feature and feature Q, query Q is obtained; S104, multiply the output of S101 by the pre-trained parameters Obtain feature K; S105, input the normalized feature output from S101 into the parameter fine-tuning module to obtain the second fine-tuned feature; the second fine-tuned feature is fused with the adjustment K to obtain the key K; S106, multiply the output of S101 by the pre-trained parameters We obtain feature V; S107: Input the normalized feature output from S101 into the parameter fine-tuning module to obtain the third fine-tuned feature; fuse the third fine-tuned feature with feature V to obtain value V; S108, a multi-head attention mechanism, transforms the query Q and key K into attention matrices by scaling the dot product and performs a softmax operation. Then, the attention matrix obtained by the operation is multiplied with the value V to obtain the final output result. The outputs of S109 and S108 are fused with the original input image data and then normalized. S110 inputs the normalized features from S109 into the feedforward neural network module, which consists of a pre-trained parameter layer. And consists of the Dropout layer; The output of the S111 and S109 normalized feature input parameter fine-tuning modules is fused with the output of S110.

[0031] The embodiments described above in this application, by inserting a parallel parameter fine-tuning module into the key modules of the pre-trained model, can adapt to new tasks by fine-tuning only a small number of parameters, and integrate pre-trained features and fine-tuned features to efficiently utilize pre-trained knowledge and improve the model's performance on new tasks.

[0032] To achieve efficient fine-tuning while maintaining low computational overhead, this application designs a parameter fine-tuning module. Some parameters of this module are globally shared among multiple modules of the pre-trained large-scale visual model, while others are independently adjusted using local features to adapt to the task requirements of the corresponding module. In some specific embodiments of this application, the parameter fine-tuning module includes an encoder, a hybrid expert submodule, a local feature demultiplexing layer, and a decoder. The encoder, hybrid expert submodule, and decoder share the same weights.

[0033] Figure 3 The processing flow of the parameter fine-tuning module is shown, which includes the following steps. S201, in the encoder: the input feature map is mapped to a low-dimensional space to reduce the dimensionality. It consists of a fully connected layer, and the input and output dimensions are the output feature dimension of the corresponding backbone network module and the set low-rank space dimension, respectively.

[0034] Specifically, the encoder's main task is to map input features from a high-dimensional space to a low-rank space. The low-rank space is constructed using low-rank matrix factorization, a method that effectively reduces the dimensionality of the feature space, thereby lowering computational overhead. By mapping to the low-rank space, the encoder compresses the complexity of the feature representation, reduces the number of parameters that need optimization, and ensures a more efficient fine-tuning process. Low-rank encoding enables the model to quickly adapt to different downstream tasks without sacrificing performance.

[0035] In S202, the hybrid expert submodule extracts multi-scale features from the dimensionality-reduced input feature map using multiple expert units. A gating unit selects expert units based on feature relevance, and a fusion unit weightedly combines the selected expert units to output hybrid-scale features.

[0036] S203, in the local demultiplexing convolutional layer. This convolutional layer consists of a single two-dimensional convolutional layer, whose input and output channel numbers are consistent with the low-rank spatial dimension. This module is used to adjust the mixed-scale features, demultiplexing each local network module to obtain demultiplexed features.

[0037] Specifically, the adjustment here refers to using convolutional layers with a non-shared kernel size of 3 for convolution, to obtain de-multiplexed features for further information aggregation. S204, Decoder: Maps demultiplexed features back to a high-dimensional space. It consists of a single fully connected network layer, with the input and output dimensions being the address space dimension and the corresponding backbone network module output feature dimension, respectively.

[0038] Specifically, the decoder is responsible for restoring features from the low-rank space to the high-dimensional space. This process recovers the compressed information in the low-rank space, ensuring that the feature representation meets the requirements of downstream tasks. While preserving the compressed feature information, the decoder ensures the integrity and accuracy of the feature representation required by the task, avoiding the loss of key information.

[0039] In the embodiments described above, the global sharing modules (encoder, hybrid expert submodule, and decoder) allow most parameters to be shared between different layers. This sharing not only reduces the model's storage requirements but also reduces computational overhead by minimizing redundant parameters. Specifically, parameters in the pre-trained model can be reused across multiple network layers, meaning that the same feature representations are shared by multiple layers instead of being stored separately for each layer, thus significantly reducing storage and computational costs. While the global sharing components reduce redundancy, they may lead to an overly uniform approach to all tasks, failing to effectively adapt to the needs of different tasks. Therefore, the embodiments described above introduce locally demultiplexed convolutional layers. This module focuses on the local adjustment of features at each layer, performing convolution operations and decoding in a low-dimensional feature space. This allows the model to specifically adjust the features of certain layers during fine-tuning without requiring retraining the entire model.

[0040] By using a globally shared module and locally demultiplexed convolutional layers, the above embodiments can maintain the efficiency brought by global sharing while ensuring that each layer can be adaptively adjusted according to task requirements.

[0041] Therefore, the embodiments described above are particularly suitable for resource-constrained application scenarios, such as edge computing platforms like mobile devices and IoT terminals. On these devices, computing power and storage space are limited, while this application can achieve high-performance visual model fine-tuning and inference with almost no additional burden, demonstrating broad application potential.

[0042] The aforementioned locally demultiplexed convolutional layers address the homogenization problem caused by parameter sharing. In addition, traditional efficient parameter fine-tuning effectively reduces the number of parameters through low-rank matrix factorization, but these methods often lead to homogenization in feature transformations between layers, similarly limiting the model's expressive power and task adaptability. To address this issue, this application introduces a hybrid expert system. Figure 4 The flowchart for the hybrid expert submodule is as follows: S2021, the expert unit, consists of an interpolation module, a convolutional layer, and an optional pooling submodule.

[0043] For any expert unit, the input dimensionality-reduced feature map is first resampled (i.e., interpolated) to match its target scale. Then, features at that scale are extracted using convolutional layers with both input and output channels equal to the low-rank spatial dimension. If the expert unit aims to capture low-resolution semantic information, pooling operations adapted to the resampling ratio are further performed to complete multi-scale feature extraction.

[0044] Specifically, the hybrid expert module comprises multiple independent experts, each processing features at a specific scale. By independently learning features at different scales, features can be extracted and processed at different levels, avoiding the feature homogenization problem that may occur when sharing weights. Each expert module extracts features through interpolation operations at a specific scale and applies convolutional layers for feature extraction, thereby enhancing the diversity of feature representations.

[0045] Furthermore, the features of the input data may have diverse scales. The hybrid expert module can simultaneously focus on features at different scales and capture the inherent patterns of the data from multiple perspectives. Each expert focuses on processing at a different scale, enabling the model to understand the input data from a local to a global, multi-layered perspective.

[0046] S2022, a gating unit, includes a gating linear subnetwork, a noise sensing subnetwork, and an output submodule.

[0047] The input features are used to generate basic weights that reflect the confidence of each expert unit through a gated linear network.

[0048] Example: w base =Wgx x∈R d The input is the reduced-dimensional feature vector; W g ∈R N×d is the learnable weight matrix of the gated linear subnetwork; w base ∈R N It is the basic weight vector, reflecting the prediction confidence of each expert unit; N represents the total number of expert units.

[0049] The output of the input features after passing through the noise-aware linear network is activated by the Softplus function and then multiplied element-wise with independently sampled standard Gaussian noise to obtain the perturbation weight term. Example: s = Softplus(W n x) = log(1 + e Wnx ) ; W n ∈R N×d Here is the learnable weight matrix for the noise-aware subnetwork; s∈R N The non-negative gain coefficient after Softplus activation; A standard Gaussian noise vector that is sampled independently; ⊙ indicates element-wise multiplication (Hadamard product); w noise ∈R N This is the perturbation weight term.

[0050] The output submodule merges and adds the base weights and perturbation weights to obtain the original expert weights. It selects the top K terms with the largest values ​​from the original expert weights and sets the rest to zero. It then normalizes the non-zero weights and outputs the final expert selection weights.

[0051] Both the gated linear network and the noisy linear network consist of a single fully connected layer, and their input-output dimensions are consistent with the low-rank space dimension.

[0052] S2023, Fusion Unit: The fusion unit only performs weighted fusion on the multi-scale features output by the target expert unit output by S2022 to generate the final feature representation, i.e., mixed-scale features.

[0053] In the embodiments described above, the gating unit dynamically calculates the expert selection weights based on the matching degree between the input features and each expert unit, and only activates the top K experts with the highest response confidence. This mechanism ensures that during model inference, only experts highly relevant to the current input semantics are invoked and executed, while the remaining experts are skipped and their outputs are set to zero. This significantly reduces redundant computations, lowers computational overhead and memory access burden, and is particularly suitable for resource-constrained scenarios such as edge devices.

[0054] Unlike traditional fixed-structure transfer learning methods, the embodiments described in this application employ an input-adaptive expert selection mechanism using gating units. This allows the model to dynamically combine the most suitable multi-scale feature extraction paths based on different tasks and input content. Multiple expert units focus on feature modeling at different spatial granularities (such as details, structure, and semantics), and form rich and diverse representations through gating and weighted fusion. This on-demand activation and collaborative expression mechanism effectively alleviates the feature homogenization problem caused by low-rank mapping, significantly improving the model's generalization performance and adaptability in complex downstream tasks.

[0055] Based on the same technical concept, some specific embodiments of this application, such as Figure 5 As shown, a high-efficiency parameter fine-tuning system 100 for large visual models is provided, comprising: Preliminary preparation module 110: Acquire pre-trained large visual model and image training data for fine-tuning; Insertion module 120: In each multi-head attention module and feedforward neural network module of the pre-trained visual large model, a parameter fine-tuning module 130 is inserted respectively. Some parameters of the parameter fine-tuning module are shared globally among multiple modules at different levels, while other parameters are set independently for the insertion module to adapt to the task requirements of local features. Generation module 140: Inputs image data into the pre-trained large visual model of the insertion parameter fine-tuning module to generate feature maps and corresponding predicted values; Loss module 150: Calculates cross-entropy loss based on predicted values ​​and the true class labels of the image training data; Fine-tuning module 160: Calculates the gradient based on the cross-entropy loss, fixes the parameters of the backbone network, and updates the parameters of the parameter fine-tuning module based on the gradient descent method; Repeat the above process of image input, feature generation, cross-entropy loss calculation, gradient solving, and parameter update until the cross-entropy loss converges.

[0056] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the efficient fine-tuning method for the parameters of the large visual model in the above embodiments, and will not be repeated here.

[0057] The above methods can effectively improve the performance of pre-trained models on downstream tasks such as image recognition. One embodiment of this application provides an image recognition method, including the following steps: Step 1: Obtain the target image data to be identified; Step 2: Based on the efficient fine-tuning method of the visual large model, the pre-trained visual large model is fine-tuned to obtain the fine-tuned visual large model.

[0058] Step 3: Input the target image data into the fine-tuned visual model, generate category prediction results through forward propagation, and complete the recognition of the target image.

[0059] Specifically, after the fine-tuning phase, the optimized basic units (parameter fine-tuning modules) are combined with the pre-trained model to ultimately construct a target neural network optimized for a specific downstream task. This network combines the knowledge learned from the pre-trained model with the fine-tuned parameter fine-tuning modules, thereby performing the downstream task while maintaining minimal parameter overhead.

[0060] The final target neural network delivers superior performance across multiple vision tasks through efficient fine-tuning and knowledge transfer, without requiring significant computational resources. This optimized network is particularly well-suited for applications requiring real-time inference and low latency, such as mobile devices and edge computing devices.

[0061] After constructing the final target neural network, various types of image data can be input into the final structure to complete downstream specific tasks. These downstream tasks can include image-related classification or processing sequential signals.

[0062] After continuous training and iterative updates of network parameters, two structural units are finally obtained: one is the basic pre-trained model, and the other is the added parameter fine-tuning module. The two target structures are outputs of the structure preservation module and then output to the structure evaluation module. To illustrate the technical effectiveness of the above method, in one embodiment of this application, the results shown are the classification accuracy after the model has been trained for 50-200 epochs to reach convergence. The method of this application achieves an average accuracy of 76.8 on the VTAB-1K benchmark with 0.07M parameters, and this accuracy can be achieved when transferred to the FGVC few-shot benchmark. This search process can be run on a single Nvidia 2080 GPU. Furthermore, the smallest parameter fine-tuning module only requires 0.009M parameters to achieve an average accuracy of 74.5 on the VTAB-1K benchmark. Therefore, this application shows significant improvements in model structure size, computational cost, and performance.

[0063] For detailed comparison results, please see [link / details]. Figure 6 and Figure 7 As shown, Figure 6 The results obtained in the embodiments of this application are evaluated and compared (on the VTAB-1K benchmark). Figure 7 The table presents a comparison of the results obtained in the few-shot image classification example of this application (on the FGVC few-shot benchmark). Higher accuracy in the table represents better performance, and smaller parameter and computational costs represent more efficient structures. GLEAM-T, B, and L represent three different scales of this application, differing in that their low-rank space dimensions are 4, 8, and 16, respectively.

[0064] The above experiments show that the proposed parameter fine-tuning method in the embodiments of this application has good performance and can significantly reduce the number of parameters, and has a wide range of application scenarios.

[0065] In another embodiment of this application, an image classification method is also provided. Image classification here refers to inputting an image to be processed and outputting a pre-designed category as a label. It is an image processing method that distinguishes different categories of targets based on their different features reflected in the image information. The sample data used for image classification can be a large amount of image data to be classified.

[0066] Specifically, this embodiment uses training images from the public dataset VTAB-1K as input data for image classification as an example, including... Specifically, taking the image classification in the dataset included in the VTAB-1K benchmark as an example, the specific process of the image classification method is as follows: M1 acquires different image data and corresponding classification labels, and uses the image label data as the overall model training data; M2, based on the image training data obtained from M1, inserts the parameter fine-tuning module into the pre-trained model to obtain the overall model, and then trains it to obtain the overall model for transfer to downstream tasks; M3 uses the overall model obtained from M2 to detect the image data to be classified, obtains the predicted probability of each category, and then converts the probability into the final classification label as the classification result.

[0067] The aforementioned M2 can be achieved using the efficient parameter fine-tuning method for the large visual model described above.

[0068] In another embodiment of this application, a few-shot learning method is also provided. Here, few-shot learning refers to a dataset containing only a specified number of samples for each category. The specified number of samples are then input into the model, enabling it to learn limited category information and output a category as a label. This is a learning method that simulates real-world scenarios by using image information with extremely limited samples to maximize model performance in the corresponding scenario.

[0069] Specifically, taking the training images from the publicly available dataset contained in the FGVC few-shot benchmark as model input to achieve few-shot learning as an example, the specific process of the method is as follows: (1) Obtain the image training data of the dataset.

[0070] In this embodiment, the image training data is the training image data from the public dataset in the FGVC few shot benchmark.

[0071] (2) Based on different random number seeds and sample size, sample the corresponding small sample training data from the dataset.

[0072] (3) Based on the small sample dataset obtained in (2), the parameter fine-tuning module is inserted into the pre-trained model to obtain the overall model, which is then trained to obtain the overall model for transfer to downstream tasks. The network update module uses the gradient descent method to continuously update the parameters of the parameter fine-tuning module based on the image classification loss function (usually multi-class cross-entropy), and combines the updated parameters with the parameters of the pre-trained model for subsequent inference; The overall network's inference features retain both the pre-trained model's processing results of the input features and the parameter fine-tuning module's processing results of the input features. The two are combined in a given ratio to serve as the output features of the overall final model.

[0073] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or run the above-described system.

[0074] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0075] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0076] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0077] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0078] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0079] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0080] In summary, this application presents an innovative transfer learning framework designed to address the challenges of fine-tuning large-scale pre-trained models for downstream tasks by improving computational efficiency and reducing storage overhead. Traditional transfer learning methods typically require fine-tuning the entire pre-trained model, which not only consumes significant computational resources but also easily generates redundant parameters. To address this issue, this application employs a global sharing and local de-reuse strategy, reducing the number of model parameters while maintaining high efficiency, making it particularly suitable for resource-constrained devices and scenarios.

[0081] Regarding parameter efficiency, this framework reduces redundant storage and computational overhead by sharing most parameters across multiple layers. Unlike traditional methods, the local demultiplexing module in this application performs convolutional operations on low-dimensional features of specific layers, enabling each layer to be refined and optimized according to task requirements. This design significantly reduces the number of parameters that need to be updated during model fine-tuning, lowers training time and resource consumption, and allows large-scale pre-trained models to run efficiently in resource-constrained environments such as edge devices and mobile devices.

[0082] Furthermore, this application framework introduces a hybrid expert (MoE) mechanism, effectively addressing the feature homogenization problem. The hybrid expert module, through a multi-scale sparse expert mechanism, allows each expert module to focus on processing features at a specific scale, thereby enhancing the diversity of feature transformations. Each expert module independently processes input features through interpolation, convolution, and pooling operations, avoiding excessive similarity in feature transformations between layers in traditional methods. In this way, this application not only improves the model's adaptability to different tasks but also enhances its expressive power, enabling it to better handle complex downstream tasks.

[0083] Finally, the framework presented in this application demonstrates superior performance in multiple vision tasks, such as image classification and few-shot learning. By combining a pre-trained large model with an efficiently fine-tuned adapter module, this application achieves efficient inference and accurate prediction in downstream tasks while maintaining low parameter overhead. Its advantages are particularly evident in scenarios with limited edge computing devices and computational resources, demonstrating broad application potential.

[0084] Overall, the framework presented in this application effectively improves the efficiency, stability, and adaptability of transfer learning through innovative design. It not only solves the homogenization problem in traditional methods but also achieves low computational overhead and high performance across multiple tasks, demonstrating its broad application prospects in resource-constrained environments.

[0085] The specific embodiments of this application have been described above. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application.

Claims

1. A method for efficient fine-tuning of parameters in a large visual model, characterized in that, include: Acquire a pre-trained large visual model and image training data for fine-tuning; In each multi-head attention module and feedforward neural network module of the pre-trained large visual model, a parameter fine-tuning module is inserted. Some parameters of the parameter fine-tuning module are shared globally among multiple modules at different levels, while other parameters are set independently for the inserted module to adapt to the task requirements of local features. The image training data is input into a pre-trained large visual model with a parameter fine-tuning module inserted to generate feature maps and corresponding predicted values. Based on the predicted values ​​and the true class labels of the image training data, calculate the cross-entropy loss; The gradient is calculated based on the cross-entropy loss. The parameters of the backbone network of the pre-trained large visual model are fixed, and the parameters of the parameter fine-tuning module are updated based on the gradient descent method. Repeat the above process of image input, feature generation, cross-entropy loss calculation, gradient solving, and parameter update until the cross-entropy loss converges.

2. The efficient parameter fine-tuning method for large visual models according to claim 1, characterized in that, The step of inserting parameter fine-tuning modules into each multi-head attention module and feedforward neural network module of the pre-trained large visual model includes: For each of the multi-head attention modules, a learnable parameter matrix for generating the query Q matrix, a learnable parameter matrix for generating the key K matrix, a learnable parameter matrix for generating the value V matrix, and a linear mapping matrix are introduced into a parallel parameter fine-tuning module. Parallel parameter fine-tuning modules are introduced for each feedforward neural network module; Each parameter fine-tuning submodule receives the same input as its parallel structure and merges its output with the output of the parallel network structure element by element.

3. The efficient fine-tuning method for parameters of a large visual model according to claim 1, characterized in that, The parameter fine-tuning module includes an encoder, a hybrid expert submodule, a locally demultiplexed convolutional layer, and a decoder connected in sequence. The encoder maps the input feature map to a low-dimensional space to obtain a dimensionality-reduced feature map; The hybrid expert submodule extracts multi-scale features from the dimensionality-reduced feature map, uses the attention weights corresponding to the feature correlations, and uses the attention weights to perform weighted fusion of the multi-scale features to output hybrid scale features. The local demultiplexed convolutional layer adjusts the mixed-scale features, removes the representation consistency constraint caused by parameter sharing, and generates features that are adapted to the task requirements of each module at its respective level. The decoder maps the features adapted to the task requirements of each module's level back to a high-dimensional space, making its output dimension consistent with the original output feature dimension of the corresponding backbone network module.

4. The efficient parameter fine-tuning method for large visual models according to claim 3, characterized in that, The hybrid expert submodule includes an expert unit, a gating unit, and a fusion unit connected in sequence. Multiple expert units configured in parallel are set up with different spatial scales, which can extract multi-scale features by interpolating and convolutionally transforming the reduced feature map. The gating unit calculates the expert selection weights based on the input dimensionality-reduced feature map, and selects one or more expert units as target expert units to participate in feature extraction according to the expert selection weights; The fusion unit only performs weighted fusion on the multi-scale features output by the target expert unit to generate the final feature representation, namely the hybrid scale feature.

5. The efficient parameter fine-tuning method for large visual models according to claim 4, characterized in that, The gating unit includes: A gated linear subnetwork generates the basic weights of each expert unit based on the input dimensionality-reduced feature map; The noise-aware subnetwork generates perturbation weights coupled with standard Gaussian noise based on the input dimensionality-reduced feature map; The output submodule merges the base weights and the perturbation weights to obtain the original expert weights. It selects the top K terms with the largest values ​​from the original expert weights and sets the remaining terms to zero. It then normalizes the non-zero weights and outputs the final expert selection weights.

6. The efficient parameter fine-tuning method for large visual models according to claim 3, characterized in that, The encoder, hybrid expert submodule, and decoder in all parameter fine-tuning modules share the same learnable parameters; the locally demultiplexed convolutional layers are configured independently for each module and have non-shared weights.

7. A highly efficient parameter fine-tuning system for large visual models, characterized in that, include: Preliminary preparation module: Acquire pre-trained large visual model and image training data for fine-tuning; Insertion Module: In each multi-head attention module and feedforward neural network module of the pre-trained large visual model, a parameter fine-tuning module is inserted respectively. Some parameters of the parameter fine-tuning module are shared globally among multiple modules at different levels, while other parameters are set independently for the insertion module to adapt to the task requirements of local features. Generation module: Inputs the image data into the pre-trained large visual model inserted into the parameter fine-tuning module to generate feature maps and corresponding predicted values; Loss module: Calculates cross-entropy loss based on the predicted value and the true class labels of the image training data; Fine-tuning module: Calculates the gradient based on the cross-entropy loss, fixes the parameters of the backbone network of the pre-trained visual large model, and updates the parameters of the parameter fine-tuning module based on the gradient descent method; Loop module: Repeat the above process of image input, feature generation, cross-entropy loss calculation, gradient solving and parameter update until the cross-entropy loss converges.

8. An image recognition method, characterized in that, include: Acquire the target image data to be identified; Based on the efficient parameter fine-tuning method for the large visual model according to any one of claims 1 to 6, the pre-trained large visual model is fine-tuned to obtain the fine-tuned large visual model. The target image data is input into the fine-tuned visual model, and a category prediction result is generated through forward propagation to complete the recognition of the target image.

9. A terminal, characterized in that, It includes a memory and a processor, the memory storing a program executed by the processor, which, when executing the program, can perform any of the following methods or systems: The efficient parameter fine-tuning method for large visual models according to any one of claims 1-6, or, The efficient parameter fine-tuning system for large visual models as described in claim 7, or, The image recognition method according to claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When this program is executed by the processor, it can execute any of the following methods or systems: The efficient parameter fine-tuning method for large visual models according to any one of claims 1-6, or, The efficient parameter fine-tuning system for large visual models as described in claim 7, or, The image recognition method according to claim 8.

Images