Vision language model fine-tuning method and device based on adaptive parameter selection

Abstract

The application provides a visual language model fine-tuning method and device based on adaptive parameter selection, and relates to the technical field of computer vision and natural language processing. The method comprises the following steps: obtaining a pre-trained target model, and performing optimization training on the target model based on a target special task; in the process of optimization training, the importance of each model parameter of the model relative to the target special task is determined by using the second moment of each model parameter estimated by an optimizer; based on the importance of each model parameter relative to the target special task, a plurality of to-be-adjusted model parameters that need to be adjusted are determined, and each to-be-adjusted model parameter in the plurality of to-be-adjusted model parameters is optimized and adjusted. The visual language model fine-tuning method and device based on adaptive parameter selection provided by the application can efficiently fine-tune by adaptively selecting key parameters while keeping the original architecture unchanged, greatly improving the efficiency of model parameter optimization and adjustment.

Description

Technical Field

[0001] This application relates to the fields of computer vision and natural language processing technology, and in particular to a method and apparatus for fine-tuning a visual language model based on adaptive parameter selection. Background Technology

[0002] Vision-Language Models (VLMs) combine visual and linguistic modalities to address cross-modal understanding between images and text. With the continuous development of deep learning technology, especially the rise of pre-trained models and self-supervised learning methods, VLMs have made significant progress in various tasks. For example, the CLIP (Contrastive Language-Image Pretraining) model achieves efficient image-to-text retrieval and text-to-image generation by mapping images and text to the same feature space. Similarly, DALL·E and DALL·E 2 generate high-quality images through natural language descriptions, demonstrating the enormous potential of VLMs in creative design and artistic creation.

[0003] However, in related technologies, parameter optimization for visual language models mainly includes two methods: full parameter fine-tuning and manually preset fine-tuning positions. However, full parameter fine-tuning may lead to a surge in computational resource consumption, especially in scenarios with scarce data, which can easily cause overfitting. Manually preset fine-tuning positions may have problems such as suboptimal position selection and poor task adaptability, and the additional learnable modules introduced increase the model complexity.

[0004] Therefore, there is an urgent need for a parameter optimization method for visual language models to improve the efficiency of model parameter optimization and adjustment. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for fine-tuning a visual language model based on adaptive parameter selection, which can efficiently fine-tune key parameters by adaptively selecting them while keeping the original architecture unchanged, thus greatly improving the efficiency of model parameter optimization and adjustment.

[0006] This application provides a method for fine-tuning a visual language model based on adaptive parameter selection, including:

[0007] A pre-trained target model is obtained, and the target model is optimized and trained based on a target specific task; the target model is a visual language model; during the optimization training process, the second moment of each model parameter estimated by the optimizer is used to determine the importance of each model parameter relative to the target specific task; based on the importance of each model parameter relative to the target specific task, multiple model parameters to be adjusted are determined, and each of the multiple model parameters to be adjusted is optimized and adjusted.

[0008] Optionally, during the optimization training process, determining the importance of each model parameter relative to the target specific task by utilizing the second moment of each model parameter estimated by the optimizer includes: forward calculation of the cosine similarity matrix between image features and text features, and calculation of the cross-entropy loss value based on the cosine similarity matrix; obtaining the gradient of each model parameter through backpropagation based on the cross-entropy loss value, and updating the second moment estimate of each model parameter based on the gradient of each model parameter; calculating the bias-corrected second moment based on the second moment estimate of each model parameter, and generating an importance score for each model parameter based on the second moment of each model parameter.

[0009] Optionally, the importance score for each model parameter is calculated based on the following formula:

[0010]

[0011] in, For the first i Importance score of each model parameter For: the i The second moments of each model parameter after bias correction; This is used to prevent the denominator from being zero in the formula.

[0012] Optionally, determining multiple model parameters to be adjusted based on the importance of each model parameter relative to the target task includes: grouping the intra-layer model parameters of the Transformer layer in the target model according to type, and determining a preset number of model parameters with the highest importance score in each group as model parameters to be adjusted.

[0013] Optionally, optimizing each of the plurality of model parameters to be adjusted includes: adaptively training the target model using target data related to the target task, and optimizing the plurality of model parameters to be adjusted using a target loss function corresponding to the target task during the adaptive training process.

[0014] This application also provides a device for fine-tuning a visual language model based on adaptive parameter selection, comprising:

[0015] An acquisition module is used to acquire a pre-trained target model and optimize the target model based on a target specific task; the target model is a visual language model; an importance assessment module is used to determine the importance of each model parameter relative to the target specific task by using the second moment of each model parameter estimated by the optimizer during the optimization training process; a parameter optimization module is used to determine multiple model parameters that need to be adjusted based on the importance of each model parameter relative to the target specific task, and optimize each of the multiple model parameters to be adjusted.

[0016] Optionally, the importance assessment module is specifically used to forward calculate the cosine similarity matrix between image features and text features, and calculate the cross-entropy loss value based on the cosine similarity matrix; the importance assessment module is further used to obtain the gradient of each model parameter through backpropagation based on the cross-entropy loss value, and update the second moment estimate of each model parameter based on the gradient of each model parameter; the importance assessment module is further used to calculate the bias-corrected second moment based on the second moment estimate of each model parameter, and generate an importance score for each model parameter based on the second moment of each model parameter.

[0017] Optionally, the parameter optimization module is specifically used to group the intra-layer model parameters of the Transformer layer in the target model according to type, and determine the preset number of model parameters with the highest importance score in each group as the model parameters to be adjusted.

[0018] Optionally, the parameter optimization module is specifically used to adaptively train the target model using target data related to the target task, and during the adaptive training process, to optimize and adjust the multiple model parameters to be adjusted using a target loss function corresponding to the target task.

[0019] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the visual language model fine-tuning method based on adaptive parameter selection as described above.

[0020] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the visual language model fine-tuning method based on adaptive parameter selection as described above.

[0021] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual language model fine-tuning method based on adaptive parameter selection as described above.

[0022] The visual language model fine-tuning method and apparatus based on adaptive parameter selection provided in this application first obtains a pre-trained target model and optimizes the target model based on a target specific task; the target model is a visual language model. Then, during the optimization training process, the importance of each model parameter relative to the target specific task is determined using the second moment estimated by the optimizer. Finally, based on the importance of each model parameter relative to the target specific task, multiple model parameters to be adjusted are determined, and each of these multiple model parameters is optimized. In this way, efficient fine-tuning can be performed by adaptively selecting key parameters while maintaining the original architecture, greatly improving the efficiency of model parameter optimization. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the system architecture of the visual language model fine-tuning method based on adaptive parameter selection provided in this application;

[0025] Figure 2 This is a flowchart illustrating the visual language model fine-tuning method based on adaptive parameter selection provided in this application;

[0026] Figure 3 This is a schematic diagram of the structure of the visual language model fine-tuning device based on adaptive parameter selection provided in this application;

[0027] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] In related technologies, cross-modal models, represented by the CLIP model, have achieved general feature representation through large-scale image-text contrastive learning. However, in practical applications, they face the following technical bottlenecks: 1. Low efficiency of full-parameter fine-tuning: Directly fine-tuning the entire model leads to a surge in computational resource consumption, especially prone to overfitting in data-scarce scenarios; 2. Limitations of manually preset fine-tuning positions: Parameter Efficient Fine-Tune (PEFT) technology relies on manually preset fine-tuning positions, resulting in suboptimal position selection, poor task adaptability, and increased model complexity due to the additional learnable modules introduced; 3. Lack of dynamic selection mechanisms: Key parameters cannot be automatically identified based on task characteristics, and fixed parameter masking strategies disrupt the model's gradient propagation path, leading to reduced knowledge transfer efficiency. These shortcomings severely restrict the deployment and application of visual language models in resource-constrained scenarios.

[0031] To address the aforementioned technical problems in related technologies, this application provides a method for fine-tuning a visual language model based on adaptive parameter selection. This method aims to solve problems such as suboptimal manual tuning positions and efficiency degradation due to the introduction of additional parameters in existing efficient parameter tuning methods, achieving efficient task adaptation without modifying the model structure. The core idea of ​​this invention is to automatically identify key parameters through dynamic gradient analysis of the optimizer and combine it with a hierarchical selective update mechanism to improve task-specific performance while retaining pre-training knowledge. Specifically, it includes the following core components:

[0032] 1. Dynamic Parameter Importance Assessment Module: This module automatically quantifies the importance of each parameter in the model based on the gradient second moment characteristics of the AdamW optimizer. By analyzing the stability of the gradient second moment estimation, it dynamically identifies "key parameters" sensitive to the current task. This process requires no manual intervention, overcoming the shortcomings of traditional methods that rely on prior knowledge to select the optimization position.

[0033] 2. Hierarchical Selective Fine-Tuning Mechanism: Addressing the hierarchical feature extraction characteristics of the Transformer layers in the CLIP model, hierarchical parameter optimization is implemented. Importance scores are independently calculated for the parameters of sub-modules within each Transformer layer (e.g., the Q / K / V projection matrix of the self-attention layer), and these scores are sorted in descending order. A dynamic unfreezing strategy is employed, unfreezing only the top K key parameters (typically K=6) of each layer for gradient updates, while the remaining parameters remain frozen, ensuring knowledge preservation and computational efficiency. Zero structural modification is maintained throughout the process, without inserting any adapters or hint modules, preserving the original CLIP model architecture and parameter scale.

[0034] like Figure 1 The diagram shows a system architecture schematic of the visual language model fine-tuning method based on adaptive parameter selection provided in this application embodiment, including: (a) Transformer fine-tuning and (b) and (c) adaptive selective fine-tuning stages. In (a), the Transformer layers of the visual and text encoders can be fine-tuned to extract parameter importance scores; (b) illustrates the process of ranking sub-layer parameters using importance scores, followed by adaptive selection of the top K sub-layers; (c) demonstrates the selective fine-tuning process, where only the most important sub-layers are updated, while other components remain frozen, improving efficiency without affecting model performance.

[0035] The following description, in conjunction with the accompanying drawings, details the method for fine-tuning a visual language model based on adaptive parameter selection provided in this application through specific embodiments and application scenarios.

[0036] like Figure 2 As shown in the embodiment of this application, a method for fine-tuning a visual language model based on adaptive parameter selection is provided. This method may include the following steps 201 to 203:

[0037] Step 201: Obtain the pre-trained target model and optimize the target model based on the target specific task.

[0038] The target model is a visual language model, and the CLIP model is used as an example in this embodiment.

[0039] It is understood that the visual language model fine-tuning method based on adaptive parameter selection provided in this application first requires pre-training the visual language model to obtain the aforementioned target model. In this application embodiment, the CLIP model is used as an example. The CLIP model, pre-trained with large-scale image and text data pairs, can map images and text to a shared high-dimensional feature space. In this application embodiment, the pre-trained CLIP model is used for preliminary feature extraction, mainly consisting of an image encoder and a text encoder.

[0040] Step 202: During the optimization training process, the importance of each model parameter relative to the target specific task is determined by using the second moment of each model parameter estimated by the optimizer.

[0041] For example, after obtaining the pre-trained target model, the target model is then trained for a specific task to optimize and adjust the model parameters. During the parameter tuning process, conventional gradient calculation is performed first. During training, the AdamW optimizer, which features adaptive learning rate, is used. The AdamW optimizer estimates the first and second moments of each parameter. By calculating the second moment, the magnitude of change for each parameter can be obtained, thereby assessing the importance of that parameter in the current task.

[0042] Specifically, step 202 above may also include steps 202a1 to 202a3:

[0043] Step 202a1: Calculate the cosine similarity matrix between image features and text features, and calculate the cross-entropy loss value based on the cosine similarity matrix.

[0044] Step 202a2: Obtain the gradient of each model parameter through backpropagation based on the cross-entropy loss value, and update the second moment estimate of each model parameter based on the gradient of each model parameter.

[0045] Step 202a3: Calculate the bias-corrected second moment based on the second moment estimate of each model parameter, and generate an importance score for each model parameter based on the second moment of each model parameter.

[0046] For example, during the initial fine-tuning phase (1 epoch), the following operations can be performed:

[0047] S1, Forward computation of image features G Text features W The cosine similarity matrix is ​​then used to calculate the cross-entropy loss using the following formula:

[0048] (Formula 1)

[0049] in, τ =0.01 is the temperature coefficient; N The size of the batch data; C Number of categories; k For a specific category among all categories; For the first i The first sample j Labels for each category.

[0050] S2. Based on the calculated cross-entropy loss, backpropagation is performed to obtain the gradients of each model parameter. The second moment estimates of the model parameters are updated using the following formula 2. :

[0051] (Formula 2)

[0052] in, It is one of the core hyperparameters of the Adam optimizer, which determines the decay rate of the exponentially weighted average of the squared gradient, directly affecting the stability and convergence speed of model parameter updates.

[0053] S3. Then, calculate the second moment after deviation correction using the following formula:

[0054] (Formula 3)

[0055] S4. Finally, calculate the importance score for each model parameter using the following formula:

[0056] (Formula 4)

[0057] in, For the first i Importance score of each model parameter For: the i The second moments of each model parameter after bias correction; To prevent errors in the denominator, a smaller value is used to prevent the denominator from being zero in the formula.

[0058] Step 203: Based on the importance of each model parameter to the target task, determine a number of model parameters that need to be adjusted, and optimize each of the multiple model parameters to be adjusted.

[0059] For example, after calculating the importance score for each parameter, the K most important parameters are automatically selected for further tuning based on the scores. Parameters with lower importance are frozen and do not participate in the tuning process.

[0060] Specifically, step 203 above, which involves determining the multiple model parameters that need to be adjusted, may include the following step 203a:

[0061] Step 203a: Group the intra-layer model parameters of the Transformer layer in the target model according to type, and determine the preset number of model parameters with the highest importance score in each group as the model parameters to be adjusted.

[0062] For example, to determine which parameters need optimization, the first step is to sort the model parameters within each Transformer layer, that is, to sort the parameters for each Transformer layer. Group the parameters within the layer by type:

[0063] The self-attention module includes: Projection matrix weights and bias Feedforward network, including: upgraded matrix And dimensionality reduction matrix Layer normalization, including scaling factor. Offset .

[0064] Then, the average score of each set of model parameters is calculated independently, and the K parameter groups with the highest importance scores in each layer (typical value K=6) are selected. The remaining parameters are set to a frozen state and do not participate in the tuning process.

[0065] For example, after selective tuning, task-specific tuning is performed on the selected K parameters. This process is similar to regular model training, but because only the important parameters are tuned, training efficiency is greatly improved.

[0066] Specifically, step 203 above, which involves optimizing and adjusting each of the plurality of model parameters to be adjusted, may further include the following step 203b:

[0067] Step 203b: Adaptively train the target model using target data related to the target task, and optimize and adjust the multiple model parameters to be adjusted using the target loss function corresponding to the target task during the adaptive training process.

[0068] For example, optimizing model parameters can include the following two aspects: 1. Task-adaptive training: Continue to fine-tune the model using task-specific data. The goal of fine-tuning is to optimize the model's performance on a specific task, further improving accuracy and task adaptability. 2. Loss function optimization: Typically, cross-entropy loss function (for classification tasks) or other task-related loss functions are used for training to optimize the model's performance on the task.

[0069] The visual language model fine-tuning method based on adaptive parameter selection provided in this application first obtains a pre-trained target model and optimizes it based on a target specific task; the target model is a visual language model. Then, during the optimization training process, the importance of each model parameter relative to the target specific task is determined using the second moment estimated by the optimizer. Finally, based on the importance of each model parameter relative to the target specific task, multiple model parameters to be adjusted are determined, and each of these parameters is optimized. In this way, while maintaining the original architecture, efficient fine-tuning can be performed by adaptively selecting key parameters, greatly improving the efficiency of model parameter optimization.

[0070] It should be noted that the visual language model fine-tuning method based on adaptive parameter selection provided in this application can be executed by a visual language model fine-tuning device based on adaptive parameter selection, or by a control module within that device for executing the method. This application uses the execution of the method by the device as an example to illustrate the visual language model fine-tuning device based on adaptive parameter selection provided in this application.

[0071] It should be noted that, in the embodiments of this application, the visual language model fine-tuning methods based on adaptive parameter selection shown in the accompanying drawings are all illustrated using one accompanying drawing from an embodiment of this application as an example. In specific implementation, the visual language model fine-tuning methods based on adaptive parameter selection shown in the accompanying drawings of the above methods can also be implemented in conjunction with any other accompanying drawings illustrated in the above embodiments, which will not be elaborated here.

[0072] The following describes the visual language model fine-tuning device based on adaptive parameter selection provided in this application. The description below can be referred to in correspondence with the visual language model fine-tuning method based on adaptive parameter selection described above.

[0073] Figure 3A schematic diagram of the structure of the visual language model fine-tuning device based on adaptive parameter selection provided in the embodiments of this application is shown below. Figure 3 As shown, it specifically includes:

[0074] The acquisition module 301 is used to acquire a pre-trained target model and optimize the target model based on a target specific task; the target model is a visual language model; the importance assessment module 302 is used to determine the importance of each model parameter relative to the target specific task by using the second moment of each model parameter estimated by the optimizer during the optimization training process; the parameter optimization module 303 is used to determine a plurality of model parameters to be adjusted based on the importance of each model parameter relative to the target specific task, and optimize each of the plurality of model parameters to be adjusted.

[0075] Optionally, the importance assessment module 302 is specifically used to forward calculate the cosine similarity matrix between image features and text features, and calculate the cross-entropy loss value based on the cosine similarity matrix; the importance assessment module 302 is also specifically used to obtain the gradient of each model parameter through backpropagation based on the cross-entropy loss value, and update the second moment estimate of each model parameter based on the gradient of each model parameter; the importance assessment module 302 is also specifically used to calculate the bias-corrected second moment based on the second moment estimate of each model parameter, and generate an importance score for each model parameter based on the second moment of each model parameter.

[0076] Optionally, the parameter optimization module 303 is specifically used to group the intra-layer model parameters of the Transformer layer in the target model according to type, and determine the preset number of model parameters with the highest importance score in each group as the model parameters to be adjusted.

[0077] Optionally, the parameter optimization module 303 is specifically used to adaptively train the target model using target data related to the target task, and during the adaptive training process, to optimize and adjust the multiple model parameters to be adjusted using a target loss function corresponding to the target task.

[0078] The visual language model fine-tuning device based on adaptive parameter selection provided in this application first obtains a pre-trained target model and optimizes the target model based on a target specific task; the target model is a visual language model. Then, during the optimization training process, the importance of each model parameter relative to the target specific task is determined using the second moment estimated by the optimizer. Finally, based on the importance of each model parameter relative to the target specific task, multiple model parameters to be adjusted are determined, and each of these multiple model parameters is optimized. In this way, while maintaining the original architecture, efficient fine-tuning can be performed by adaptively selecting key parameters, greatly improving the efficiency of model parameter optimization.

[0079] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a visual language model fine-tuning method based on adaptive parameter selection. This method includes: first, acquiring a pre-trained target model and optimizing the target model based on a target specific task; the target model is a visual language model; then, during the optimization training process, using the second moment of each model parameter estimated by the optimizer, determining the importance of each model parameter relative to the target specific task; finally, based on the importance of each model parameter relative to the target specific task, determining multiple model parameters to be adjusted, and optimizing each of the multiple model parameters to be adjusted. In this way, while maintaining the original architecture, efficient fine-tuning can be performed by adaptively selecting key parameters, greatly improving the efficiency of model parameter optimization.

[0080] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the visual language model fine-tuning method based on adaptive parameter selection provided by the above methods. This method includes: first, acquiring a pre-trained target model and optimizing the target model based on a target specific task; the target model is a visual language model; then, during the optimization training process, using the second moment of each model parameter estimated by the optimizer, determining the importance of each model parameter relative to the target specific task; finally, based on the importance of each model parameter relative to the target specific task, determining multiple model parameters to be adjusted, and optimizing each of the multiple model parameters to be adjusted. Thus, while maintaining the original architecture, efficient fine-tuning can be performed by adaptively selecting key parameters, greatly improving the efficiency of model parameter optimization.

[0082] Furthermore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the aforementioned methods for fine-tuning visual language models based on adaptive parameter selection. This method includes: first, acquiring a pre-trained target model and optimizing the target model based on a target specific task; the target model is a visual language model; then, during the optimization training process, using the second moment of each model parameter estimated by the optimizer, determining the importance of each model parameter relative to the target specific task; finally, based on the importance of each model parameter relative to the target specific task, determining a plurality of model parameters to be adjusted, and optimizing each of the plurality of model parameters to be adjusted. Thus, while maintaining the original architecture, efficient fine-tuning can be performed by adaptively selecting key parameters, greatly improving the efficiency of model parameter optimization.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An adaptive parameter selection based visual language model fine-tuning method, characterized in that, include: Obtain a pre-trained target model and optimize the target model based on the target specific task; The target model is a visual language model; During the optimization training process, the importance of each model parameter relative to the target specific task is determined by using the second moment of each model parameter estimated by the optimizer. Based on the importance of each model parameter relative to the target task, a number of model parameters that need to be adjusted are identified, and each of the multiple model parameters to be adjusted is optimized and adjusted. During the optimization training process, the importance of each model parameter relative to the target specific task is determined by using the second moment of each model parameter estimated by the optimizer, including: The cosine similarity matrix between image features and text features is calculated forward, and the cross-entropy loss value is calculated based on the cosine similarity matrix. Based on the cross-entropy loss value, the gradient of each model parameter is obtained through backpropagation, and the second moment estimate of each model parameter is updated based on the gradient of each model parameter. The bias-corrected second moment is calculated based on the second moment estimate of each model parameter, and an importance score for each model parameter is generated based on the second moment of each model parameter.

2. The visual language model fine-tuning method based on adaptive parameter selection according to claim 1, characterized in that, The importance score for each model parameter is calculated based on the following formula: ； wherein, is the importance score of the i th model parameter, is the second moment of the i th model parameter after bias correction; is used to prevent the denominator from being zero in the formula.

3. The visual language model fine-tuning method based on adaptive parameter selection according to claim 1 or 2, characterized in that, Based on the importance of each model parameter relative to the target specific task, several model parameters that need to be adjusted are determined, including: The intra-layer model parameters of the Transformer layer in the target model are grouped according to type, and the preset number of model parameters with the highest importance score in each group are determined as the model parameters to be adjusted.

4. The visual language model fine-tuning method based on adaptive parameter selection according to claim 3, characterized in that, The optimization adjustment of each of the plurality of model parameters to be adjusted includes: The target model is adaptively trained using target data related to the target specific task, and during the adaptive training process, the target loss function corresponding to the target specific task is used to optimize and adjust the multiple model parameters to be adjusted.

5. A device for fine-tuning a visual language model based on adaptive parameter selection, characterized in that, The device includes: The acquisition module is used to acquire a pre-trained target model and optimize the target model based on the target specific task; the target model is a visual language model. An importance assessment module is used to determine the importance of each model parameter relative to the target specific task by using the second moment of each model parameter estimated by the optimizer during the optimization training process. The parameter optimization module is used to determine multiple model parameters that need to be adjusted based on the importance of each model parameter relative to the target task, and to optimize and adjust each of the multiple model parameters to be adjusted. The importance assessment module is specifically used to forward calculate the cosine similarity matrix between image features and text features, and to calculate the cross-entropy loss value based on the cosine similarity matrix. The importance assessment module is further configured to obtain the gradient of each model parameter through backpropagation based on the cross-entropy loss value, and update the second moment estimate of each model parameter based on the gradient of each model parameter. The importance assessment module is further used to calculate the bias-corrected second moment based on the second moment estimate of each model parameter, and to generate an importance score for each model parameter based on the second moment of each model parameter.

6. The visual language model fine-tuning device based on adaptive parameter selection according to claim 5, characterized in that, The parameter optimization module is specifically used to group the intra-layer model parameters of the Transformer layer in the target model according to type, and determine the preset number of model parameters with the highest importance score in each group as the model parameters to be adjusted.

7. An electronic device, comprising: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the visual language model fine-tuning method based on adaptive parameter selection as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the visual language model fine-tuning method based on adaptive parameter selection as described in any one of claims 1 to 4.

Images