A method, system, device and medium for reprogramming a pre-trained visual large model

Abstract

The present invention discloses a method, system, device and medium for reprogramming a pre-trained visual large model, and belongs to the technical field of visual large models. The method comprises: freezing the teacher extractor of the base model; reprogramming the teacher classifier of the base model using the classifier of the target model that directly matches the downstream task to align the base model with the task; introducing a projector to project the knowledge in the base model to the downstream task. Through the process of reprogramming and distillation, the present invention can transfer the general and high-level features and knowledge learned by the pre-trained model to the small model, thereby improving the generalization ability and adaptability of the small model; it can greatly reduce the cost of large model migration without fine-tuning the pre-trained large model.

Description

Technical Field

[0001] The present invention relates to the field of large visual model technology, and in particular to a method, system, device and medium for reprogramming a pre-trained large visual model. Background Art

[0002] Large visual models are deep neural network models pre-trained on large-scale image datasets. They can extract high-level semantic features of images, thereby achieving excellent performance on a variety of visual tasks. However, large visual models also face challenges and limitations, such as insufficient data, insufficient computing resources, task mismatch, and domain inconsistency. These factors affect the generalization ability and adaptability of large visual models in practical applications. To address these issues, fine-tuning large visual models for downstream tasks is an effective method. It can use pre-trained large visual models as initial parameters or feature extractors, and perform fine-tuning or transfer learning for specific downstream tasks to improve the performance and efficiency of the model.

[0003] Transferring and fine-tuning large visual models for downstream tasks can leverage the knowledge and capabilities of pre-trained large visual models to improve the performance and efficiency of downstream tasks. However, pre-trained large visual models typically have a high number of parameters and computational complexity, while downstream tasks often need to be deployed on different hardware platforms, such as the cloud, edge, and end devices, resulting in slow model inference or inability to deploy. This results in large visual models facing high computational costs and difficulty in popularization in practical applications. Therefore, there is an urgent need to propose an efficient fine-tuning algorithm based on the transfer of large visual models for downstream tasks, which can transfer the rich knowledge of large visual models to lightweight models and reduce the cost and time of transfer learning. Summary of the Invention

[0004] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a method, system, device and medium for reprogramming a pre-trained visual large model.

[0005] The technical solution adopted in the present invention is:

[0006] A method for reprogramming a large pre-trained visual model, comprising the following steps:

[0007] Freeze the teacher extractor of the base model (i.e., the pre-trained large visual model);

[0008] Reprogram the teacher classifier of the base model using the classifier of the target model that is directly matched to the downstream task to align the base model to the task;

[0009] A projector is introduced to project the knowledge in the base model into downstream tasks.

[0010] Furthermore, the teacher classifier of the basic model includes two design methods, which are respectively applicable to the CNN type basic model and the Transformer type basic model.

[0011] Furthermore, for the CNN type base model, a linear layer is used as the teacher classifier to map the last layer features of the base model into the proxy space:

[0012] Assume that the last layer of features of the basic model is Where C is the number of channels, H and W are the height and width; the parameters of the teacher classifier are Where K is the dimension of the agent space, then the output of the teacher classifier is:

[0013]

[0014] where f T is to flatten the feature f into a vector.

[0015] Furthermore, for the Transformer-type base model, a multi-head self-attention mechanism is used as the teacher classifier to map the last layer features of the base model into the proxy space:

[0016] Assume that the last layer of features of the basic model is Where N is the sequence length and C is the number of channels; the parameters of the teacher classifier are three sets of linear transformation matrices: query matrix Bond Matrix Sum Matrix Where D is the dimension of each head; then the output of the teacher classifier is:

[0017]

[0018] The softmax function is normalized along the last dimension.

[0019] Furthermore, the introduction of the projector to project the knowledge in the basic model into downstream tasks includes:

[0020] Add an adaptation layer between the teacher extractor and the teacher classifier to adjust the feature dimension and enhance feature expression:

[0021] Assume that the output feature of the teacher extractor is Where N is the sequence length or spatial size, C is the number of channels or hidden dimensions; the parameters of the adaptation layer are a linear transformation matrix Where D is the dimension of the proxy space; then the output of the adaptation layer is:

[0022]

[0023] In this way, the adaptation layer can transform the output features of the teacher extractor into a dimension that matches the teacher classifier and retain the information of the original features;

[0024] In addition, a normalization layer is added after the adaptation layer to eliminate the difference in feature distribution, and a residual connection is added to balance the importance of features;

[0025] Finally, the network structure between the teacher extractor and the classifier is:

[0026]

[0027] Among them, LN is the layer normalization operation, W t are the parameters of the teacher classifier and K is the dimension of the surrogate space.

[0028] Furthermore, the projector is implemented using the backbone of the base model.

[0029] Furthermore, for a given downstream data The reprogramming formula is expressed as follows:

[0030]

[0031] in, Indicates that it is based on the target data domain The mean loss, f t () represents the basic model backbone, f p () represents the model reprogramming module, c t () represents the classifier, CE() represents the cross entropy loss function that guides the reprogramming process, x s represents the input data, and y represents the true label.

[0032] Another technical solution adopted in the present invention is:

[0033] A pre-trained large visual model reprogramming system, comprising:

[0034] The extractor freezing module is used to freeze the teacher extractor of the base model;

[0035] A classifier alignment module is used to reprogram the teacher classifier of the base model using the classifier of the target model that directly matches the downstream task to align the base model with the task;

[0036] The knowledge distillation module is used to introduce a projector to project the knowledge in the base model into downstream tasks.

[0037] Another technical solution adopted in the present invention is:

[0038] A pre-trained large visual model reprogramming device, comprising:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0042] Another technical solution adopted in the present invention is:

[0043] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0044] The present invention has the following beneficial effects: Through a process of reprogramming and distillation, it can transfer the common and high-level features and knowledge learned by the pre-trained model to the small model, thereby improving the generalization and adaptability of the small model. This can significantly reduce the cost of migrating the large model, eliminating the need to fine-tune the pre-trained large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flowchart of the steps of a method for reprogramming a pre-trained visual large model for downstream task applications in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0049] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0050] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0051] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0052] Based on the pre-trained large visual model, the parameters of all or part of the layers are fine-tuned to adapt to the data distribution and objective function of the downstream tasks. This method can make full use of the knowledge of the pre-trained model, and can also be flexibly adjusted according to the downstream tasks. However, this method also has a significant disadvantage, that is, it requires a lot of computing resources and time to train and deploy the large visual model. Pre-trained large visual models usually have a high number of parameters and computing power, and downstream tasks often need to be deployed on different hardware platforms, such as the cloud, edge, and end devices, resulting in slow model inference speed or inability to deploy. In order to overcome this problem, the present invention proposes a novel technical solution, namely, the large model is driven by downstream tasks to be reprogrammed. This solution adds a new network module to the large model, and only updates the reprogramming part during the migration process, performing efficient large model migration and distillation, without the need to fine-tune the pre-trained large model, which can greatly reduce the cost of large model migration.

[0053] like Figure 1As shown, this embodiment provides a method for reprogramming a large pre-trained visual model for downstream task applications. The first step is to keep the feature extractor of the base model frozen. The frozen feature extractor maintains powerful and universal feature extraction capabilities without the computational cost of learning. The second step is to construct a proxy reprogramming space to project the knowledge in the base model into the downstream task. The method specifically includes the following steps:

[0054] S1. Freeze the teacher extractor of the base model (i.e., the pre-trained visual large model).

[0055] In this step, the pre-trained visual model f t Instead of making any modifications or updates, it is frozen as a powerful feature extractor. The advantage of this is that f t The general and high-level features and knowledge learned on large-scale datasets without spending additional computing resources and time to train it. t The input image can be converted into a set of meaningful feature vectors, providing a basis for subsequent reprogramming and distillation processes.

[0056] S2. Reprogram the teacher classifier of the base model using the classifier of the target model that is directly matched to the downstream task to align the base model to the task.

[0057] The task prototype is aligned with the classifier used in the downstream target model. This classifier is directly related to the downstream task. It is well-fitted to the target data and contains some data domain information, such as the extracted feature dimensions and classification space. By performing the same classification task as the downstream data in the proxy space, we can reprogram the base model to extract downstream perceptual knowledge, which helps mitigate the adverse effects of task mismatch and domain differences.

[0058] Among them, the design of the teacher classifier structure is flexible, and there are two design methods, which can be applied to CNN and Transformer type basic models respectively.

[0059] For a CNN-type basic model, such as ResNet-101, this embodiment uses a simple linear layer as a teacher classifier to map the last layer features of the basic model to the proxy space. Specifically, let the last layer features of the basic model be Where C is the number of channels, H and W are the height and width. The parameters of the teacher classifier are Where K is the dimension of the agent space. Then the output of the teacher classifier is: where f T is to flatten the feature f into a vector. In this way, the teacher classifier can compress the knowledge of the base model into a low-dimensional proxy space.

[0060] For the basic model of the Transformer type, such as the Swin Transformer, this embodiment uses a multi-head self-attention mechanism as the teacher classifier to map the last layer of features of the basic model to the proxy space. Specifically, let the last layer of features of the basic model be Where N is the sequence length and C is the number of channels. The parameters of the teacher classifier are three sets of linear transformation matrices: query matrix Bond Matrix Sum Matrix Where D is the dimension of each head. Then the output of the teacher classifier is:

[0061]

[0062] The softmax function is normalized along the last dimension. In this way, the teacher classifier can transform the knowledge of the base model into a proxy space with self-attention relationship.

[0063] S3. Introduce a projector to project the knowledge in the base model into downstream tasks.

[0064] There is still a gap between the teacher extractor and the classifier. When we use the target data x s As input, the fixed feature extractor f t It is not completely applicable because the domain gap makes the extracted features inaccurate and invalid. t It is targeted at the source domain and is used to draw on the knowledge of the source domain and its own capabilities. t For the target domain, it is used to make the reprogrammed model more suitable for downstream tasks. We want to keep their advantages while adjusting the domain, so we introduce a projector f p To transform the features from f t Converted to a more refined counterpart. The specific network structure design of the projector is as follows:

[0065] An adaptation layer is added between the teacher extractor and the classifier to adjust the feature dimension and enhance the feature expression. Specifically, the output feature of the teacher extractor is Where N is the sequence length or spatial size, C is the number of channels or hidden dimensions. The parameters of the adaptation layer are a linear transformation matrix Where D is the dimension of the proxy space. Then the output of the adaptation layer is: In this way, the adaptation layer can transform the output features of the teacher extractor into a dimension that matches the teacher classifier, while retaining the information of the original features. In addition, a normalization layer is added after the adaptation layer to eliminate the difference in feature distribution, and a residual connection is added to balance the importance of features. Finally, the network structure between the teacher extractor and the classifier is:

[0066]

[0067] Among them, LN is the layer normalization operation, W t are the parameters of the teacher classifier and K is the dimension of the surrogate space.

[0068] The projector further t The extracted features are converted to the target domain and sent to c t Driven by the front-end and back-end, the projector is forced to break through the data domain differences. In this way, we use downstream tasks to drive model reprogramming, preparing for subsequent knowledge distillation. In line with the principles of simplicity and adaptability, we use the teacher's backbone block as the projector.

[0069] Formally, given downstream data The reprogramming formula can be expressed as follows:

[0070]

[0071] where f p () and c t () are the neural module and classifier we should learn at this stage respectively, usually with some parameters. CE() represents the cross entropy loss function that guides the reprogramming process.

[0072] In summary, the present invention can effectively utilize the knowledge and capabilities of the pre-trained large visual model to improve the performance and efficiency of the small model for downstream tasks. Through the process of reprogramming and distillation, we can transfer the general and high-level features and knowledge learned by the pre-trained model to the small model, thereby improving the generalization ability and adaptability of the small model. The cost of migrating large models can be greatly reduced without fine-tuning the pre-trained large model. By freezing the pre-trained large model as a feature extractor, we can avoid any modification or update to it, thereby saving a lot of computing resources and time. At the same time, by introducing new network modules as projectors and classifiers, we can achieve a lightweight reprogramming space, thereby reducing the complexity and difficulty of transfer learning.

[0073] This embodiment also provides a pre-trained visual large model reprogramming system, including:

[0074] The extractor freezing module is used to freeze the teacher extractor of the base model;

[0075] A classifier alignment module is used to reprogram the teacher classifier of the base model using the classifier of the target model that directly matches the downstream task to align the base model with the task;

[0076] The knowledge distillation module is used to introduce a projector to project the knowledge in the base model into downstream tasks.

[0077] A pre-trained visual large model reprogramming system of this embodiment can execute a pre-trained visual large model reprogramming method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0078] This embodiment also provides a pre-trained visual model reprogramming device, comprising:

[0079] at least one processor;

[0080] at least one memory for storing at least one program;

[0081] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 1 The method shown.

[0082] A pre-trained visual large model reprogramming device of this embodiment can execute a pre-trained visual large model reprogramming method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0083] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0084] This embodiment also provides a storage medium storing instructions or programs that can execute a pre-trained visual large model reprogramming method provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0085] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0086] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0087] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0088] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0089] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0090] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0091] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0092] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0093] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for reprogramming a large pre-trained visual model, applied in the field of image processing, characterized in that: The following steps are involved: Freeze the teacher extractor of the base model; Reprogram the teacher classifier of the base model using the classifier of the target model that directly matches the downstream task, To align the base model to the task; Introducing a projector to project knowledge from the base model into downstream tasks; The teacher classifier of the basic model is implemented using a basic model of the Transformer type; For the Transformer-type base model, a multi-head self-attention mechanism is used as the teacher classifier to map the last layer of features of the base model into the proxy space: Assume that the last layer of features of the basic model is Where N is the sequence length and C is the number of channels; the parameters of the teacher classifier are three sets of linear transformation matrices: query matrix Bond Matrix Sum Matrix Where D is the dimension of each head; then the output of the teacher classifier is: The softmax function is normalized along the last dimension; The introduction of the projector projects the knowledge in the base model into downstream tasks, including: An adaptation layer is added between the teacher extractor and the teacher classifier to adjust the feature dimension and enhance the feature expression: the output feature of the teacher extractor is Where N is the sequence length or spatial size, C is the number of channels or hidden dimensions; the parameters of the adaptation layer are a linear transformation matrix Where D is the dimension of the proxy space; then the output of the adaptation layer is: in a =fW a In this way, the adaptation layer can transform the output features of the teacher extractor into a dimension that matches the teacher classifier and retain the information of the original features; In addition, a normalization layer is added after the adaptation layer to eliminate the difference in feature distribution, and a residual connection is added to balance the importance of features; Finally, the network structure between the teacher extractor and the classifier is: y t =T(f)=W t (LN(f a +f)) Among them, LN is the layer normalization operation, W t are the parameters of the teacher classifier, K is the dimension of the agent space; For a given downstream data The reprogramming formula is expressed as follows: in, Indicates that it is based on the target data domain The mean loss, f t () represents the basic model backbone, f p () represents the model reprogramming module, c t () represents the classifier, CE() represents the cross entropy loss function that guides the reprogramming process, x s represents the input data, and y represents the true label.

2. A pre-trained visual large model reprogramming method according to claim 1, characterized in that: The teacher classifier of the basic model can also be implemented using a CNN type basic model; For the CNN type base model, a linear layer is used as the teacher classifier to map the last layer features of the base model into the proxy space: Assume that the last layer of features of the basic model is Where C is the number of channels, H and W are the height and width; the parameters of the teacher classifier are Where K is the dimension of the agent space, then the output of the teacher classifier is: y t =W t f T where f T is to flatten the feature f into a vector.

3. A pre-trained visual large model reprogramming method according to claim 1, characterized in that: The projector is implemented using the backbone of the base model.

4. A pre-trained visual large model reprogramming system, applied to a pre-trained visual large model reprogramming method according to any one of claims 1 to 3, characterized in that: include: The extractor freezing module is used to freeze the teacher extractor of the base model; A classifier alignment module is used to reprogram the teacher classifier of the base model using the classifier of the target model that directly matches the downstream task to align the base model with the task; The knowledge distillation module is used to introduce a projector to project the knowledge in the base model into downstream tasks.

5. A pre-trained visual large model reprogramming device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 3 when executed by the processor.

Images