Adapter searching method and device based on pre-training large model

Abstract

The invention provides an adapter searching method based on a pre-training large model. The method comprises the steps that the pre-training large model is selected according to a target data set; constructing a search space for the depth, width and channel combination of the adapter; based on the search space, constructing a network containing all learnable low-rank matrixes as a super network, and performing sampling pre-training on the super network; and searching a final adapter matched with the target data set based on the pre-training weight of the super network by using an evolutionary algorithm. According to the method, a hierarchical adapter search space is designed, so that the adapter has good generalization ability in various downstream tasks, and a super-network sampling training mechanism based on exploration and utilization is adopted to balance training of different channels, so that fair comparison between the channels is ensured, a higher convergence speed is realized, and the method is suitable for being applied to various downstream tasks. The channel convergence is improved, the exploration and utilization of the channel are better balanced, and the adapter structure capable of adapting to different downstream tasks is successfully searched.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular, relates to a search method and device based on a pre-trained large model efficient adapter. Background Art

[0002] In recent years, the development of large models has grown exponentially, and Transformer-based visual models have achieved excellent performance in various fields such as image classification, semantic segmentation, and content generation. However, since the training resources and data of downstream tasks are usually very limited, large models with massive parameters are difficult to fine-tune directly on downstream tasks. In this case, parameter-efficient adapters have become an effective solution. Many methods adopt the paradigm of parameter-efficient fine-tuning (PEFT). PEFT keeps the pre-trained weights of the large model unchanged and only fine-tunes an efficient adapter, so that the model can quickly adapt to downstream tasks. Due to its high efficiency, PEFT has been widely used in various fields.

[0003] Existing hand-designed visual adapters usually use a parallel or serial approach to add learnable embedding vectors to enhance the input, such as VPT, etc.; or learn low-rank matrices to perform linear transformations on pre-trained weights, such as LoRA, etc. However, these hand-designed adapters not only require rich expert experience, but also lack versatility and require time-consuming and laborious adjustment of hyperparameters. In addition, some adapter structures such as VPT and AdaptFormer cannot be integrated into large models after fine-tuning, which introduces additional reasoning overhead, further exacerbating the problem of low reasoning efficiency of large models. Since hand-designed adapters cannot adapt to diverse downstream tasks, researchers began to use Neural Architecture Search (NAS) to automatically design adapters.

[0004] In the field of natural language processing, AdaLoRA and DyLoRA proposed a method to use NAS to search for the optimal rank of low-rank matrices. The former uses a gradient search method based on weight sharing, while the latter uses a heuristic search method based on weight sharing. In the field of computer vision, NOAH and GLoRA summarize the existing manually designed adapters and propose a unified search space. NOAH uses VPT, LoRA and Adapter to construct a joint search space to find the optimal width and position of each adapter. GLoRA constructs a search space for adapters based on LoRA, SSF and RepAdapter to search for the optimal structure and width of each adapter.

[0005] By investigating existing adapter work, the inventors found that existing adapter search methods simply use NAS technology based on manually designed adapter search hyperparameters, without systematically designing the adapter search space or solving the channel training offset problem during the adapter search process. In addition, the adapters searched by methods such as NOAH still bring additional inference costs. Summary of the invention

[0006] In view of the above problems, the present invention proposes an adapter search method based on a pre-trained large model, comprising: a model selection step, selecting a pre-trained large model according to a target data set; a search space construction step, constructing a search space for adapter depth, width and channel combination; a supernet training step, based on the search space, constructing a network containing all learnable low-rank matrices as a supernet, and sampling and pre-training the supernet; an adapter search step, using an evolutionary algorithm to search for a final adapter that matches the target data set based on the pre-trained weights of the supernet.

[0007] Furthermore, the search space construction step specifically includes: a weight modeling step, in which the pre-trained weights of each adapter are modeled with a low-rank matrix; a depth setting step, in which the search depth is set, and within the set depth, each layer selects a learnable low-rank matrix or performs a jump connection with the Identity() function; a width setting step, in which the search width is set, and within the set width, a rank set is formed in ascending order to search for a suitable rank r i As the pre-trained weights W of the i-th adapter i width; channel combination setting step, when the pre-trained weight W of the i-th adapter i Choose the rank size r i When the selected rank size r i , modeling the change ΔW of the pre-trained weights, Among them, A i and B i is a low-rank matrix, represents r obtained by sampling without replacement from all candidate channels i channels.

[0008] Furthermore, the supernet training step includes: using the sampling probability p i Perform channel sampling, in

[0009]

[0010]

[0011] T represents the total number of sampling training times, t j represents the sampling times of the jth channel, represents the gradient norm of the i-th adapter weight and the j-th channel loss, r n represents the width of the exploration sampling, and C represents all candidate channels of the adapter.

[0012] Furthermore, the adapter search step includes: an initial search step, randomly generating adapters that meet the parameter constraints to form an initial adapter population; an iterative search step, sorting the adapter weights of the previous round of adapter population by accuracy, selecting a preset proportion of high-accuracy adapters for crossover and mutation to generate adapters for this round, forming the adapter population for this round; performing a specified number of iterative searches; an adapter selection step, forming a Pareto curve of all searched adapters with accuracy and parameter quantity, and taking the adapter corresponding to the optimal balance point of the Pareto curve as the final adapter.

[0013] The present invention also proposes an adapter search device based on a pre-trained large model, comprising: a pre-trained large model selection module, used to select a pre-trained large model according to a target data set; a hierarchical search space construction module, used to construct a search space for adapter depth, width and channel combination; a supernet sampling training module, used to construct a network containing all learnable low-rank matrices as a supernet based on the search space, and perform sampling pre-training on the supernet; an adapter search module, used to use an evolutionary algorithm to search for a final adapter that matches the target data set based on the pre-trained weights of the supernet.

[0014] Furthermore, the search space construction module includes: a weight modeling module, which is used to model the pre-trained weights of each adapter with a low-rank matrix; a depth setting module, which is used to set the search depth, and within the set depth, each layer selects a learnable low-rank matrix or performs a jump connection with the Identity() function; a width setting module, which is used to set the search width, and within the set width, a rank set is formed in ascending order to search for a suitable rank r i As the pre-trained weights W of the i-th adapter i The width of the channel combination setting module is used as the pre-trained weight W of the i-th adapter i Choose the rank size r i When the selected rank size r i , modeling the change ΔW of the pre-trained weights, Among them, A i and B i is a low-rank matrix, represents r obtained by sampling without replacement from all candidate channels i channels.

[0015] Furthermore, the supernet training module includes: using sampling probability pi to perform channel sampling, in

[0016]

[0017]

[0018] T represents the total number of sampling training times, t j represents the sampling times of the jth channel, represents the gradient norm of the i-th adapter weight and the j-th channel loss, r n represents the width of the exploration sampling, and C represents all candidate channels of the adapter.

[0019] Furthermore, the adapter search module includes: an initial search module, which is used to randomly generate adapters that meet the parameter constraints to form an initial adapter population; an iterative search module, which is used to sort the adapter weights of the previous round of adapter population by accuracy, select a preset proportion of high-accuracy adapters for crossover and mutation to generate adapters for this round, and form the adapter population for this round; call the iterative search module for a specified round to complete the iterative search; and an adapter selection module, which is used to form a Pareto curve with accuracy and parameter quantity for all searched adapters, and take the adapter corresponding to the optimal balance point of the Pareto curve as the final adapter.

[0020] The present invention also proposes an electronic device, comprising the adapter search device based on the pre-trained large model as described above.

[0021] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the adapter search method based on the pre-trained large model as described above is implemented.

[0022] The present invention designs an adapter search space with three granularities of adapter depth, width and channel combination, so that the adapter has good generalization ability in various downstream tasks, and a supernet sampling training mechanism based on exploration and utilization to balance the training of different channels, thereby ensuring fair comparison between channels and achieving faster convergence speed, so that channels that have not been fully trained but have potential can be fully trained, thereby enhancing the fairness of training between different channels and improving channel convergence, thereby better balancing the exploration and utilization of channels and successfully searching for adapter structures that can adapt to different downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the adapter search method based on the pre-trained large model of the present invention.

[0024] Figure 2 It is a schematic diagram of an adapter search device based on a pre-trained large model of the present invention.

[0025] Figure 3 It is a schematic diagram of an electronic device of the present invention.

[0026] Figure 4 It is a schematic diagram of the hardware structure of an electronic device of the present invention.

[0027] Wherein, the accompanying drawings are marked as follows:

[0028] 100: Electronic devices

[0029] 20: Adapter search device based on pre-trained large model

[0030] 21: Pre-trained large model selection module

[0031] 22: Data reading and preprocessing module

[0032] 23: Hierarchical search space building blocks

[0033] 24: Supernet sampling training module based on exploration and utilization

[0034] 25: Adapter search module based on evolutionary algorithm

[0035] S11, S12, S121, S122, S123, S13, S14: Steps DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation method described herein is only used to explain the present invention and is not used to limit the present invention.

[0037] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0038] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0039] In view of the problems existing in the existing adapter search methods, the inventors propose to design a hierarchical search space from a system perspective without introducing additional reasoning overhead, and then find an efficient adapter structure. The primary design principle is to ensure that the adapter structure obtained by the search can be seamlessly integrated into the large model, so as not to introduce additional reasoning overhead. This is crucial to maintaining the efficiency and practicality of the adapter. Secondly, the method systematically designs the adapter search space from three granularities: adapter depth, width, and channel combination, so that the adapter has good generalization ability in various downstream tasks. For example, by exploring different adapter depths, the method can adapt to the features of different levels of the model; the width determines the ability of the adapter to capture task-specific information, and the appropriate width has better adaptability to downstream tasks; the channel combination makes it possible to further explore the effectiveness of different channels at the same width, thereby improving the migration performance of the large model. In addition, in order to reduce the offset of supernet training to different channels, the method also proposes a supernet sampling training mechanism based on exploration and utilization to balance the training of different channels, thereby ensuring fair comparison between channels and achieving faster convergence. This sampling training mechanism utilizes both the channel sampling frequency and the gradient norm, so that channels that have not been fully trained but have potential can be fully trained, thereby enhancing the training fairness between different channels and improving channel convergence, thus better balancing the exploration and utilization of channels and successfully searching for adapter structures that can adapt to different downstream tasks.

[0040] The goal of this invention is to solve the problem that efficient adapters for pre-trained large models are difficult to design manually and have additional inference overhead. Figure 1 : is a flow chart of the adapter search method based on the pre-trained large model of the present invention. Figure 1 As shown, in the first embodiment of the present invention, a search method for efficient adapters based on pre-trained large models is proposed, which mainly includes: constructing a hierarchical search space including adapter depth, width and channel combination; using a supernet sampling training mechanism based on exploration and utilization; using an evolutionary algorithm to search for adapters based on the pre-trained weights of the supernet. The above steps of the present invention are described in detail below.

[0041] Step S11: Select a suitable pre-trained large model according to the target data set

[0042] Select a suitable pre-trained large model according to the target data set to achieve rapid migration of downstream tasks. For example, when performing natural language processing tasks, use the RoBERTa model pre-trained on multiple text data sets to quickly migrate to the sentiment analysis SST2 and logical classification MNLI data sets; when performing image classification tasks, use the ViT-B / 16 model pre-trained on ImageNet to migrate to the Food-101 data set containing 101 different food categories, etc. The present invention is not limited to this.

[0043] Step S12: Constructing a hierarchical search space of adapter depth, width, and channel combinations

[0044] Given an input x and the pre-trained weights W of a large model (0) , LoRA proposes to use low-rank matrices A and B to pre-train weights W (0) The weight variable ΔW is modeled to obtain the forward feature y = xW 0 +xΔW=xW 0 +xBA; that is: ΔW=BA

[0045] Step S121: Depth Search

[0046] The present invention is for each pre-training weight W (0) We design parallel and serial low-rank matrices [W0,W1,...,W m ]. The parallel low-rank matrix is ​​used as a mask to extract W (0) , and the serial low-rank matrix helps improve the learning of downstream task features. In order to achieve deep searchability, each layer can choose a learnable low-rank matrix or a skip connection (Identity(), which directly outputs the input without any operation, which is equivalent to skipping the layer). For example, taking an m-layer searchable adapter as an example, when the search result of the 0th layer is a skip connection, the search result of the 1st layer is a low-rank matrix W1, ..., and the search result of the m-1th layer is W m-1 When the forward feature y m It is expressed as follows:

[0047]

[0048] This adapter structure design that combines parallel and serial can simultaneously capture changes in the weight space ΔW and the feature space ΔY, allowing large models to better learn the characteristics of downstream tasks.

[0049] Step S122: Width search

[0050] In terms of matrix width, this work focuses on searching for a suitable rank size for trainable low-rank matrices. Given n searchable ranks, arrange them in ascending order to form a rank set R = [r0, r1, ..., rn ], search for a suitable rank r i As the i-th adapter weight W i Since the changes in pre-trained weights at different layers and positions in a pre-trained large model vary, searching for a suitable rank for each pre-trained weight can better adapt to downstream tasks.

[0051] Step S123: Channel combination search

[0052] When the i-th adapter weight W i Choose the rank size r i When the previous research work usually defaults to the former r i channels to construct adapter weights. This approach not only limits the diversity of the search space, but also affects the migration performance. To solve this problem, the present invention proposes to select a rank size r i , different channel combinations can be freely selected from the candidate channels. Specifically, for the i-th trainable low-rank matrix A i and B i , given its rank size is r i When , the present invention proposes to sample channels from all optional channels, so as to model the change in weight W i , which can be expressed by the following formula:

[0053] ΔW=B i [:,C ri ]A i [:,C ri ]

[0054] C ri represents r obtained by sampling without replacement from all candidate channels i It can be seen that this adapter search method eliminates the constraint of channel combinations, so it can build a more diverse search space, thereby improving the model capacity of the adapter when adapting to downstream tasks.

[0055] Step S13: Using a supernet sampling training mechanism based on exploration and exploitation

[0056] Based on the above search space, a network is constructed that contains all learnable low-rank matrices. This network is called a supernet. In supernet training, the present invention proposes to first randomly sample a depth d from the deep search space. i Then, based on the selected depth d i Randomly sample a width (i.e., matrix rank) r for each layer from the width search space i Finally, for each sample width r i , randomly sample a channel combination C from the optional channel combinations ri .

[0057] Given the width r of the sample i , existing work usually samples the first r in each training iteration i channels, namely C ri =[0,1,...,r i-1 ], resulting in the training times of the preceding channel being more than that of the succeeding channel. This unbalanced training strategy makes it difficult to fairly compare the performance of different channels. Therefore, based on the channel combination search space proposed above, the present invention further proposes a supernet sampling training mechanism based on exploration and utilization to balance the sampling frequencies of different channels. In order to enable the early training to fully explore the potential of different channels, it is proposed to sample according to the following channel sampling probabilities:

[0058]

[0059] Where T represents the total number of sampling training times, t j Indicates the number of sampling times for each channel. This method can make the channels with higher training frequency in the early stage of training be sampled less, and the channels with lower training frequency be sampled more, so as to maintain fair training for all channels as much as possible.

[0060] As the training progresses, due to p i exploration The growth rate of the numerator logarithmic function is lower than the growth rate of the denominator linear function, so the influence of the exploration part on the sampling will gradually weaken. Therefore, the present invention further proposes training dynamics based on different channels, sampling according to the gradient norm of each channel, so that channels with higher gradient norms can get more attention and training, and this sampling method is called sampling training based on utilization:

[0061]

[0062] in represents the gradient 2-norm of the i-th adapter weight and the j-th channel loss.

[0063] Inspired by the Upper Confidence Bound (UCB) algorithm that balances exploration and exploitation, the present invention proposes to further add the two sampling probabilities to obtain the final sampling distribution p. For the i-th adapter weight, according to the sampling distribution p i Sampling channel combination

[0064]

[0065] In the initial training phase, the combined sampling distribution p i Will be more inclined to explore some p iexploration , to ensure fair sampling across channels. As training progresses, due to p i exploration The difference in the growth rates of the numerator and denominator makes their values ​​average, p i Will naturally turn to p i exploitation , focusing more on the convergence status of different channels. This method does not require manual adjustment of hyperparameters, and enables all channels to be trained fairly and fully during the training process, thereby improving the generalization of the supernet.

[0066] Step S14: Searching for an adapter using an evolutionary algorithm based on the pre-trained weights of the supernet

[0067] The present invention is based on a weight sharing strategy, and performs efficient adapter evaluation by inheriting the pre-trained supernet weights. Specifically, an evolutionary algorithm is used to search for the optimal adapter structure. In the first search iteration, adapters that meet the parameter constraints are randomly generated to form an initial population. Then, these adapters are evaluated by inheriting the pre-trained supernet weights and sorted according to the accuracy of the adapters. The top-ranked adapters will undergo crossover and mutation to generate the next generation of adapter structures. By iterating the above search, the algorithm will automatically search for diverse and efficient adapter structures, and find the optimal balance point on a Pareto curve that balances accuracy and parameter quantity as the searched adapter structure.

[0068] It should be noted that, in various embodiments of the present invention, the sequence numbers of the above steps do not mean the order of execution. The order of execution of the steps should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0069] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0070] Figure 2 Schematic diagram of the adapter search device based on the pre-trained large model of the present invention. Figure 2 As shown, in the second embodiment of the present invention, an adapter search device 20 based on a pre-trained large model is provided, and the device includes: a pre-trained large model selection module 21, a data reading and pre-processing module 22, a hierarchical search space construction module 23, a supernet sampling training module 24 based on exploration and utilization, and an adapter search module 25 based on an evolutionary algorithm:

[0071] A. Pre-trained large model selection module 21: This module selects a suitable pre-trained large model according to the target data set, thereby achieving rapid migration of downstream tasks;

[0072] B. Data reading and preprocessing module 22: This module divides the training data into a training set and a validation set according to the corresponding hyperparameter settings, which are used for supernet training and adapter evaluation respectively;

[0073] C. Hierarchical search space construction module 23: This module constructs a supernet from three granularities of depth, width and channel combination according to the hierarchical search space proposed in the present invention; in terms of depth, it searches for the number of layers of learnable low-rank matrices; in terms of width, it searches for the rank size of learnable low-rank matrices; in terms of channel combination, it samples channels from all candidate channels based on the selected rank size to form a low-rank matrix.

[0074] D. Supernet sampling training module 24 based on exploration and utilization: This module samples efficient adapter structures for training according to the sampling training mechanism based on exploration and utilization proposed in the present invention, extracts the sampled adapter weights from the supernet for forward and backward propagation and weight update, thereby achieving the purpose of training the supernet; in the early stage of training, sampling is performed based on the number of training times of different channels, so that channels with higher training frequencies are sampled less and channels with lower training frequencies are sampled more, thereby maintaining fair training for all channels as much as possible; in the later stage of training, sampling is performed based on the normalized gradient norms of different channels, thereby focusing more on the convergence states of different channels. This method does not require manual adjustment of hyperparameters, and enables all channels to be trained fairly and fully during the training process, thereby improving the generalization of the supernet.

[0075] E. Adapter search module based on evolutionary algorithm 25: This module uses evolutionary algorithm to search for adapter structure, and inherits pre-trained supernet weights to quickly evaluate the sampled adapters, and finds adapter structures that meet computational constraints and have excellent performance through iterative search.

[0076] In the search device based on the pre-trained large model efficient adapter provided by the embodiment of the present invention, the working process of each module has the same technical characteristics as the search method based on the pre-trained large model efficient adapter. Therefore, the aforementioned functions can also be achieved and will not be repeated here.

[0077] In the third embodiment of the present invention, a computer-readable storage medium is proposed. The task scheduling device for edge-side video processing of the present invention, if its function is implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. Therefore, in the third embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program for an adapter search method based on a pre-trained large model. It should be understood that the computer-readable storage medium in the embodiment of the present invention may be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct rambus RAM (DRRAM).

[0078] Figure 3 Schematic diagram of an electronic device of the present invention. Figure 3As shown, in the fourth embodiment of the present invention, an electronic device 100 is proposed, including an adapter search device 20 based on a pre-trained large model as described above. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software function module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiments of the present invention are not limited to any specific form of combination of hardware and software.

[0079] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0080] The electronic device of the present invention may be any device with data processing capability, and the device with data processing capability may be a device or apparatus such as a computer. The device embodiment may be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of any device with data processing capability reads the corresponding computer program instructions in the non-volatile memory into the memory and runs the instructions. Figure 4 This is a schematic diagram of the hardware structure of an electronic device of the present invention. Figure 4 As shown, from the hardware level, it is a hardware structure diagram of any device with data processing capability where the adapter search device based on the pre-trained large model of the present invention is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0081] When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0082] The following is a specific embodiment of the present invention:

[0083] (1) Migration model: The ViT-B model pre-trained on the ImageNet dataset is migrated to the CIFAR-100 dataset;

[0084] (2) Migration settings: Set a searchable adapter for each linear layer in the ViT-B model. The depth of the adapter is searched from D = {1, 2, 3}. After the depth is selected, the matrix rank r of each layer of the adapter is searched from R = {2, 4, 8}. After the matrix rank r is selected, r channels are searched from all optional channels. Freeze the ViT-B model parameters and only fine-tune the adapter parameters;

[0085] (3) Migration effect: Using this patented method to migrate the ViT-B model to CIFAR-100 achieves a classification accuracy of 94.09%, which is 4.99% higher than directly training the ViT-B model on CIFAR-100 (89.1%).

[0086] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. An adapter search method based on a pre-trained large model, characterized in that: include: Model selection step: select a pre-trained large model based on the target data set; A search space construction step, constructing a search space for adapter depth, width, and channel combinations; A supernet training step is to construct a network containing all learnable low-rank matrices as a supernet based on the search space, and perform sampling pre-training on the supernet; In the adapter search step, an evolutionary algorithm is used to search for the final adapter that matches the target dataset based on the pre-trained weights of the supernet.

2. The adapter search method according to claim 1, wherein: The search space construction steps specifically include: The weight modeling step models the pre-trained weights of each adapter with a low-rank matrix; Depth setting step, setting the search depth, and within the set depth, each layer selects a learnable low-rank matrix or performs a jump connection with the Identity() function; Width setting step: set the search width, and within the set width, arrange the rank set in ascending order to search for the appropriate rank r i As the pre-trained weights W of the i-th adapter i Width; In the channel combination setting step, when the pre-trained weight W of the i-th adapter i Choose the rank size r i When the selected rank size r i , modeling the change ΔW of the pre-trained weights, Among them, A i and B i is a low-rank matrix, represents r obtained by sampling without replacement from all candidate channels i channels.

3. The adapter search method according to claim 2, wherein: The supernet training steps include: Using the sampling probability p i Perform channel sampling, in T represents the total number of sampling training times, t j represents the sampling times of the jth channel, represents the gradient norm of the i-th adapter weight and the j-th channel loss, r n represents the width of the exploration sampling, and C represents all candidate channels of the adapter.

4. The adapter search method according to claim 2, wherein: The adapter search steps include: In the initial search step, adapters that meet the parameter constraints are randomly generated to form the initial adapter population; Iterative search step, sorting the adapter weights of the previous round of adapter population by accuracy, selecting a preset proportion of high-accuracy adapters for crossover and mutation to generate adapters for this round, forming the adapter population for this round; performing a specified round of iterative search; In the adapter selection step, all the searched adapters are combined into a Pareto curve based on accuracy and parameter quantity, and the adapter corresponding to the optimal balance point of the Pareto curve is the final adapter.

5. An adapter search device based on a pre-trained large model, characterized in that: include: A pre-trained large model selection module is used to select a pre-trained large model according to the target data set; Hierarchical search space building module, used to build the search space for adapter depth, width and channel combinations; A supernet sampling training module is used to construct a network containing all learnable low-rank matrices as a supernet based on the search space, and perform sampling pre-training on the supernet; The adapter search module is used to search for a final adapter matching the target data set based on the pre-trained weights of the supernet using an evolutionary algorithm.

6. The adapter search device according to claim 5, characterized in that: The search space building blocks include: A weight modeling module for modeling the pre-trained weights of each adapter as a low-rank matrix; The depth setting module is used to set the search depth, and within the set depth, each layer selects a learnable low-rank matrix or performs a jump connection with the Identity() function; The width setting module is used to set the search width, and within the set width, the rank set is arranged in ascending order to search for the appropriate rank r i As the pre-trained weights W of the i-th adapter i Width; The channel combination setting module is used as the pre-trained weight W of the i-th adapter i Choose the rank size r i When the selected rank size r i , modeling the change ΔW of the pre-trained weights, Among them, A i and B i is a low-rank matrix, represents r obtained by sampling without replacement from all candidate channels i channels.

7. The adapter search device according to claim 6, characterized in that: The supernet training module includes: Using the sampling probability p i Perform channel sampling, in T represents the total number of sampling training times, t j represents the sampling times of the jth channel, represents the gradient norm of the i-th adapter weight and the j-th channel loss, r n represents the width of the exploration sampling, and C represents all candidate channels of the adapter.

8. The adapter search device according to claim 6, characterized in that: The adapter search module includes: The initial search module is used to randomly generate adapters that meet the parameter constraints to form the initial adapter population; An iterative search module is used to sort the adapter weights of the previous round of adapter population by accuracy, select a preset proportion of high-accuracy adapters for crossover and mutation to generate adapters for this round and constitute the adapter population for this round; the iterative search module is called for a specified round to complete the iterative search; The adapter selection module is used to form a Pareto curve with accuracy and parameter quantity for all the searched adapters, and the adapter corresponding to the optimal balance point of the Pareto curve is the final adapter.

9. An electronic device comprising a search device based on a pre-trained large model adapter as described in any one of claims 5 to 8.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, the adapter search method based on the pre-trained large model as described in any one of claims 1 to 4 is implemented.

Images