An adaptive fine-tuning method and system based on contribution measurement

Abstract

The present invention provides an adaptive fine-tuning method and system based on contribution measurement, and relates to the technical field of parameter fine-tuning; the present invention uses particle swarm optimization PSO to accurately evaluate the parameter fine-tuning ratio in each layer of a pre-trained model, obtains a fine-tuning ratio suitable for a target task, measures the contribution of parameters in each convolutional layer of the pre-trained model, provides a basis for selecting fine-tuning parameters, calculates a mask matrix through parameter contribution and an optimal fine-tuning ratio, fine-tunes the parameters in the model on target training data through the mask matrix, and obtains a fine-tuned training model. The present invention realizes parameter-level fine-tuning, so that the model reaches a balance between mismatching target tasks and overfitting target tasks, and significantly improves the performance of the model on target tasks through a small amount of training time.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of parameter fine-tuning, and in particular to an adaptive fine-tuning method and system based on contribution measurement. Background Art

[0002] Transfer learning is an important machine learning technique that improves the performance of a model by applying knowledge learned from an existing task (i.e., source domain) to another related task (i.e., target domain). Extracting knowledge from an existing task means training a model with data and labels from the source domain and extracting knowledge from the model, such as model parameters or feature representations. Fine-tuning is a popular transfer learning technique that starts with pre-trained model parameters in a source domain with a large amount of training data and fine-tunes them on the target task to improve the learning performance of the model. During the fine-tuning process, parameters that are closely related to the target task are useful knowledge and should be reused in the target task, while parameters that are not related to the target task should be updated. In traditional fine-tuning methods, the selection of fine-tuning parameters is usually done manually based on expert experience.

[0003] Compared with traditional fine-tuning methods, fine-tuning solutions suitable for target tasks are solved through iterative optimization. Optimization-based fine-tuning methods are divided into two categories: fine-tuning methods based on policy network optimization and fine-tuning methods based on evolutionary optimization. One is a method that automatically selects fine-tuning parameters, making a binary decision on whether to fine-tune a single layer or filter in the pre-trained model. After multiple iterations, a fine-tuning solution suitable for the target task is obtained. However, this method requires training a large number of parameters. When gradient descent is directly applied, the discreteness and non-differentiability of the fine-tuning parameter selection problem bring challenges, thus affecting the accuracy of the fine-tuning solution. The other is a fine-tuning method based on evolutionary optimization, which regards the fine-tuning layer selection problem as a combinatorial optimization problem and searches for layer fine-tuning solutions suitable for the target task through population evolution. This method regards each fine-tuning solution as a single individual in the population, and iteratively performs crossover, mutation, and selection operations through evolutionary algorithms to solve an optimal layer fine-tuning solution. However, when the pre-trained model is large or the fine-tuning unit is a single weight, the solution accuracy of the fine-tuning method based on evolutionary optimization is easily limited by high dimensions. Therefore, it is necessary to accurately estimate the fine-tuning ratio of the parameters, and an adaptive fine-tuning method of parameter contribution is urgently needed. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an adaptive fine-tuning method and system based on contribution measurement in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] An adaptive fine-tuning method based on contribution measurement includes the following steps:

[0007] S1. Importing a pre-trained model, wherein the pre-trained model includes multiple convolutional layers, and each convolutional layer includes multiple parameters;

[0008] S2, fine-tune and evaluate multiple parameters in each layer through particle swarm optimization (PSO) to obtain multiple candidate solutions;

[0009] S3, measuring multiple parameters of each convolution layer by a contribution measurement criterion to obtain contributions corresponding to multiple parameters in each convolution layer, calculating contributions corresponding to multiple parameters of each convolution layer by multiple candidate solutions to obtain multiple initial mask matrices, performing parameter adjustment calculations on the pre-trained model and the multiple initial mask matrices according to a first preset period to obtain multiple initial fine-tuning training models, inputting preset target data sets into the multiple initial fine-tuning training models for calculation to obtain multiple fitness values;

[0010] S4, updating the plurality of candidate solutions by using the plurality of fitness values ​​until a preset number of iterations is met, and obtaining an optimal solution according to the updated plurality of candidate solutions;

[0011] S5. Calculate the contribution corresponding to multiple parameters of each convolutional layer through the optimal solution to obtain a mask matrix, and adjust the parameters of the pre-trained model and the mask matrix according to a second preset period to obtain a fine-tuning training model.

[0012] Among them, steps S2-S4 are the adaptive fine-tuning ratio search strategy of the present invention.

[0013] The beneficial effects of the present invention are: parameter-level fine-tuning is achieved through contribution measurement criteria and adaptive fine-tuning ratio search strategy, so that the model reaches a balance between mismatching target tasks and overfitting target tasks, and the performance of the model on the target task is significantly improved through a small amount of training time.

[0014] Furthermore, the multiple parameters of each convolutional layer are measured by the contribution measurement criterion to obtain the contribution corresponding to the multiple parameters in each convolutional layer, specifically:

[0015] The weights of multiple parameters of each convolutional layer are measured by the contribution measurement criterion, and the weights corresponding to the multiple parameters are set as the contributions corresponding to the multiple parameters in each convolutional layer. The parameter contribution measurement criterion is:

[0016]

[0017] in, represents the contribution of the jth parameter of the i-th layer, wj (i) represents the weight of the jth parameter of the i-th layer.

[0018] The parameter contribution measurement criterion is used to measure the contribution of the model parameters to the target data set. The contribution of the parameters to the target data set is measured according to the parameter weight itself, and the contribution of each parameter to the target data set can be accurately quantified.

[0019] The beneficial effect of adopting the above further scheme is: by measuring the contribution of model parameters to the pre-trained model, the fine-tuning process can be better guided, thereby improving the model's processing performance on the target data set. Since the size of the parameter value itself can reflect its importance to the model, the contribution of the parameter to the target data set is measured according to the size of the parameter weight itself to obtain the parameter contribution, so as to fine-tune some parameters with lower contribution to improve the performance of the model on the target data set.

[0020] Furthermore, the candidate solution includes a fine-tuning ratio of each convolution layer; the contribution corresponding to multiple parameters of each convolution layer is calculated by using multiple candidate solutions to obtain multiple initial mask matrices, specifically:

[0021] By fine-tuning the ratio of each convolution layer in the multiple candidate solutions, multiple parameters of each convolution layer are calculated respectively to obtain multiple contribution thresholds of each convolution layer, and the contributions corresponding to the multiple parameters in each convolution layer are compared with the multiple contribution thresholds of the corresponding layers, and multiple initial mask matrices are obtained according to the comparison results.

[0022] Furthermore, the contribution corresponding to the multiple parameters in each convolutional layer is compared with the multiple contribution thresholds of the corresponding layer, specifically:

[0023] The contribution corresponding to multiple parameters in each convolution layer is compared with multiple contribution thresholds of the corresponding layer through a mask value calculation expression, and the mask value calculation expression is:

[0024]

[0025] i∈{1,2,…,L},j∈{1,2,…,N i},

[0026] in, represents the mask of the jth parameter of the i-th layer, represents the contribution of the jth parameter of the i-th layer, N represents the contribution threshold of the i-th layer in the k-th candidate solution of the t-th generation population, i represents the total number of parameters in the i-th layer, ξ k,i represents the fine-tuning ratio of the i-th layer in the k-th candidate solution, and L represents the total number of convolutional layers of the pre-trained model.

[0027] The beneficial effect of adopting the above further scheme is: according to the comparison between the contribution of the parameters and the contribution threshold of the parameters of each layer, the mask matrix is ​​generated based on the comparison result, and the mask matrix freezes some parameters during fine-tuning, and some parameters are fine-tuned. The parameters in the pre-trained model are partially fine-tuned according to the corresponding mask values ​​in the mask matrix, which determines the final performance of the model.

[0028] Furthermore, the pre-training model and the plurality of initial mask matrices are respectively adjusted and calculated according to the first preset period to obtain a plurality of initial fine-tuning training models, specifically:

[0029] According to the first preset period, the pre-trained model and the multiple initial mask matrices are respectively subjected to multiple parameter adjustment calculations to obtain multiple initial fine-tuning training models, wherein the expression of one parameter adjustment calculation is:

[0030]

[0031] in, represents the initial fine-tuning training model, W represents the pre-training model, η represents the learning rate during training, represents the fine-tuning function, M represents the initial mask matrix, and ⊙ represents element-by-element multiplication.

[0032] The beneficial effect of adopting the above further scheme is: the model is initially fine-tuned through the initial mask matrix to obtain the training model after the initial fine-tuning, which is convenient for evaluating the performance of the training model after the initial fine-tuning according to the fine-tuning ratio, and judging whether the corresponding fine-tuning ratio is suitable for the model, so as to update the fine-tuning ratio and obtain a parameter fine-tuning ratio that is more suitable for the model.

[0033] Furthermore, the preset target data set is input into the multiple initial fine-tuning training models for calculation to obtain multiple fitness values, specifically:

[0034] The preset target data set is input into the multiple initial fine-tuning training models respectively, and the classification accuracy of the target data set and the multiple initial fine-tuning training models is calculated respectively by the fitness function to obtain multiple fitness values. The fitness function is:

[0035]

[0036] Where f(·) represents the fitness function, ξ k represents the kth candidate solution among multiple candidate solutions, D t represents the target dataset, represents the initial fine-tuning training model, and Acc(·) represents the classification accuracy function.

[0037] The beneficial effect of adopting the above further scheme is: by calculating the fitness of each candidate solution to the initial fine-tuning training model, the candidate solution that best suits the pre-trained model is found according to the obtained fitness value, so that multiple candidate solutions can be updated according to the fine-tuning ratio with high fitness value, and then the parameters can be fine-tuned in the most appropriate way.

[0038] Furthermore, in the particle swarm optimization PSO, the multiple candidate solutions are a population, and the population includes multiple individuals, then the candidate solution is an individual, and the multiple individuals include individual historical optimal solutions and group global optimal solutions;

[0039] The updating of the plurality of candidate solutions by using the plurality of fitness values ​​until a preset number of iterations is satisfied, and obtaining the optimal solution according to the updated plurality of candidate solutions, is specifically as follows:

[0040] The individuals corresponding to the plurality of fitness values ​​are selected by the particle swarm optimization PSO to obtain the optimal individual, and the individual historical optimal solution and the group global optimal solution are respectively updated by the optimal individual, and the plurality of individuals are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution until a preset number of iterations is met, so as to obtain a plurality of generations of populations, each generation of populations including a plurality of updated individuals, and a plurality of update mask matrices are obtained by respectively calculating a plurality of parameters of each convolutional layer with the plurality of updated individuals;

[0041] According to the first preset period, the pre-trained model and the multiple update mask matrices are respectively adjusted and calculated to obtain multiple update fine-tuning training models; the target data set is respectively input into the multiple update fine-tuning training models for calculation to obtain multiple update fitness values, and the update individuals corresponding to the multiple update fitness values ​​are selected to obtain the optimal solution for the optimal update individual.

[0042] The beneficial effect of adopting the above further scheme is: since too large a fine-tuning ratio may cause the model to overfit on the target data set, and too small a ratio may cause the model to be unable to adapt well to the target data set, the fine-tuning ratio is accurately estimated, and the optimal fine-tuning ratio is searched for the target data set through particle swarm optimization PSO. The particle swarm optimization PSO search strategy is easy to calculate and has strong global search capabilities.

[0043] Furthermore, the individual includes multiple positions and multiple speeds, and the position is a fine-tuning ratio of a convolution layer, and the multiple individuals are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution, specifically:

[0044] The multiple speeds are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution to obtain multiple updated speeds. The calculation expression of the iterative update is:

[0045]

[0046] The corresponding positions are updated according to the multiple update speeds to obtain multiple updated positions, and the calculation expression for the position update is:

[0047]

[0048] in, represents the velocity of the dth layer in the kth individual of the t+1th generation population, α represents the inertia weight, represents the speed of the dth layer in the kth individual of the tth generation population, c1 represents the individual learning factor, r1 represents the individual optimization parameter, and p k,d represents the individual historical optimal solution of the kth individual at the dth level, represents the position of the dth layer in the kth individual of the tth generation population, c2 represents the population learning factor, r2 represents the population optimization parameter, and p g,d represents the global optimal solution of the group of the kth individual at the dth level, Represents the position of the dth layer in the kth individual of the t+1th generation population.

[0049] The beneficial effect of adopting the above further scheme is: the position and speed of each individual are continuously updated according to the guidance of the updated individual historical optimal solution and the updated group global optimal solution until the number of iterations is reached to generate the best individual, that is, the optimal solution, so as to fine-tune the parameters of each layer in the model through the fine-tuning ratio of each layer of parameters in the optimal solution.

[0050] Another technical solution of the present invention to solve the above technical problems is as follows:

[0051] An adaptive fine-tuning system based on contribution measurement, comprising: an import model module, a fine-tuning ratio search module, an initial fine-tuning module, a fine-tuning ratio update module and a final fine-tuning module;

[0052] The import model module is used to import a pre-trained model, wherein the pre-trained model includes multiple convolutional layers, and each convolutional layer includes multiple parameters;

[0053] The fine-tuning ratio search module is used to fine-tune and evaluate multiple parameters in each layer through particle swarm optimization (PSO) to obtain multiple candidate solutions;

[0054] The initial fine-tuning module is used to measure multiple parameters of each convolution layer by a contribution measurement criterion to obtain the contribution corresponding to the multiple parameters in each convolution layer, calculate the contribution corresponding to the multiple parameters of each convolution layer by using the multiple candidate solutions, obtain multiple initial mask matrices, perform parameter adjustment calculation on the pre-trained model and the multiple initial mask matrices according to a first preset period, obtain multiple initial fine-tuning training models, input the preset target data sets into the multiple initial fine-tuning training models for calculation, and obtain multiple fitness values;

[0055] The fine-tuning ratio update module is used to update the multiple candidate solutions through the multiple fitness values ​​until a preset number of iterations is met, and obtain the optimal solution according to the updated multiple candidate solutions;

[0056] The final fine-tuning module is used to calculate the contribution corresponding to multiple parameters of each convolutional layer through the optimal solution to obtain a mask matrix, and to adjust the parameters of the pre-trained model and the mask matrix according to a second preset period to obtain a fine-tuned training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of an adaptive fine-tuning method based on contribution measurement provided by an embodiment of the present invention;

[0058] Figure 2 A module block diagram of an adaptive fine-tuning system based on contribution measurement provided by an embodiment of the present invention;

[0059] Figure 3 A framework diagram of an adaptive fine-tuning method based on contribution measurement provided by an embodiment of the present invention;

[0060] Figure 4 A flow chart of an adaptive fine-tuning ratio search strategy provided by an embodiment of the present invention;

[0061] Figure 5 The effect diagram of the test accuracy curve of the Stanford-Dogs data set provided by the embodiment of the present invention;

[0062] Figure 6 This is a graph showing the effect of the test accuracy curve on the MIT Indoors dataset provided by an embodiment of the present invention;

[0063] Figure 7 The effect diagram of the test accuracy curve of the Caltech256-30 data set provided by the embodiment of the present invention;

[0064] Figure 8 This is a diagram showing the effect of the test accuracy curve on the Caltech256-60 dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0066] like Figure 1 and Figure 3 As shown, the adaptive fine-tuning method based on contribution measurement provided by an embodiment of the present invention includes the following steps:

[0067] S1. Importing a pre-trained model, wherein the pre-trained model includes multiple convolutional layers, and each convolutional layer includes multiple parameters;

[0068] S2, fine-tune and evaluate multiple parameters in each layer through particle swarm optimization (PSO) to obtain multiple candidate solutions;

[0069] S3, measuring multiple parameters of each convolution layer by a contribution measurement criterion to obtain contributions corresponding to multiple parameters in each convolution layer, calculating contributions corresponding to multiple parameters of each convolution layer by multiple candidate solutions to obtain multiple initial mask matrices, performing parameter adjustment calculations on the pre-trained model and the multiple initial mask matrices according to a first preset period to obtain multiple initial fine-tuning training models, inputting preset target data sets into the multiple initial fine-tuning training models for calculation to obtain multiple fitness values;

[0070] S4, updating the plurality of candidate solutions by using the plurality of fitness values ​​until a preset number of iterations is met, and obtaining an optimal solution according to the updated plurality of candidate solutions;

[0071] S5. Calculate the contribution corresponding to multiple parameters of each convolutional layer through the optimal solution to obtain a mask matrix, and adjust the parameters of the pre-trained model and the mask matrix according to a second preset period to obtain a fine-tuning training model.

[0072] Specifically, let W represent the pre-trained model, represents the weights of all parameters of the pre-trained model W, where L represents the total number of layers of the network model (i.e., the pre-trained model), and N i represents the total number of parameters in the i-th layer of the pre-trained model, represents the weight of the jth parameter of the i-th layer of the pre-trained model, represents the mask matrix corresponding to the pre-trained model W.

[0073] It should be understood that the first preset cycle can be set to 5 times, and the second preset cycle can be set to 110 times. The preset cycle is the fine-tuning cycle of the pre-trained model. A fine-tuning cycle of 1 time means that the pre-trained model is fine-tuned once through the candidate solution.

[0074] In the embodiment of the present invention, parameter-level fine-tuning is achieved through contribution measurement criteria and adaptive fine-tuning ratio search strategy, so that the model reaches a balance between mismatching target tasks and overfitting target tasks, and the performance of the model on the target task is significantly improved through a small amount of training time.

[0075] Preferably, the multiple parameters of each convolutional layer are measured by the contribution measurement criterion to obtain the contribution corresponding to the multiple parameters in each convolutional layer, specifically:

[0076] The weights of multiple parameters of each convolutional layer are measured by the contribution measurement criterion, and the weights corresponding to the multiple parameters are set as the contributions corresponding to the multiple parameters in each convolutional layer. The parameter contribution measurement criterion is:

[0077]

[0078] in, represents the contribution of the jth parameter of the i-th layer, represents the weight of the jth parameter of the i-th layer.

[0079] It should be understood that in the particle swarm optimization PSO, each particle represents an individual in the search space, and the particle swarm optimization PSO searches for the multiple parameters to be fine-tuned and randomly generates multiple individuals, that is, multiple candidate solutions.

[0080] In an embodiment of the present invention, by measuring the contribution of model parameters to the pre-trained model, the fine-tuning process can be better guided, thereby improving the processing performance of the model on the target data set. Since the size of the parameter value itself can reflect its importance to the model, the contribution of the parameter to the target data set is measured according to the size of the parameter weight itself to obtain the contribution of the parameter, so as to fine-tune some parameters with lower contribution to improve the performance of the model on the target data set.

[0081] Preferably, the candidate solution includes a fine-tuning ratio of each convolution layer; the contribution corresponding to multiple parameters of each convolution layer is calculated by using multiple candidate solutions to obtain multiple initial mask matrices, specifically:

[0082] By fine-tuning the ratio of each convolution layer in the multiple candidate solutions, multiple parameters of each convolution layer are calculated respectively to obtain multiple contribution thresholds of each convolution layer, and the contributions corresponding to the multiple parameters in each convolution layer are compared with the multiple contribution thresholds of the corresponding layers, and multiple initial mask matrices are obtained according to the comparison results.

[0083] Preferably, the contribution corresponding to the multiple parameters in each convolutional layer is compared with the multiple contribution thresholds of the corresponding layer respectively, specifically:

[0084] The contribution corresponding to multiple parameters in each convolution layer is compared with multiple contribution thresholds of the corresponding layer through a mask value calculation expression, and the mask value calculation expression is:

[0085]

[0086] i∈{1,2,…,L},j∈{1,2,…,N i},

[0087] in, represents the mask of the jth parameter of the i-th layer, represents the contribution of the jth parameter of the i-th layer, N represents the contribution threshold of the i-th layer in the k-th candidate solution of the t-th generation population, i represents the total number of parameters in the i-th layer, ξ k,i represents the fine-tuning ratio of the i-th layer in the k-th candidate solution, and L represents the total number of convolutional layers of the pre-trained model.

[0088] Specifically, parameters with a mask value of 1 will be updated (i.e., fine-tuned), while parameters with a mask value of 0 will remain unchanged (i.e., frozen).

[0089] In an embodiment of the present invention, the contribution of the parameters is compared with the contribution threshold of the parameters of each layer, and the comparison result is used to generate a mask matrix. The mask matrix freezes some parameters during fine-tuning and fine-tunes some parameters. The parameters in the pre-trained model are partially fine-tuned according to the corresponding mask values ​​in the mask matrix, which determines the final performance of the model.

[0090] Preferably, the pre-training model and the plurality of initial mask matrices are respectively adjusted and calculated according to the first preset period to obtain a plurality of initial fine-tuning training models, specifically:

[0091] According to the first preset period, the pre-trained model and the multiple initial mask matrices are respectively subjected to multiple parameter adjustment calculations to obtain multiple initial fine-tuning training models, wherein the expression of one parameter adjustment calculation is:

[0092]

[0093] in, represents the initial fine-tuning training model, W represents the pre-training model, η represents the learning rate during training, represents the fine-tuning function, M represents the initial mask matrix, and ⊙ represents element-by-element multiplication.

[0094] In an embodiment of the present invention, the model is initially fine-tuned through an initial mask matrix to obtain a training model after the initial fine-tuning, which is convenient for evaluating the performance of the training model after the initial fine-tuning according to the fine-tuning ratio, and judging whether the corresponding fine-tuning ratio is suitable for the model, so as to update the fine-tuning ratio and obtain a parameter fine-tuning ratio that is more suitable for the model.

[0095] Preferably, the preset target data set is input into a plurality of the initial fine-tuning training models for calculation to obtain a plurality of fitness values, specifically:

[0096] The preset target data set is input into the multiple initial fine-tuning training models respectively, and the classification accuracy of the target data set and the multiple initial fine-tuning training models is calculated respectively by the fitness function to obtain multiple fitness values. The fitness function is:

[0097]

[0098] Where f(·) represents the fitness function, ξ k represents the kth candidate solution among multiple candidate solutions, D t represents the target dataset, represents the initial fine-tuning training model, and Acc(·) represents the classification accuracy function.

[0099] It should be understood that the fitness calculation is performed on the multiple updated individuals to obtain multiple updated fitness values; the implementation method is the same as the fitness calculation is performed on the multiple initial individuals to obtain multiple fitness values.

[0100] In an embodiment of the present invention, by calculating the fitness of each candidate solution to the initial fine-tuning training model, the candidate solution that best suits the pre-training model is found based on the obtained fitness value, so that multiple candidate solutions can be updated according to the fine-tuning ratio with a high fitness value, and then the parameters can be fine-tuned in the most appropriate way.

[0101] Preferably, if Figure 4 As shown, in the particle swarm optimization PSO, the multiple candidate solutions are a population, and the population includes multiple individuals, then the candidate solution is an individual, and the multiple individuals include individual historical optimal solutions and group global optimal solutions;

[0102] The updating of the plurality of candidate solutions by using the plurality of fitness values ​​until a preset number of iterations is satisfied, and obtaining the optimal solution according to the updated plurality of candidate solutions, is specifically as follows:

[0103] The individuals corresponding to the plurality of fitness values ​​are selected by the particle swarm optimization PSO to obtain the optimal individual, and the individual historical optimal solution and the group global optimal solution are respectively updated by the optimal individual, and the plurality of individuals are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution until a preset number of iterations is met, so as to obtain a plurality of generations of populations, each generation of populations including a plurality of updated individuals, and a plurality of update mask matrices are obtained by respectively calculating a plurality of parameters of each convolutional layer with the plurality of updated individuals;

[0104] According to the first preset period, the pre-trained model and the multiple update mask matrices are respectively adjusted and calculated to obtain multiple update fine-tuning training models; the target data set is respectively input into the multiple update fine-tuning training models for calculation to obtain multiple update fitness values, and the update individuals corresponding to the multiple update fitness values ​​are selected to obtain the optimal solution for the optimal update individual.

[0105] Specifically, multiple individuals ξ=(ξ1,…,ξ k ,…,ξ n ),

[0106] The optimal individual ξ k =(ξ k,1 ,…,ξ k,i ,…,ξ k,L ),

[0107] Among them, ξ k represents the kth individual in the population, ξ k,i represents the fine-tuning ratio of the i-th layer parameters of the k-th individual, and ξ k,i ∈[0,1]. The preset number of iterations can be set to 15 times.

[0108] In the embodiment of the present invention, since too large a fine-tuning ratio may cause the model to overfit on the target data set, and too small a ratio may cause the model to be unable to adapt to the target data set well, the fine-tuning ratio is accurately estimated, and the optimal fine-tuning ratio is searched for the target data set through particle swarm optimization PSO, and the particle swarm optimization PSO search strategy is easy to calculate and has strong global search capability.

[0109] Preferably, the individual includes multiple positions and multiple speeds, and the position is a fine-tuning ratio of a convolution layer, and the iterative update of the multiple individuals according to the updated individual historical optimal solution and the updated group global optimal solution is specifically as follows:

[0110] The multiple speeds are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution to obtain multiple updated speeds. The calculation expression of the iterative update is:

[0111]

[0112] The corresponding positions are updated according to the multiple update speeds to obtain multiple updated positions, and the calculation expression for the position update is:

[0113]

[0114] in, represents the velocity of the dth layer in the kth individual of the t+1th generation population, α represents the inertia weight, represents the speed of the dth layer in the kth individual of the tth generation population, c1 represents the individual learning factor, r1 represents the individual optimization parameter, and p k,d represents the individual historical optimal solution of the kth individual at the dth level, represents the position of the dth layer in the kth individual of the tth generation population, c2 represents the population learning factor, r2 represents the population optimization parameter, and p g,d represents the global optimal solution of the group of the kth individual at the dth level, Represents the position of the dth layer in the kth individual of the t+1th generation population.

[0115] It should be understood that in particle swarm optimization PSO, each particle represents an individual in the search space, and they adjust their own position and speed through the guidance of the individual's historical optimal solution and the group's global optimal solution, and continuously update the position and speed of the particle (i.e., individual) in an iterative manner until an individual that meets the stopping condition is found. Among them, the position of the dth layer in the kth individual of the tth generation population corresponds to the fine-tuning ratio of the dth layer in the kth candidate solution among the updated tth generation multiple candidate solutions.

[0116] In an embodiment of the present invention, the position and speed of each individual are continuously updated based on the guidance of the updated individual historical optimal solution and the updated group global optimal solution until the number of iterations is reached to generate the best individual, that is, the optimal solution, so that each layer of parameters in the model is fine-tuned by the fine-tuning ratio of each layer of parameters in the optimal solution.

[0117] Preferably, in order to evaluate the effectiveness of the present invention, the parameter fine-tuning method of the present invention and the existing fine-tuning method are applied to 7 public data sets for comparison through experiments. In addition, the effectiveness of the contribution measurement criterion and the adaptive fine-tuning ratio search strategy in the present invention is also analyzed. Among them, the pre-trained model ResNet50 is used as the pre-training model, and the public image classification data sets in 7 different fields are: data set Stanford dogs, data set MIT Indoors, data set Caltech256-30, data set Caltech256-60, data set Aircraft, data set UCF-101 and data set Omniglot.

[0118] Preferably, the effectiveness of the contribution measurement criteria is analyzed through Experiment 1, specifically:

[0119] The effectiveness of the contribution metric in the present invention is verified by comparing the performance of the random fine-tuning method and the contribution fine-tuning method. In the random fine-tuning experiment, the parameters of each layer of the pre-trained model are randomly selected for fine-tuning based on the conditional parameters according to the fine-tuning ratio. In the contribution fine-tuning experiment, the parameters with smaller contribution in each layer of the pre-trained model are selected for fine-tuning according to the contribution value of each parameter and the fine-tuning ratio.

[0120] In order to better verify the effectiveness of the contribution metric, the two fine-tuning methods were compared at layer fine-tuning ratios of 0.4, 0.5, and 0.6, respectively. The layer fine-tuning ratio here refers to the fine-tuning ratio of each layer parameter in the pre-trained model. All experiments were tested on the 7 datasets introduced above.

[0121] Table 1 shows the classification accuracy of different data sets when the model is applied with two fine-tuning methods. As shown in Table 1, under the preset fine-tuning ratio, the two fine-tuning methods are applied to 7 data sets, and the experimental results after fine-tuning the parameters in the pre-trained model ResNet50 (i.e., the classification accuracy of the model) are obtained. Under the three set fine-tuning ratios, in the experiment of each fine-tuning ratio, the contribution fine-tuning method in the invention obtained better results on 5 of the 7 data sets. The reason for this result is that the target data set lacks sufficient training data, and overfitting problems are prone to occur during the fine-tuning process. This paper fine-tunes the parameters with small contribution to make the model match the target data set while also greatly maintaining the initial generalization ability of the model. Therefore, based on this experimental result, the effectiveness of the proposed contribution measurement criterion in the fine-tuning process is verified.

[0122] Table 1

[0123]

[0124] Preferably, the effectiveness of the adaptive fine-tuning ratio search strategy is analyzed through Experiment 2, specifically:

[0125] In the experiment of the adaptive fine-tuning ratio search strategy, the number of populations N initially generated was set to 10, the number of population iterations G (i.e., the preset number of iterations) was set to 15 times, and when evaluating the fitness of individuals (i.e., candidate solutions), the fine-tuning cycle of each individual on the model (i.e., the first preset cycle) was set to 5 times. In the individual speed update formula, the individual learning factor c1 and the group learning factor c2 were both set to 1.5, and the inertia weight α was linearly decreased in the range of (0.7, 1.4). In the experiment of random fine-tuning ratio, the fine-tuning ratio of each data set was randomly generated according to the preset parameters. According to the generated fine-tuning ratio, the second preset cycle was set to 110 times, and the pre-trained model was periodically fine-tuned 110 times, and the fine-tuning performance of the two methods was evaluated.

[0126] Table 2 shows the classification accuracy of different data sets when the model applies two different fine-tuning ratios. As shown in Table 2, different fine-tuning ratios are generated for 7 data sets in the pre-trained model ResNet50 through two fine-tuning methods, and the parameters in the pre-trained model ResNet50 are fine-tuned according to the fine-tuning ratios of the corresponding data sets, and the experimental results (i.e., classification accuracy) of the fine-tuned model are calculated. It can be seen from Table 2 that the adaptive fine-tuning ratio search strategy of the present invention obtains the best fine-tuning results on multiple experimental data sets, which shows that the adaptive fine-tuning ratio search strategy can accurately find the best fine-tuning ratio.

[0127] Table 2

[0128]

[0129] In addition, Table 3 shows the average value of the fine-tuning ratio of each layer generated by the present invention for different data sets. As shown in Table 3, the optimal fine-tuning ratios corresponding to the 7 data sets searched by the adaptive fine-tuning ratio search strategy on the pre-trained model ResNet50, and the average value of the parameter fine-tuning ratio of each layer in the optimal fine-tuning ratio are calculated. Among them, the average fine-tuning ratios of the data sets Stanford-Dogs, Caltech256-30, and Caltech256-60 are about 0.35, indicating that these three data sets are similar to the data set ImageNet of the source task, and a large amount of source domain knowledge can be reused (that is, the model is trained through the initial data set ImageNet, and the parameters or feature representation of the model after training) and only a small number of parameters need to be fine-tuned. The fine-tuning ratios of the data sets MIT Indoors, Aircraft, UCF-101, and Omniglot are above 0.5, indicating that these four data sets are far away from the data sets of the source task and more parameters need to be fine-tuned.

[0130] In summary, through the analysis of Table 2 and Table 3, the effectiveness of the adaptive fine-tuning ratio search strategy is fully verified, which can accurately determine the fine-tuning ratio suitable for the target dataset, thereby improving the model performance during the fine-tuning process.

[0131] Table 3

[0132]

[0133] Preferably, the fine-tuning method of the present invention is compared with the advanced fine-tuning method through Experiment 3, specifically:

[0134] The standard fine-tuning baseline (i.e., the Standard Fine-Tuning method) was compared with the fine-tuning method of the present invention, and the second preset cycle of all methods was set to 110 times. In order to eliminate the influence of randomness, the experiment was repeated 5 times to obtain the average classification accuracy.

[0135] Table 4 shows the classification accuracy of different fine-tuning methods for different data sets. As shown in Table 4, the above 7 data sets are input into the standard fine-tuning baseline and the present invention for parameter fine-tuning to obtain the experimental results. The experimental results show that the present invention obtains the best classification accuracy on all 7 data sets.

[0136] Table 4

[0137]

[0138] like Figure 5-Figure 8 The figure shows the test accuracy curves of the present invention (i.e., Our Method) and the standard fine-tuning baseline (i.e., StandardFine-Tuning method) on the data sets Stanford-Dogs, MIT Indoors, Caltech256-30, and Caltech256-60, where the curve marked with the symbol "×" represents the fine-tuning result produced by the present invention, the curve marked with the symbol "+" represents the fine-tuning result of the standard fine-tuning baseline, Epoch represents the fine-tuning cycle, and TestAccuracy represents the classification accuracy (i.e., test accuracy). It can be clearly seen that on all data sets, the present invention has higher accuracy than the standard fine-tuning baseline.

[0139] Specifically, after five periodic fine-tuning of the model parameters according to the first preset cycle, the test accuracy of the present invention on the Stanford-Dogs dataset, the MIT Indoors dataset, the Caltech256-30 dataset, and the Caltech256-60 dataset reached 83.67%, 71.86%, 80.52% and 83.51%, while the standard fine-tuning baseline only reached 67.73%, 62.61%, 69.66% and 71.28%, which shows that the present invention can reach or even exceed the accuracy level of the standard fine-tuning baseline with fewer fine-tuning cycles, and can reduce the training time of new tasks, which has important practical significance.

[0140] like Figure 2 As shown, the adaptive fine-tuning system based on contribution measurement provided by the embodiment of the present invention includes: an import model module, a fine-tuning ratio search module, an initial fine-tuning module, a fine-tuning ratio update module and a final fine-tuning module;

[0141] The import model module is used to import a pre-trained model, wherein the pre-trained model includes multiple convolutional layers, and each convolutional layer includes multiple parameters;

[0142] The fine-tuning ratio search module is used to fine-tune and evaluate multiple parameters in each layer through particle swarm optimization (PSO) to obtain multiple candidate solutions;

[0143] The initial fine-tuning module is used to measure multiple parameters of each convolution layer by a contribution measurement criterion to obtain the contribution corresponding to the multiple parameters in each convolution layer, calculate the contribution corresponding to the multiple parameters of each convolution layer by using the multiple candidate solutions, obtain multiple initial mask matrices, perform parameter adjustment calculation on the pre-trained model and the multiple initial mask matrices according to a first preset period, obtain multiple initial fine-tuning training models, input the preset target data sets into the multiple initial fine-tuning training models for calculation, and obtain multiple fitness values;

[0144] The fine-tuning ratio update module is used to update the multiple candidate solutions through the multiple fitness values ​​until a preset number of iterations is met, and obtain the optimal solution according to the updated multiple candidate solutions;

[0145] The final fine-tuning module is used to calculate the contribution corresponding to multiple parameters of each convolutional layer through the optimal solution to obtain a mask matrix, and to adjust the parameters of the pre-trained model and the mask matrix according to a second preset period to obtain a fine-tuned training model.

[0146] The above-mentioned adaptive fine-tuning system based on contribution measurement can refer to the implementation content and beneficial effects of the adaptive fine-tuning method based on contribution measurement specifically described above, which will not be repeated here.

[0147] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive fine-tuning method based on contribution measurement, characterized in that: Applied to the field of image processing, it includes the following steps: S1, importing a pre-trained image model obtained by pre-training an initial image data set, wherein the pre-trained image model includes multiple convolutional layers, and each convolutional layer includes multiple parameters; S2, fine-tune and evaluate multiple parameters in each layer through particle swarm optimization (PSO) to obtain multiple candidate solutions; S3, measuring multiple parameters of each convolution layer by a contribution measurement criterion to obtain contributions corresponding to multiple parameters in each convolution layer, calculating contributions corresponding to multiple parameters of each convolution layer by multiple candidate solutions to obtain multiple initial mask matrices, performing parameter adjustment calculations on the pre-trained image model and the multiple initial mask matrices according to a first preset period to obtain multiple initial fine-tuning training image models, inputting preset target image data sets into the multiple initial fine-tuning training image models for calculation to obtain multiple fitness values; S4, updating the plurality of candidate solutions by using the plurality of fitness values ​​until a preset number of iterations is met, and obtaining an optimal solution according to the updated plurality of candidate solutions; S5. Calculate the contribution corresponding to multiple parameters of each convolutional layer through the optimal solution to obtain a mask matrix, and adjust the parameters of the pre-trained image model and the mask matrix according to a second preset period to obtain a fine-tuned training image model.

2. The adaptive fine-tuning method according to claim 1, characterized in that: The multiple parameters of each convolution layer are measured by the contribution measurement criterion to obtain the contribution corresponding to the multiple parameters in each convolution layer, specifically: The weights of multiple parameters of each convolutional layer are measured by the contribution measurement criterion, and the weights corresponding to the multiple parameters are set as the contributions corresponding to the multiple parameters in each convolutional layer. The parameter contribution measurement criterion is: in, represents the contribution of the jth parameter of the i-th layer, represents the weight of the jth parameter of the i-th layer.

3. The adaptive fine-tuning method according to claim 1, characterized in that: The candidate solution includes a fine-tuning ratio of each convolution layer; the contribution corresponding to multiple parameters of each convolution layer is calculated by using multiple candidate solutions to obtain multiple initial mask matrices, specifically: By fine-tuning the ratio of each convolution layer in the multiple candidate solutions, multiple parameters of each convolution layer are calculated respectively to obtain multiple contribution thresholds of each convolution layer, and the contributions corresponding to the multiple parameters in each convolution layer are compared with the multiple contribution thresholds of the corresponding layers, and multiple initial mask matrices are obtained according to the comparison results.

4. The adaptive fine-tuning method according to claim 3, characterized in that: The contribution corresponding to the multiple parameters in each convolutional layer is compared with the multiple contribution thresholds of the corresponding layer, specifically: The contribution corresponding to multiple parameters in each convolution layer is compared with multiple contribution thresholds of the corresponding layer through a mask value calculation expression, and the mask value calculation expression is: in, represents the mask of the jth parameter of the i-th layer, represents the contribution of the jth parameter of the i-th layer, represents the contribution threshold of the i-th layer, N i represents the total number of parameters in the i-th layer, ξ k,i represents the fine-tuning ratio of the i-th layer in the k-th candidate solution, and L represents the total number of convolutional layers of the pre-trained image model.

5. The adaptive fine-tuning method according to claim 1, characterized in that: The pre-trained image model and the plurality of initial mask matrices are respectively adjusted and calculated according to the first preset period to obtain a plurality of initial fine-tuned training image models, specifically: According to the first preset period, the pre-trained image model and the multiple initial mask matrices are respectively subjected to multiple parameter adjustment calculations to obtain multiple initial fine-tuned training image models, wherein the expression for one parameter adjustment calculation is: in, represents the initial fine-tuning training image model, W represents the pre-training image model, η represents the learning rate during training, represents the fine-tuning function, M represents the initial mask matrix, and ⊙ represents element-by-element multiplication.

6. The adaptive fine-tuning method according to claim 1, characterized in that: The preset target image data set is input into the multiple initial fine-tuning training image models for calculation to obtain multiple fitness values, specifically: The preset target image data set is input into the multiple initial fine-tuning training image models respectively, and the classification accuracy of the target image data set and the multiple initial fine-tuning training image models is calculated respectively by the fitness function to obtain multiple fitness values, and the fitness function is: Where f(·) represents the fitness function, ξ k represents the kth candidate solution among multiple candidate solutions, D t represents the target image dataset, represents the initial fine-tuned training image model, and Acc(·) represents the classification accuracy function.

7. The adaptive fine-tuning method according to claim 1, characterized in that: In the particle swarm optimization PSO, the multiple candidate solutions are a population, and the population includes multiple individuals, then the candidate solution is an individual, and the multiple individuals include individual historical optimal solutions and group global optimal solutions; The updating of the plurality of candidate solutions by using the plurality of fitness values ​​until a preset number of iterations is satisfied, and obtaining the optimal solution according to the updated plurality of candidate solutions, is specifically as follows: The individuals corresponding to the plurality of fitness values ​​are selected by the particle swarm optimization PSO to obtain the optimal individual, and the individual historical optimal solution and the group global optimal solution are respectively updated by the optimal individual, and the plurality of individuals are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution until a preset number of iterations is met, so as to obtain a plurality of generations of populations, each generation of populations including a plurality of updated individuals, and a plurality of update mask matrices are obtained by respectively calculating a plurality of parameters of each convolutional layer with the plurality of updated individuals; According to a first preset period, the pre-trained image model and the multiple update mask matrices are respectively adjusted and calculated to obtain multiple updated fine-tuning training image models; the target image data set is respectively input into the multiple updated fine-tuning training image models for calculation to obtain multiple updated fitness values, and the update individuals corresponding to the multiple updated fitness values ​​are selected to obtain the optimal solution for the optimal update individual.

8. The adaptive fine-tuning method according to claim 7, characterized in that: The individual includes multiple positions and multiple speeds, and the position is a fine-tuning ratio of a convolution layer. The multiple individuals are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution, specifically: The multiple speeds are iteratively updated according to the updated individual historical optimal solution and the updated group global optimal solution to obtain multiple updated speeds. The calculation expression of the iterative update is: The corresponding positions are updated according to the multiple update speeds to obtain multiple updated positions, and the calculation expression for the position update is: in, represents the velocity of the dth layer in the kth individual of the t+1th generation population, α represents the inertia weight, represents the speed of the dth layer in the kth individual of the tth generation population, c1 represents the individual learning factor, r1 represents the individual optimization parameter, and p k,d represents the individual historical optimal solution of the kth individual at the dth level, represents the position of the dth layer in the kth individual of the tth generation population, c2 represents the population learning factor, r2 represents the population optimization parameter, and p g,d represents the global optimal solution of the group of the kth individual at the dth level, Represents the position of the dth layer in the kth individual of the t+1th generation population.

9. An adaptive fine-tuning system based on contribution measurement, characterized in that: Applied in the field of image processing, including: import model module, fine-tuning scale search module, initial fine-tuning module, fine-tuning scale update module and final fine-tuning module; The import model module is used to import a pre-trained image model obtained by pre-training the initial image data set, wherein the pre-trained image model includes multiple convolutional layers, and each convolutional layer includes multiple parameters; The fine-tuning ratio search module is used to fine-tune and evaluate multiple parameters in each layer through particle swarm optimization (PSO) to obtain multiple candidate solutions; The initial fine-tuning module is used to measure multiple parameters of each convolution layer by a contribution measurement criterion to obtain contributions corresponding to multiple parameters in each convolution layer, calculate contributions corresponding to multiple parameters of each convolution layer by using multiple candidate solutions to obtain multiple initial mask matrices, perform parameter adjustment calculations on the pre-trained image model and the multiple initial mask matrices according to a first preset period to obtain multiple initial fine-tuning training image models, input preset target image data sets into the multiple initial fine-tuning training image models for calculation to obtain multiple fitness values; The fine-tuning ratio update module is used to update the multiple candidate solutions through the multiple fitness values ​​until a preset number of iterations is met, and obtain the optimal solution according to the updated multiple candidate solutions; The final fine-tuning module is used to calculate the contribution corresponding to multiple parameters of each convolutional layer through the optimal solution to obtain a mask matrix, and to adjust the parameters of the pre-trained image model and the mask matrix according to a second preset period to obtain a fine-tuned training image model.

10. The adaptive fine-tuning system according to claim 9, characterized in that: In the contribution measurement module, multiple parameters of each convolution layer are measured by the contribution measurement criterion to obtain the contribution corresponding to the multiple parameters in each convolution layer, specifically: The weights of multiple parameters of each convolutional layer are measured by the contribution measurement criterion, and the weights corresponding to the multiple parameters are set as the contributions corresponding to the multiple parameters in each convolutional layer. The parameter contribution measurement criterion is: in, represents the contribution of the jth parameter of the i-th layer, represents the weight of the jth parameter of the i-th layer.

Images