Model fine tuning method, device and equipment and computer readable storage medium

Abstract

The invention provides a model fine tuning method, device and equipment and a computer readable storage medium, and the method comprises the steps: testing a pre-training model through a general data set, and obtaining the classification characteristic value of each layer in the pre-training model; performing fine tuning on each layer in the pre-training model by using the fine tuning data set to obtain a precision contribution value of each layer in the pre-training model and a fine tuning duration of each layer; the optimal layer fine tuning strategy mathematical model is solved, an optimal layer fine tuning strategy is obtained, and the optimal layer fine tuning strategy mathematical model is constructed according to the classification characteristic value, the precision contribution value and the fine tuning duration. According to the method and the device, the pre-training model can adapt to various scenes by performing fine tuning on the pre-training model, the fine tuning strategy is determined by utilizing the precision contribution value of each layer of the pre-training model, the training data volume of the fine tuning pre-training model can be reduced, and the calculation cost is reduced while the training efficiency and precision are improved.

Description

Technical Field

[0001] The present application relates to the field of model processing technology, and in particular to a model fine-tuning method, device, equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of deep learning technology, especially in the field of computer vision (CV), large-scale pre-trained models have demonstrated strong generalization capabilities and excellent performance on a variety of downstream tasks. However, these pre-trained models usually contain billions or even trillions of parameters. If training is performed directly on the original pre-trained model for specific tasks, it will not only require a lot of computing resources and storage space, but may also lead to overfitting problems, especially when the training data is limited.

[0003] Model fine-tuning is the process of adjusting a small number of parameters (such as weights, biases, etc.) on a pre-trained model to adapt to new tasks and data. This can reduce resource consumption during fine-tuning while maintaining the generality of the model. However, existing fine-tuning methods still have some shortcomings, such as low fine-tuning efficiency, long time consumption, and fine-tuning effects that do not meet expectations. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a model fine-tuning method, device, equipment and computer-readable storage medium, which uses the accuracy contribution value of each layer of the pre-trained model to determine the fine-tuning strategy and improve the fine-tuning efficiency and accuracy. The specific technical solution is as follows:

[0005] In a first aspect, the present application provides a model fine-tuning method, the method comprising:

[0006] Using a general data set to test the pre-trained model, and obtaining the classification characteristic value of each layer in the pre-trained model;

[0007] Using the fine-tuning data set, fine-tune each layer in the pre-trained model to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning time of each layer;

[0008] Solve the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy, wherein the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the accuracy contribution value and the fine-tuning duration.

[0009] In a possible implementation, fine-tuning each layer in the pre-trained model using the fine-tuning data set to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning duration of each layer includes:

[0010] Performing precision calculation on the pre-trained model to obtain the original precision of the pre-trained model;

[0011] Using the fine-tuning data set to fine-tune each layer in the pre-trained model, respectively, to obtain multiple fine-tuning models and the fine-tuning duration corresponding to each of the fine-tuning models, wherein different fine-tuning models correspond to different fine-tuning layers;

[0012] Performing precision calculation on a plurality of the fine-tuning models to obtain a plurality of fine-tuning precisions;

[0013] According to the original accuracy of the pre-trained model and the multiple fine-tuning accuracy, the accuracy contribution value of each layer in the pre-trained model is calculated.

[0014] In a possible implementation, the fine-tuning each layer in the pre-trained model is fine-tuned using the fine-tuning dataset to obtain multiple fine-tuning models, including:

[0015] Fine-tune the fine-tuning layer of the pre-trained model using the fine-tuning data set to obtain the fine-tuning model, where the fine-tuning layer is any layer in the pre-trained model;

[0016] Each layer in the pre-trained model is traversed to obtain a plurality of the fine-tuning models.

[0017] In a possible implementation, solving the optimal layer fine-tuning strategy mathematical model to obtain the optimal layer fine-tuning strategy includes:

[0018] Genetic algorithm is used to solve the mathematical model of the optimal layer fine-tuning strategy and obtain the optimal layer fine-tuning strategy.

[0019] In a possible implementation, the method of testing the pre-trained model using a general data set to obtain the classification characteristic value of each layer in the pre-trained model includes:

[0020] Inputting the general data set into the pre-trained model, and determining the test results of each layer of the test data set in the pre-trained model;

[0021] The classification characteristic value of each layer in the pre-trained model is determined according to the test results of each layer in the pre-trained model.

[0022] In a second aspect, the present application also provides a model fine-tuning device, the device comprising:

[0023] A testing module, used to test the pre-trained model using a general data set to obtain the classification characteristic values ​​of each layer in the pre-trained model;

[0024] A fine-tuning module is used to fine-tune each layer in the pre-trained model using a fine-tuning data set to obtain an accuracy contribution value of each layer in the pre-trained model and a fine-tuning duration of each layer;

[0025] A solution module is used to solve the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy, wherein the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the accuracy contribution value and the fine-tuning duration.

[0026] In a possible implementation, the fine-tuning module includes:

[0027] An original precision calculation unit, used to perform precision calculation on the pre-trained model to obtain the original precision of the pre-trained model;

[0028] A fine-tuning unit, used to use the fine-tuning data set to fine-tune each layer in the pre-trained model respectively, to obtain multiple fine-tuning models and the fine-tuning duration corresponding to each of the fine-tuning models, wherein different fine-tuning models correspond to different fine-tuning layers;

[0029] A fine-tuning accuracy calculation unit, used to perform accuracy calculation on a plurality of the fine-tuning models to obtain a plurality of fine-tuning accuracies;

[0030] A calculation unit is used to calculate the accuracy contribution value of each layer in the pre-trained model according to the original accuracy of the pre-trained model and the multiple fine-tuning accuracy.

[0031] In a possible implementation manner, the fine-tuning unit is specifically configured to:

[0032] Fine-tune the fine-tuning layer of the pre-trained model using the fine-tuning data set to obtain the fine-tuning model, where the fine-tuning layer is any layer in the pre-trained model;

[0033] Each layer in the pre-trained model is traversed to obtain a plurality of the fine-tuning models.

[0034] In a third aspect, the present application further provides a computer device, including: a memory and a processor;

[0035] Wherein, the memory is used to store computer programs;

[0036] The processor is used to execute the computer program in the memory to implement the method described in the first aspect or any one of the first aspects.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any one of the first aspects.

[0038] In this application, a general data set is used to test the pre-trained model to obtain the classification characteristic value of each layer in the pre-trained model; each layer in the pre-trained model is fine-tuned using a fine-tuning data set to obtain the precision contribution value of each layer in the pre-trained model and the fine-tuning duration of each layer; the mathematical model of the optimal layer fine-tuning strategy is solved to obtain the optimal layer fine-tuning strategy, and the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the precision contribution value and the fine-tuning duration. This application can make the pre-trained model adapt to various scenarios by fine-tuning the pre-trained model, and determine the fine-tuning strategy using the precision contribution value of each layer of the pre-trained model, which can reduce the amount of training data for fine-tuning the pre-trained model, while improving training efficiency and accuracy, and reducing computational costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flow chart of a method for fine-tuning a model provided in an embodiment of the present application is shown;

[0041] Figure 2 A schematic diagram of the classification characteristics of the ResNet-18 model provided in an embodiment of the present application is shown;

[0042] Figure 3 A schematic diagram showing the accuracy of each layer of the ResNet-18 model provided in an embodiment of the present application is shown;

[0043] Figure 4 A schematic diagram of the fine-tuning time consumption of each layer of the ResNet-18 model provided in an embodiment of the present application is shown;

[0044] Figure 5 A schematic diagram showing a comparison of the model accuracy provided by the embodiment of the present application and the traditional fine-tuning method;

[0045] Figure 6 A schematic diagram showing a comparison of the fine-tuning duration provided by the embodiment of the present application and the traditional fine-tuning method;

[0046] Figure 7 A schematic structural diagram of a fine-tuning model device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] With the rapid development of IoT technology, more and more smart devices are widely used in various scenarios, from smart homes to smart cities to industrial automation. However, with the popularity of these smart devices, privacy issues are also increasingly attracting attention. In order to protect user privacy, it has become a trend for smart devices to perform data processing and model training locally. However, smart devices usually face the problem of limited computing resources and low computing power. How to effectively train models under the constraints of limited computing resources and low computing power has become a major challenge.

[0049] Specific tasks in IoT scenarios usually come from a single device in a single specific scenario, and the amount of data involved in these tasks is usually small. In this case, using traditional deep neural network models rarely achieves satisfactory results. This is because deep neural network models usually require a large amount of labeled data to be effectively trained, and the small sample data sets on smart devices can hardly meet this requirement. In order to solve this problem, the technology of few-sample learning is proposed, which aims to use a small amount of sample data for effective model training.

[0050] Transfer learning, as a variant of few-shot learning, has shown great potential in solving this problem. Transfer learning usually includes two main steps: transferring pre-trained models and fine-tuning. First, a pre-trained model is trained on a large-scale dataset, which helps to acquire a lot of prior knowledge. Then, this pre-trained model is applied to new tasks and fine-tuned to adapt to specific tasks in specific scenarios. In this way, even with less data, the model can use the knowledge learned from large-scale datasets to make effective predictions.

[0051] In the process of transfer learning, feature reuse is a key step. By using the feature extractor in the pre-trained model, feature vectors can be extracted from the few-shot dataset. These feature vectors contain a lot of useful information, which helps the model to make more accurate classification and predictions. However, simply using the classifier of the few-shot dataset based on cross entropy training model often leads to overfitting problems. This is because the model cannot generalize well to new samples when the amount of data is small.

[0052] In order to solve the overfitting problem, the few-shot classification method came into being. The few-shot classification method does not need to update the model parameters through gradient descent, but is implemented by building an end-to-end few-shot classifier. The few-shot classification method is based on the distribution characteristics of the samples, models the distance between samples, and classifies by clustering similar samples and separating dissimilar samples. Specifically, these methods usually use techniques such as metric learning or prototype networks to measure the similarity between samples and classify them based on the similarity. Metric learning maps samples to a new space by learning a mapping function so that the distance between similar samples is as close as possible and the distance between dissimilar samples is as far as possible. This method can effectively avoid the overfitting problem because it does not rely on a large amount of training data to update the model parameters, but classifies by the relative distance between samples. The prototype network performs classification by constructing a prototype vector for each category. Each prototype vector represents the center point of the category, and the new sample determines the category it belongs to based on the distance from these prototype vectors. This method has also shown good results in few-shot learning.

[0053] Although the above few-sample classification methods have solved the overfitting problem to a certain extent, their performance is still not ideal. In practical applications, how to further improve the effect of few-sample classification methods while ensuring model performance is still an important research direction.

[0054] In order to achieve efficient training of network models on IoT devices, the embodiments of the present application need to solve the problem of additional computing costs that are generated while improving the accuracy of the network model when fine-tuning the existing model fine-tuning methods on IoT devices. It is also necessary to solve the contradiction between the limited computing power of IoT devices and the huge computing power required for model training. The embodiments of the present application propose a model fine-tuning method, which quantitatively analyzes the contribution of each layer of the network model to the model performance, finds the optimal fine-tuning strategy based on these analyses to maximize the model accuracy, and then uses a solver to obtain the optimal fine-tuning strategy. Through multiple iterations, the time required for fine-tuning can be greatly reduced while ensuring the accuracy of the model, so that resource-constrained IoT devices can also perform high-quality model training after obtaining the optimal fine-tuning strategy.

[0055] See also Figure 1 , shows a flow chart of a model fine-tuning method provided by an embodiment of the present application. The embodiment of the present application can be applied to a server, which can communicate with an IoT device or other smart devices with limited resources and low computing power, and provide data support service devices for smart devices. The server can provide services for one smart device or multiple smart devices at the same time. The embodiment of the present application includes at least the following steps:

[0056] S11, using a general data set to test the pre-trained model to obtain the classification feature value of each layer in the pre-trained model.

[0057] When model training is required, select an applicable pre-trained model. Commonly used pre-trained models include VGG (Visual Geometry Group), Residual Neural Network (ResNet) and other models.

[0058] The neural network model consists of multiple layers, each of which has its own specific classification characteristics and functions. Figure 2 As shown in the figure, generally speaking, the lower layers of the neural network model are used to extract basic features such as edges and lines of the image. Correspondingly, the higher layers of the neural network model have better classification characteristics and can classify images into different categories. Therefore, fine-tuning the higher layers of the neural network model may enable the fine-tuned model to achieve higher model accuracy. Model accuracy can reflect the accuracy of the model in predicting unseen data.

[0059] The embodiment of the present application can use a general data set to test the pre-trained model and quantify the classification characteristics of each layer of the pre-trained model. Specifically, the general data set can be input into the pre-trained model to determine the test results of the general data set at each layer in the pre-trained model, and then the classification characteristic values ​​of each layer in the pre-trained model are determined according to the test results of each layer in the pre-trained model. The classification characteristic values ​​can reflect the classification characteristics of each layer of the model. The general data set can be the CUB-200-2011 data set, and the embodiment of the present application is not limited to a specific general data set.

[0060] The classification characteristic values ​​of each layer in the pre-trained model are determined according to the test results of each layer in the pre-trained model. The specific formula is as follows:

[0061]

[0062] Among them, S l is the classification feature value of the lth layer of the pre-trained model, N is the number of samples in the general dataset, i and j are the indexes of the samples, and y i is the category label of sample i, y j is the category label of sample j, w ij is a weight factor to adjust the importance of the distance between samples i and j. K(·) is the original distance Kernel function that maps to a higher dimensional feature space, is the distance between the representations of samples i and j at layer l, For category y i The variance within , indicating the category y i The degree of dispersion of the internal samples.

[0063] S12, using the fine-tuning data set to fine-tune each layer in the pre-trained model respectively, to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning time of each layer.

[0064] If directly migrating the parameters of the pre-trained model cannot adapt to the data set in a specific scenario, it will result in lower model accuracy. At the same time, fine-tuning all layers of the pre-trained model cannot achieve the optimal model accuracy and meet the fine-tuning time limit. Therefore, the embodiment of the present application analyzes the contribution of each layer of the pre-trained model to the model accuracy in order to derive the optimal layer fine-tuning strategy.

[0065] The embodiment of the present application can fine-tune the pre-trained model by fine-tuning the data set, so as to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning time of each layer. The fine-tuning data set can be a data set for a specific scenario, or a small sample data set for a specific scenario. If the optimal layer fine-tuning strategy of the pre-trained model is used for an IoT device, the fine-tuning data set can be a data set related to the IoT device, or the fine-tuning data set can be provided by the IoT device.

[0066] Specifically, fine-tuning each layer in the pre-trained model using the fine-tuning data set to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning time of each layer may include the following steps:

[0067] S121, performing accuracy evaluation on the pre-trained model to obtain the original accuracy of the pre-trained model.

[0068] The embodiment of the present application can use the fine-tuning data set to calculate the accuracy of the pre-trained model to obtain the original accuracy of the pre-trained model.

[0069] For classification problems, accuracy is usually the ratio of the number of correctly predicted samples to the total number of samples. Therefore, the fine-tuning dataset can be input into the pre-trained model to obtain the output of the pre-trained model, and the output of the pre-trained model can be compared with the fine-tuning dataset to determine the number of correctly predicted samples, thereby calculating the original accuracy of the pre-trained model.

[0070] The accuracy calculation process can be selected according to the actual scenario and is not limited in the embodiments of the present application.

[0071] S122, using the fine-tuning data set to fine-tune each layer in the pre-trained model respectively, to obtain multiple fine-tuning models and the fine-tuning duration corresponding to each fine-tuning model.

[0072] In the embodiment of the present application, different fine-tuning models correspond to different fine-tuning layers. That is to say, when fine-tuning each layer in the pre-trained model, the embodiment of the present application only fine-tunes one layer of the pre-trained model at a time, and the other layers remain unchanged. Assuming that the pre-trained model includes a first layer, a second layer, a third layer, and a fourth layer, when fine-tuning the first layer using a fine-tuning data set, the first layer is a fine-tuning layer, and the second layer, the third layer, and the fourth layer are frozen layers, that is, the model parameters of the second layer, the third layer, and the fourth layer remain unchanged, and only the model parameters of the first layer are adjusted; when fine-tuning the second layer using a fine-tuning data set, the second layer is a fine-tuning layer, and the first layer, the third layer, and the fourth layer are frozen layers, that is, the model parameters of the first layer, the third layer, and the fourth layer remain unchanged, and only the model parameters of the second layer are adjusted.

[0073] Specifically, the fine-tuning data set can be used to fine-tune the fine-tuning layer of the pre-trained model to obtain a fine-tuning model, and each layer in the pre-trained model can be traversed to obtain multiple fine-tuning models. The fine-tuning layer is any layer in the pre-trained model.

[0074] S123, performing precision calculation on multiple fine-tuning models to obtain multiple fine-tuning precisions.

[0075] The embodiment of the present application can use the fine-tuning data set to calculate the accuracy of each fine-tuning model to obtain multiple fine-tuning accuracies. Different fine-tuning accuracies correspond to different fine-tuning models. The embodiment of the present application takes the ResNet-18 model as an example. Figure 3 This is a schematic diagram of the accuracy of each layer of the ResNet-18 model. Figure 4 Schematic diagram of the fine-tuning time consumption of each layer of the ResNet-18 model.

[0076] In the embodiment of the present application, the fine-tuning data set can be input into the fine-tuning model to obtain the output result of the fine-tuning model, and the output result of the fine-tuning model can be compared with the fine-tuning data set to determine the number of correctly predicted samples, thereby calculating the original accuracy of the fine-tuning model.

[0077] S124, calculating the accuracy contribution value of each layer in the pre-trained model according to the original accuracy of the pre-trained model and multiple fine-tuning accuracy.

[0078] In the embodiment of the present application, the precision contribution value = fine-tuning precision - original precision. That is, the precision contribution value of each layer is the fine-tuning precision corresponding to the fine-tuning model obtained by fine-tuning each layer minus the original precision of the pre-trained model.

[0079] If the precision contribution value is negative, it means that fine-tuning the layer will not improve the model accuracy; if the precision contribution value is positive, it means that fine-tuning the layer can improve the model accuracy. Calculating the precision contribution value makes it easier to find a set of layers with high fine-tuning benefits. Obtaining the precision contribution value and fine-tuning duration corresponding to each layer provides a data basis for searching for the optimal layer fine-tuning strategy.

[0080] S13, solving the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy.

[0081] The embodiment of the present application can model the problem of searching for the optimal layer fine-tuning strategy as an integer programming problem, which is used to balance the model accuracy and fine-tuning duration while maximizing the model accuracy.

[0082] The mathematical model of the optimal layer fine-tuning strategy can be pre-built based on the classification feature value, accuracy contribution value and fine-tuning duration. The specific formula is as follows:

[0083]

[0084]

[0085] in, The representation layer fine-tuning strategy includes the k layers of the selected model M, K represents the layer index, Represents the accuracy function, acc origin represents the original accuracy of the model, Represents the model based on the fine-tuning dataset from acc origin Start fine-tuning the first k The fine-tuning accuracy achieved by the layer, Indicates l k The classification characteristics of the layer, It represents the correlation between the fine-tuning dataset and the pre-trained model, w represents the number of categories in the support dataset in the fine-tuning dataset, s represents the number of samples of each category in the support dataset, and cost() represents the use strategy Fine-tuned consumption function, Indicates usage strategy Time consumption of fine-tuning, Should be less than or equal to the time allowed at a given compression ratio c.

[0086] In an embodiment of the present application, the support dataset can be set to way=5, shot=5, where way represents the number of categories in the task, and shot represents the number of samples in each category; the query dataset can be set to query=10, where query represents the number of samples in each category in the query dataset; and tasks=100 is set, where tasks represents the number of tasks. Support datasets are typically used to provide additional, but limited supervisory information to the model during training or inference. In small sample learning tasks, the support dataset contains a small but critical amount of annotated data that is used to guide the model to quickly adapt to new tasks or categories. Query datasets are typically used to evaluate the performance of a model on unseen data. In the field of machine learning or deep learning, query datasets are datasets used to test the generalization ability of a model.

[0087] In the embodiment of the present application, the solver can be used to search for the optimal layer fine-tuning strategy. Specifically, the optimal layer fine-tuning strategy mathematical model is solved by a genetic algorithm to obtain the optimal layer fine-tuning strategy. The optimal layer fine-tuning strategy may include the level that needs to be fine-tuned.

[0088] Genetic Algorithm (GA) is a search heuristic algorithm that simulates natural selection and genetics mechanisms and is used to solve optimization and search problems. The embodiment of the present application uses a genetic algorithm to solve the argmax function.

[0089] Using genetic algorithm to solve the optimal fine-tuning strategy mathematical model and obtain the optimal layer fine-tuning strategy can include the following steps:

[0090] S131, setting relevant parameters of the genetic algorithm to generate individuals and populations.

[0091] The embodiment of the present application can set the population size, crossover probability, mutation probability and evolutionary generation.

[0092] An individual is a binary string indicating whether each layer is selected for fine-tuning, where 0 can indicate not selected and 1 can indicate selected. The population is the collection of these individuals.

[0093] S132, calculating the fitness of each individual using a fitness function according to the fine-tuning accuracy and the original accuracy.

[0094] The formula of the fitness function is as follows:

[0095]

[0096] The embodiment of the present application can also define a fitness function. If the constraints are met, that is, the fine-tuning duration is within the limit range, the Fit value is directly output; if the fine-tuning duration exceeds the limit range, a penalty value can be subtracted from the Fit value.

[0097] S133, perform genetic operations.

[0098] Selection operation: The tournament selection method can be used to select individuals with higher fitness from the population for reproduction. The fitness threshold can be set to determine whether the individual's fitness is high. When the individual's fitness is greater than the fitness threshold, the individual's fitness is high. When the individual's fitness is less than or equal to the fitness threshold, the individual does not belong to the individuals with higher fitness.

[0099] Crossover operation: The two-point crossover method can be used to exchange some genes of two individuals.

[0100] Mutation operation: The bit flip mutation method can be used to flip the gene value of an individual with a certain probability.

[0101] S134, output the optimal individual and the fitness corresponding to the optimal individual.

[0102] The embodiment of the present application first generates individuals and populations, calculates the fitness of individuals in the population, and then generates a new generation of populations through selection, crossover and mutation operations, and calculates the fitness of individuals in the new generation of populations. Repeat the above steps until a certain number of iterations is reached or other termination conditions are met (such as the fitness reaches a set threshold), and finally outputs the fitness corresponding to the optimal individual and the optimal individual. The optimal individual is the optimal layer fine-tuning strategy.

[0103] The embodiment of the present application uses a genetic algorithm to gradually evolve the optimal solution to the optimal layer fine-tuning strategy problem through selection, crossover and mutation operations, thereby improving the solution efficiency. After multiple iterations, the fine-tuning time can be reduced while ensuring the accuracy of the model, thereby improving the fine-tuning efficiency of the pre-trained model. After obtaining the optimal layer fine-tuning strategy, the embodiment of the present application can determine whether the fine-tuning time corresponding to the optimal layer fine-tuning strategy meets the time limit condition. After the fine-tuning time corresponding to the optimal layer fine-tuning strategy meets the time limit condition, it is determined whether the model accuracy corresponding to the optimal layer fine-tuning strategy is the optimal accuracy. If the fine-tuning time corresponding to the optimal layer fine-tuning strategy does not meet the time limit condition, or the model accuracy corresponding to the optimal layer fine-tuning strategy is not the optimal accuracy, the optimal layer fine-tuning strategy mathematical model is re-solved.

[0104] The optimal layer fine-tuning strategy is used to indicate which layers of the pre-trained model are fine-tuned. Therefore, after the optimal layer fine-tuning strategy is solved in the embodiment of the present application, the IoT device can directly obtain the optimal layer fine-tuning strategy and use the optimal layer fine-tuning strategy to fine-tune the pre-trained model.

[0105] In an embodiment of the present application, a pre-trained model is tested using a test data set to obtain the classification characteristic value of each layer in the pre-trained model; each layer in the pre-trained model is fine-tuned using a fine-tuning data set to obtain the precision contribution value of each layer in the pre-trained model and the fine-tuning duration of each layer; the mathematical model of the optimal layer fine-tuning strategy is solved to obtain the optimal layer fine-tuning strategy, and the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the precision contribution value and the fine-tuning duration. By fine-tuning the pre-trained model, the embodiment of the present application can adapt the pre-trained model to various scenarios, and the fine-tuning strategy is determined using the precision contribution value of each layer of the pre-trained model, which can reduce the amount of training data for fine-tuning the pre-trained model, thereby improving training efficiency and accuracy while reducing computational costs.

[0106] Figure 5 and Figure 6 A comparison chart of the model accuracy and fine-tuning time of the embodiment of the present application and the traditional fine-tuning method is shown. Figure 5 and Figure 6The shaded bar graph in the figure represents the model accuracy or fine-tuning time when each category in the fine-tuning dataset includes 1 picture, and the solid bar graph represents the model accuracy or fine-tuning time when each category in the fine-tuning dataset includes 5 pictures. Figure 5 and Figure 6 It can be seen that the model accuracy of the embodiment of the present application is higher than that of the traditional fine-tuning method, and the fine-tuning time of the embodiment of the present application is shorter than that of the traditional fine-tuning method. Through the embodiment of the present application, the IoT device can directly obtain the optimal layer fine-tuning strategy, which solves the computational efficiency problem of model training on the IoT device, ensures the accuracy and efficiency of model training, and enables fast reasoning of a small amount of data generated by an independent device in a single scene. The embodiment of the present application is suitable for IoT devices with limited resources, and also has the potential to be expanded to other devices.

[0107] Next, a model fine-tuning device provided by the present application is introduced. The model fine-tuning device introduced below and the model fine-tuning method introduced above can refer to each other.

[0108] See also Figure 7 , showing a schematic structural diagram of a model fine-tuning device provided by the present application, the device comprising:

[0109] A testing module 701 is used to test the pre-trained model using a test data set to obtain the classification characteristic value of each layer in the pre-trained model;

[0110] A fine-tuning module 702 is used to fine-tune each layer in the pre-trained model using a fine-tuning data set to obtain an accuracy contribution value of each layer in the pre-trained model and a fine-tuning duration of each layer;

[0111] The solving module 703 is used to solve the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy, wherein the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the accuracy contribution value and the fine-tuning duration.

[0112] In the embodiment of the present application, the fine-tuning module 702 includes:

[0113] An original precision calculation unit, used to perform precision calculation on the pre-trained model to obtain the original precision of the pre-trained model;

[0114] A fine-tuning unit, used to use the fine-tuning data set to fine-tune each layer in the pre-trained model respectively, to obtain multiple fine-tuning models and the fine-tuning duration corresponding to each of the fine-tuning models, wherein different fine-tuning models correspond to different fine-tuning layers;

[0115] A fine-tuning accuracy calculation unit, used to perform accuracy calculation on a plurality of the fine-tuning models to obtain a plurality of fine-tuning accuracies;

[0116] A calculation unit is used to calculate the accuracy contribution value of each layer in the pre-trained model according to the original accuracy of the pre-trained model and the multiple fine-tuning accuracy.

[0117] In the embodiment of the present application, the fine-tuning unit is specifically used for:

[0118] Fine-tune the fine-tuning layer of the pre-trained model using the fine-tuning data set to obtain the fine-tuning model, where the fine-tuning layer is any layer in the pre-trained model;

[0119] Each layer in the pre-trained model is traversed to obtain a plurality of the fine-tuning models.

[0120] In the embodiment of the present application, the solution module 703 is specifically used for:

[0121] Genetic algorithm is used to solve the mathematical model of the optimal layer fine-tuning strategy and obtain the optimal layer fine-tuning strategy.

[0122] In the embodiment of the present application, the test module includes:

[0123] An input unit, used to input the test data set into the pre-trained model, and determine the test results of each layer of the test data set in the pre-trained model;

[0124] A determination unit is used to determine the classification characteristic value of each layer in the pre-trained model according to the test results of each layer in the pre-trained model.

[0125] The embodiment of the present application also provides a computer device, including: a memory and a processor;

[0126] Wherein, the memory is used to store computer programs;

[0127] The processor is used to execute the computer program in the memory to implement the method described in the above method embodiment.

[0128] The embodiment of the present application also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the method described in the above method embodiment.

[0129] In an embodiment of the present application, the testing module uses a test data set to test the pre-trained model to obtain the classification characteristic value of each layer in the pre-trained model; the fine-tuning module uses the fine-tuning data set to fine-tune each layer in the pre-trained model respectively to obtain the precision contribution value of each layer in the pre-trained model and the fine-tuning duration of each layer; the solution module solves the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy, and the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the precision contribution value and the fine-tuning duration. By fine-tuning the pre-trained model, the embodiment of the present application can make the pre-trained model adapt to various scenarios, and the fine-tuning strategy is determined by using the precision contribution value of each layer of the pre-trained model, which can reduce the amount of training data for fine-tuning the pre-trained model, while improving the training efficiency and accuracy, and reducing the computational cost.

[0130] It should be noted that the same or similar parts between the various embodiments can be referred to each other. As for the device embodiment and the system embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0131] For the above-mentioned embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0132] Finally, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0134] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model fine-tuning method, characterized in that: The method comprises: Using a general data set to test the pre-trained model, and obtaining the classification characteristic value of each layer in the pre-trained model; Using the fine-tuning data set, fine-tune each layer in the pre-trained model to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning time of each layer; Solve the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy, wherein the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the accuracy contribution value and the fine-tuning duration.

2. The method according to claim 1, characterized in that The fine-tuning of each layer in the pre-trained model using the fine-tuning data set to obtain the accuracy contribution value of each layer in the pre-trained model and the fine-tuning duration of each layer includes: Performing precision calculation on the pre-trained model to obtain the original precision of the pre-trained model; Using the fine-tuning data set to fine-tune each layer in the pre-trained model, respectively, to obtain multiple fine-tuning models and the fine-tuning duration corresponding to each of the fine-tuning models, wherein different fine-tuning models correspond to different fine-tuning layers; Performing precision calculation on a plurality of the fine-tuning models to obtain a plurality of fine-tuning precisions; According to the original accuracy of the pre-trained model and the multiple fine-tuning accuracy, the accuracy contribution value of each layer in the pre-trained model is calculated.

3. The method according to claim 2, characterized in that The fine-tuning data set is used to fine-tune each layer in the pre-trained model to obtain multiple fine-tuning models, including: Fine-tune the fine-tuning layer of the pre-trained model using the fine-tuning data set to obtain the fine-tuning model, where the fine-tuning layer is any layer in the pre-trained model; Each layer in the pre-trained model is traversed to obtain a plurality of the fine-tuning models.

4. The method according to any one of claims 1 to 3, characterized in that: The method of solving the optimal layer fine-tuning strategy mathematical model to obtain the optimal layer fine-tuning strategy includes: Genetic algorithm is used to solve the mathematical model of the optimal layer fine-tuning strategy and obtain the optimal layer fine-tuning strategy.

5. The method according to any one of claims 1 to 3, characterized in that: The method of testing the pre-trained model using a general data set to obtain the classification characteristic values ​​of each layer in the pre-trained model includes: Inputting the general data set into the pre-trained model, and determining the test results of each layer of the general data set in the pre-trained model; The classification characteristic value of each layer in the pre-trained model is determined according to the test results of each layer in the pre-trained model.

6. A model fine-tuning device, characterized in that: The device comprises: A testing module, used to test the pre-trained model using a general data set to obtain the classification characteristic values ​​of each layer in the pre-trained model; A fine-tuning module is used to fine-tune each layer in the pre-trained model using a fine-tuning data set to obtain an accuracy contribution value of each layer in the pre-trained model and a fine-tuning duration of each layer; A solution module is used to solve the mathematical model of the optimal layer fine-tuning strategy to obtain the optimal layer fine-tuning strategy, wherein the mathematical model of the optimal layer fine-tuning strategy is constructed according to the classification characteristic value, the accuracy contribution value and the fine-tuning duration.

7. The device according to claim 6, characterized in that The fine-tuning module comprises: An original precision calculation unit, used to perform precision calculation on the pre-trained model to obtain the original precision of the pre-trained model; A fine-tuning unit, used to use the fine-tuning data set to fine-tune each layer in the pre-trained model respectively, to obtain multiple fine-tuning models and the fine-tuning duration corresponding to each of the fine-tuning models, wherein different fine-tuning models correspond to different fine-tuning layers; A fine-tuning accuracy calculation unit, used to perform accuracy calculation on a plurality of the fine-tuning models to obtain a plurality of fine-tuning accuracies; A calculation unit is used to calculate the accuracy contribution value of each layer in the pre-trained model according to the original accuracy of the pre-trained model and the multiple fine-tuning accuracy.

8. The device according to claim 7, characterized in that The fine-tuning unit is specifically used for: Fine-tune the fine-tuning layer of the pre-trained model using the fine-tuning data set to obtain the fine-tuning model, where the fine-tuning layer is any layer in the pre-trained model; Each layer in the pre-trained model is traversed to obtain a plurality of the fine-tuning models.

9. A computer device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The device stores instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 5.

Images