A method and a computing device for fine-tuning an image classification model

Abstract

A method and a computing device for fine-tuning an image classification model, obtaining a sample image and a classification label, generating a first prompt distribution according to the sample image by using a prompt generator, randomly sampling a first prompt in the first prompt distribution, splicing an embedding representation of the sample image and the first prompt to obtain a first comprehensive embedding representation, determining a first classification feature corresponding to the first comprehensive embedding representation by using an encoding layer, determining a first classification result distribution corresponding to the first classification feature by using an output layer, and training at least the prompt generator according to the first classification result distribution and the classification label. The pre-trained classification model is fine-tuned in a manner of externalizing the prompt generator. On the one hand, the method can reduce the algorithm consumption of the fine-tuning process. On the other hand, the first prompt distribution generated by the prompt generator can better fit the classification information in the image. Even under the condition of freezing the parameters of the pre-trained model, a good fine-tuning effect can still be achieved.

Description

TECHNICAL FIELD

[0001] The embodiment of the present specification belongs to the technical field of data processing, and particularly relates to a method for fine-tuning an image classification model and a computing device. BACKGROUND

[0002] Nowadays, large pre-trained models have been widely used in the field of image classification. Generally, a pre-trained image classification model is trained using a public dataset containing a large amount of image data, and can complete basic image classification tasks. Therefore, users can further fine-tune the public image recognition model using private domain data, so as to obtain a personalized model that is adapted to their own scene requirements and has higher classification accuracy for specific types of images by consuming only a small amount of training resources.

[0003] However, on the one hand, when the parameter size of the pre-trained model is large, even if only a small amount of sample data is used to fine-tune the pre-trained model, the parameter adjustment still consumes a large amount of computing resources; on the other hand, when the amount of private domain data used by the user for fine-tuning is small, the fine-tuned personalized model is difficult to have a large degree of distinction from the pre-trained model. SUMMARY

[0004] The purpose of the present application is to provide a method for fine-tuning an image classification model and a computing device, comprising:

[0005] The first aspect of the present specification provides a method for fine-tuning an image classification model, which is executed by using a prompt generator and a pre-trained classification model, the pre-trained classification model comprising an encoding layer and an output layer, and the method comprising:

[0006] Obtaining a training sample, the training sample comprising a sample image and a classification label corresponding to the sample image;

[0007] Using the prompt generator, generating a first prompt distribution according to the sample image;

[0008] Randomly sampling in the first prompt distribution to determine a first prompt;

[0009] Concatenating the embedding representation of the sample image with the first prompt to obtain a first comprehensive embedding representation;

[0010] Using the encoding layer, determining a first classification feature corresponding to the first comprehensive embedding representation;

[0011] Using the output layer, determining a first classification result distribution corresponding to the first classification feature;

[0012] According to the first classification result distribution and the classification label, at least the prompt generator is trained.

[0013] The second aspect of the specification provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0014] The method and computing device for fine-tuning an image classification model provided by the embodiments of the specification fine-tune a pre-trained classification model in the manner of an external prompt generator, which can reduce the computing power consumption of the fine-tuning process on the one hand, and the first prompt distribution generated by the prompt generator can better fit the classification information in the image, that is, even under the condition of freezing the parameters of the pre-trained model, a good fine-tuning effect can still be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the specification, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments described in the specification, and those skilled in the art can obtain other drawings according to these drawings without any creative labor.

[0016] Figure 1 is a structural schematic diagram of a fine-tuning system of an image classification model in an embodiment of the specification;

[0017] Figure 2 is a flowchart of a method for fine-tuning an image classification model in an embodiment of the specification;

[0018] Figure 3 is a method schematic diagram for masking the first image feature in an embodiment of the specification;

[0019] Figure 4(A) is a flowchart of a method for image classification using a fine-tuned image classification model in an embodiment of the specification;

[0020] Figure 4(B) is a flowchart of a method for image classification using a fine-tuned image classification model in an embodiment of the specification. DETAILED DESCRIPTION

[0021] In order to make the person skilled in the art better understand the technical solutions in the specification, the technical solutions in the embodiments of the specification will be described clearly and completely in the following with reference to the drawings in the embodiments of the specification. Obviously, the described embodiments are only some embodiments of the specification, not all embodiments. Based on the embodiments in the specification, all other embodiments obtained by those skilled in the art without any creative labor should be within the scope of protection of the specification.

[0022] Generally, in the process of fine-tuning a pre-trained model, users will choose to freeze part of the model parameters to save the computational overhead of fine-tuning, and only adjust part of the model layers of the pre-trained model. However, when the parameter size of the pre-trained model is large, adjusting part of the model layers of the pre-trained model is still a large overhead. Visual prompt tuning (VPT) can significantly reduce the storage cost in the fine-tuning process by introducing a small number of trainable prompt parameters (usually less than 1% of the model parameters) in the input space of the pre-trained model while keeping the backbone network of the pre-trained model frozen. However, how to make the introduced prompt parameters have stronger generalization is still a problem to be solved.

[0023] Figure 1 A structure diagram of a fine-tuning system of an image classification model is provided for the present specification, which can be deployed in a computing device. As shown in the figure, the fine-tuning system can include a feature extraction module, a prompt generator, an image processing module, a pre-trained classification model, and a training module. Among them, the feature extraction module can be used to extract the first image features of the sample image; the prompt generator can be used to determine the corresponding first prompt distribution according to the first image features output by the feature extraction module; the image processing module can be used to convert the input sample image into an embedded representation, and sample the first prompt from the first prompt distribution output by the prompt generator to obtain the first prompt. The first comprehensive embedded representation is obtained by splicing the first prompt and the embedded representation of the sample image; the pre-trained classification model can output the first classification result distribution corresponding to the image according to the input first comprehensive embedded representation; the training module can adjust at least the parameters of the prompt generator according to the first classification result distribution and the classification label corresponding to the sample image using a pre-set target function.

[0024] In some implementations, the feature extraction module can be a deep learning model, such as a convolutional neural network; it can also be a computing module with built-in feature extraction algorithms, such as the Histogram of Oriented Gradient (HOG) feature extraction method or the Scale-Invariant Feature Transform (SIFT) algorithm, which is not limited in the present specification. The prompt generator can be a deep learning model, such as a convolutional neural network, it should be noted that the output space of the feature extraction module is the same as the input space of the prompt generator. The pre-trained classification model can be a pre-trained Vision Transformer (ViT) model, which can include an encoding layer and an output layer. The encoding layer can extract the corresponding classification features according to the embedded representation of the input image, and the output layer can further extract the corresponding classification results according to the classification features.

[0025] Figure 2 A flow of a method of fine-tuning an image classification model in the specification is shown, comprising:

[0026] S201: Obtain a training sample, the training sample comprising a sample image and a classification label corresponding to the sample image.

[0027] First, the computing device deployed with the fine-tuning system can obtain the training sample from the pre-stored device memory or other communicable database, which can store a training sample set consisting of a plurality of training samples. One training sample can include a sample image and a classification label corresponding to the sample image.

[0028] In some implementations, in the application scenario of personalized federated learning, each user can obtain a training sample from the respective private training sample set using the method as shown in Figure 2 , based on the same pre-trained classification model, fine-tune to obtain a personalized image classification model adapted to the characteristics of the own training sample set.

[0029] S203: Using the prompt generator, generate a first prompt distribution according to the sample image.

[0030] After obtaining the training image, the probability distribution of the classification information corresponding to the sample image, i.e., the first prompt distribution, can be generated according to the sample image.

[0031] Specifically, the fine-tuning system can first encode the sample image using the feature extraction module to determine the first image feature corresponding to the sample; then input the first image feature into the prompt generator to generate the first prompt distribution corresponding to the sample image using the prompt generator. The first prompt distribution is the probability distribution of the prompt information corresponding to the sample image.

[0032] It should be noted that, since in the subsequent step S207, the first prompt needs to be spliced with the embedded representation of the sample image, and in step S209, the first comprehensive embedded representation obtained by splicing the first prompt and the embedded representation of the sample image is input into the encoding layer of the pre-trained classification model. Therefore, the dimension of the first prompt needs to be adapted to the input space of the encoding layer. For example, when the input space of the encoding layer is a one-dimensional vector with a hidden layer dimension of d, the first prompt can be a one-dimensional vector with a hidden layer dimension of d.

[0033] On the other hand, the user can pre-set a target profile of the probability distribution generated by the hint generator. Thus, in step S203, the output of the hint generator is the parameter of the target profile. For example, when the target profile is a Gaussian distribution, the output of the hint generator can be the mean and variance of the Gaussian distribution; when the target profile is a uniform distribution, the output of the hint generator can be the interval endpoints of the uniform distribution. Wherein, the specific type of the target profile is not limited in the present specification.

[0034] S205: Randomly sampling in the first hint distribution to determine the first hint.

[0035] After determining the first hint distribution, the image processing module in the fine-tuning system can randomly sample in the first hint distribution to determine the first hint according to the sampling result. Specifically, one random sampling can be performed in the first hint distribution, and the sampling result of the random sampling can be directly taken as the first hint; or several random samplings can be performed in the first hint distribution, and the first hint can be further determined according to the sampling results of the several random samplings, which is not limited in the present specification.

[0036] Thus, compared with directly generating the first hint according to the sample image, in the process of determining the first hint by steps S203-S205, the idea of Bayesian optimization is used, the probability distribution accurately representing the classification information of the sample image is regarded as the target distribution, the process of generating the first hint distribution by the hint generator and then sampling the first hint according to the first hint distribution is regarded as the approximate fitting of the process of sampling in the target distribution. When the appropriate target profile is pre-determined, compared with directly generating the first hint, the classification information of the sample image has stronger fitting ability and expression ability. On the other hand, the process of determining the first hint in step S205 has certain randomness, which can reduce the risk of overfitting in the subsequent training process.

[0037] It should be noted that although the first hint contains classification information in the sample image, the classification information in the first hint can be hidden features, without corresponding text or images with explicit semantics.

[0038] S207: Concatenating the embedding representation of the sample image with the first hint to obtain a first comprehensive embedding representation.

[0039] Between performing step S207, the fine-tuning system can pre-determine the embedding representation of the sample image according to the input sample image, so that after determining the first hint, the embedding representation of the sample image can be concatenated with the first hint to obtain the first comprehensive embedding representation.

[0040] Specifically, when the pre-training classification model is a ViT model, the embedding representation corresponding to the sample image can be obtained by dividing the sample image into image blocks of a preset size, then mapping the pixel points of each image block to a one-dimensional vector with a hidden layer dimension d, and finally concatenating the one-dimensional vectors obtained by the mapping.

[0041] It should be noted that when the pre-training classification model is a ViT model, the embedding representation corresponding to the sample image can be a one-dimensional vector with a hidden layer dimension d and a length a. Correspondingly, the first prompt can be a one-dimensional vector with a hidden layer dimension d and a length b. By concatenating the embedding representation of the sample image and the first prompt, a first comprehensive embedding representation with a hidden layer dimension d and a length a+b can be obtained.

[0042] S209: Determine the first classification feature corresponding to the first comprehensive embedding representation using the encoding layer.

[0043] After determining the first comprehensive embedding representation, the fine-tuning system can process the first comprehensive embedding representation using the encoding layer of the pre-training classification model. The output of the encoding layer is the first classification feature corresponding to the first comprehensive embedding representation.

[0044] Since the first comprehensive embedding representation is obtained by concatenating the first prompt and the embedding representation of the sample image, when the prompt generator has the ability to extract classification information in the sample image, the weight of the classification information in the first comprehensive embedding representation is greater than the weight of the classification information in the embedding representation of the sample image. Further, when the encoding layer is the same, compared to feature extraction on the embedding representation of the sample image, the classification information in the input of the encoding layer should also have a higher weight.

[0045] In some implementations, when the pre-training classification model is a ViT model, the encoding layer can include several transformer layers.

[0046] S211: Determine the first classification result distribution corresponding to the first classification feature using the output layer.

[0047] After determining the first classification feature, the fine-tuning system can continue to input the first classification feature into the output layer of the pre-training classification model. The output layer can reduce the dimension of the first classification feature and map the reduced first classification feature to a first classification result distribution. The first classification result distribution can be a probability distribution of each classification category corresponding to the sample image.

[0048] In some implementations, when the pre-trained classification model is a ViT model, the output layer can include several fully connected layers, and an activation layer can be included between any adjacent fully connected layers.

[0049] S213: At least train the prompt generator according to the first classification result distribution and the classification label.

[0050] After determining the first classification result distribution, the fine-tuning system can input the classification label corresponding to the sample image and the first classification result distribution into a training module, so that the training module determines the posterior distribution of the to-be-adjusted parameters according to the preset target function, according to the current first classification result part and the classification label, and then adjusts the parameters of the prompt generator.

[0051] Specifically, the idea of Bayesian optimization can be used:

[0052]

[0053] where X represents the observation data, i.e., the sample image and the label corresponding to the sample image; Z represents the latent variable, including the adjustable parameters and the non-adjustable parameters of the model, wherein the adjustable parameters at least include the parameters of the prompt generator; P(Z) represents the prior distribution of the latent variable, which can be determined according to the parameters of the latent variable in the model before step S213 is performed; P(X|Z) represents the probability distribution of the observation data under the current latent variable, and similarly, it can be directly determined according to the parameters of the latent variable in the model; P(X) represents the marginal probability distribution of the observation data, which can be determined according to P(X) = ∫P(Z,X)dψ = ∫P(X|Z)·P(Z)dψ.

[0054] The optimization goal is to determine the adjustable parameters in the posterior distribution P(Z|X) of the latent variable.

[0055] Specifically, after P(Z), P(X|Z) and P(X) are determined, the posterior distribution P(Z|X) of the latent variable can be directly determined, and further, the parameters of the prompt generator can be adjusted according to P(Z|X); or the prompt generator parameter, the expected distribution Q(Z), can be set, and further, the above determination so that the difference between Q(Z) and P(Z|X) is minimized, and the prompt generator is at least adjusted. The method of training the prompt generator is not limited in the specification.

[0056] Since the structure of the prompt generator is independent of the pre-trained classification model, its parameter size is typically much smaller than that of each model layer in the pre-trained classification model. Therefore, adjusting the parameters of the prompt generator requires far less computational power than adjusting the native model layers in the pre-trained classification model. Furthermore, by introducing the influence of the prompt generator through concatenation of the first prompt with the embedded representation of the sample image, even with a small parameter size, the prompt generator can still have a significant impact on the distribution of the first classification result.

[0057] like Figure 2 The method shown here is to fine-tune an image classification model by using an external cue generator to fine-tune the pre-trained classification model. On the one hand, this can reduce the computational cost of the fine-tuning process. On the other hand, the first cue distribution generated by the cue generator can better fit the classification information in the image. Even when the parameters of the pre-trained model are frozen, good fine-tuning results can still be achieved.

[0058] In some implementations, such as Figure 2 In step S203 shown, the sample image is encoded, the first image feature corresponding to the sample image is determined, the first image feature is randomly masked, the first mask feature corresponding to the first image feature is determined, and the first mask feature is input into the prompt generator to generate a first prompt distribution corresponding to the sample image.

[0059] Specifically, the process of masking the features of the first image can be as follows: Figure 3 As shown. First, a first mask adapted to the dimension of the first image feature is randomly generated. This first mask is then multiplied bit-by-bit with the first image feature to determine the first mask feature corresponding to the first image feature. Each element in the first mask can include either 0 or 1 elements, such as... Figure 3 As shown, black squares in the image represent elements with a value of 0, and white squares represent elements with a value of 1. After the first image feature is multiplied bitwise with the first mask, the elements corresponding to the 0 elements are masked, while the elements corresponding to the 1 elements are preserved.

[0060] In some implementations, a Bernoulli distribution with predetermined parameters can be sampled to generate a first mask that is adapted to the dimension of the first image features.

[0061] Therefore, for the same sample image, the first mask features obtained when training with the same sample image in different rounds are not the same. On the other hand, to avoid the prompt generator from falling into overfitting due to excessive reliance on image features corresponding to certain positions, the robustness of the trained prompt generator can be improved.

[0062] In some implementations, the first prompt distribution is a Gaussian distribution, and in step S203 as shown in Figure 3 a mean and a variance of the first prompt distribution are generated.

[0063] When the first prompt distribution is a Gaussian distribution, the first prompt distribution can be uniquely represented by the determined mean and variance.

[0064] In some implementations, in step S203 as shown in Figure 2 a plurality of initial prompt distributions are generated according to the sample image, and a plurality of first prompt distributions are determined according to the plurality of initial prompt distributions; and in step S205 as shown in Figure 2 an initial prompt corresponding to each first prompt distribution is determined by randomly sampling in each first prompt distribution, and the first prompt is determined according to the initial prompts.

[0065] wherein the number of the initial prompt distributions is irrelevant to the number of the first prompt distributions, and each first prompt distribution is determined according to a weighted sum of parameters of the initial prompt distributions by using different weights.

[0066] When the initial prompt distributions are Gaussian distributions, the first prompt distribution is also a Gaussian distribution.

[0067] Further, each initial prompt can be randomly sampled in the first prompt distribution to determine the first prompt, and the first prompt can be represented as a weighted sum of the initial prompts.

[0068] As described above, the target distribution is the true distribution of the classification information of the sample image, and is also the optimization target of the process of determining the first prompt by the prompt generator. Generally, the actual distribution function of the target distribution is irregular and complex. By the method of generating the first prompt distribution by the prompt generator and then sampling the first prompt in the first prompt distribution, the distribution of the first prompt can be extended beyond the exponential distribution family, so that the process has stronger fitting ability for the target distribution. Further, the trained prompt generator can more accurately generate prompt information containing classification information.

[0069] In some implementations, in step S207 as shown in Figure 2 the embedding representation of the sample image, the first prompt, and the classifier are spliced.

[0070] When the pre-trained classification model is a ViT model, the output space of the encoding layer is generally the same as the input space dimension of the encoding layer, that is, when the input is a vector of length n, the output is also a vector of length n.

[0071] Therefore, when concatenating the first comprehensive embedding representation, the embedding representation of the sample image, the first prompt, and the classifier can be concatenated. In each first comprehensive embedding representation corresponding to different sample images, each classifier is located in the same position. Consequently, in the output of the encoding layer, the elements corresponding to each classifier are also located in the same position. Thus, the element corresponding to the classifier in the output can be extracted as the first classification feature, thereby eliminating the need to input the complete output of the encoding layer into subsequent output layers.

[0072] In some implementations, such as Figure 2 In step S213 shown, the prompt generator and the output layer are trained according to the first classification result distribution and the classification label.

[0073] Based on the previous description of step S213, when it is necessary to train the output layer, the parameters of the output layer are regarded as part of the latent variable Z. Using the method corresponding to step S213, the posterior distribution of the latent variable is determined, and the parameters of the output layer can be adjusted.

[0074] Typically, the parameter size of the output layer is much smaller than that of the encoding layer. In step S213, the parameters of the output layer are also adjusted based on the adjustment of the parameters of the prompt generator. This can further improve the matching degree between the model and the training samples, improve the classification effect of the fine-tuned pre-trained classification model, and at the same time, it does not consume too much computing power.

[0075] In some implementations, such as Figure 2 In step S213 shown, the prompt generator is trained at least to maximize the preset objective function based on the first classification result distribution and the classification label. The objective function represents the lower bound of evidence in the observed data, and the observed data consists of the sample image and the classification label.

[0076] According to the aforementioned formula (1), it is usually difficult to directly determine P(Z|X) during the training of the model - it requires a lot of computing power to determine the step of P(X)=∫P(Z,X)dψ=∫P(X|Z)·P(Z)dψ (when there are too many latent variables, that is, model parameters, integrating the value of each model parameter requires a lot of computing power).

[0077] Therefore, the expected distribution can be pre-defined. Using the lower bound of evidence from observational data to measure The difference from the true posterior distribution Pθ(X,Z) is considered to be maximized when the lower bound of evidence reaches its maximum value. The difference with Pθ(X,Z) is minimized, according to the minimum difference The parameters of the corresponding latent variables adjust the parameters of the cue generator, allowing the cue generator to accurately generate classification information corresponding to any image as a cue for that image. It should be noted that in Bayesian inference, we typically need to maximize the log-likelihood function logPθ(X) of the observed data X to adjust the model parameters. Maximizing the log-likelihood function also means determining the model parameters θ that maximize the probability of the current observed data X. The log-likelihood function is usually expressed using the following formula:

[0078]

[0079] The previous text explained the difficulty of integrating Pθ(X,Z), and utilized... Fitting Pθ(Z|X) and substituting it into formula (2) yields:

[0080]

[0081] again but Therefore, This is called the lower bound of evidence. Only when this lower bound of evidence is maximized can the log-likelihood function logPθ(X) of the observed data X reach its maximum value.

[0082] Specifically, X represents the sample image, and Y represents the classification label. This indicates a prompt message. The average value of the prompt message. The variance of the prompt message, the prompt message probability distribution When S initial cue distributions ψ1,…,ψS are generated based on the sample image, and j first cue distributions ψ1,…,ψJ are determined, then j initial cue distributions are sampled for each first cue distribution. At that time, the objective function It can be represented as:

[0083]

[0084] Where qφ(ψ) represents the mixed distribution, which corresponds to the target parameters of the generator; q(V|ψ,X)q(ψ|X)=

[0085] q(V,ψ|X) represents the joint probability distribution of the parameters of the first prompt distribution based on the sample image and the initial prompt, which can be determined according to the preset parameters of the prompt generator and the aforementioned process of determining the initial prompt; This represents the joint distribution of classification labels and initial prompts based on sample images, where... represents a probability distribution of the initial prompt based on the classification result of the sample image, since the sample image and the first prompt (determined according to the initial prompt) are input data of the pre-trained model, may be determined by the current parameters of the pre-trained model, represents a distribution of the initial prompt based on the sample image, since the initial prompt is randomly sampled from the first prompt distribution determined according to the sample image generated by the prompt generator, may be determined according to the current parameters of the prompt generator; is a probability distribution of the initial prompt based on the first prompt distribution, after the first prompt distribution is determined, the distribution corresponding to the initial prompt can be determined; is a joint distribution of the initial prompt distribution and the initial prompt, the joint distribution can be determined according to the target parameters of the prompt generator and the step of sampling the initial prompt in the first prompt distribution determined according to the initial prompt distribution.

[0086] Therefore, the parameters of the prompt generator that can make the loss function increase can be determined by the stochastic gradient ascent method as the updated parameters of the prompt generator.

[0087] FIG. 4(A) shows a flowchart of an image classification method provided by the present specification, which uses a fine-tuned image classification model, and the method is performed by using a prompt generator and a pre-trained classification model including an encoding layer and an output layer, and the method includes:

[0088] Step S401A: obtaining an image to be classified.

[0089] Step S403A: using the prompt generator, generating a mean of the second prompt distribution as a second prompt corresponding to the image to be classified according to the image to be classified.

[0090] Specifically, the second prompt distribution can be generated according to the description corresponding to step S203. However, after the mean of the second prompt distribution is generated, sampling is not continued, and the mean of the second prompt distribution is directly used as the second prompt corresponding to the image to be classified

[0091] It should be noted that step S403A only needs to generate one second prompt for the image to be classified.

[0092] Step S405A: determining a classification result of the image to be classified according to an embedding representation of the image to be classified and the second prompt.

[0093] Specifically, the second classification result distribution of the to-be-classified image can be determined according to the embedding representation of the to-be-classified image and the second prompt, and a classification category corresponding to the highest probability in the second classification result distribution can be taken as the classification result corresponding to the to-be-classified image.

[0094] FIG. 4(B) also shows a flowchart of an image classification method provided in the specification, which utilizes a fine-tuned image classification model, and the method is performed by using a prompt generator and a pre-trained classification model including an encoding layer and an output layer, and the method includes:

[0095] Step S401B: obtaining a to-be-classified image.

[0096] Step S401B is the same as step S401A, and will not be repeated here.

[0097] Step S403B: encoding the to-be-classified image to determine a second image feature corresponding to the to-be-classified image.

[0098] Specifically, refer to the corresponding description of S203, which will not be repeated here.

[0099] Step S405B: performing n times of random masking on the second image feature to determine n second masked features corresponding to the image feature.

[0100] The method of performing random masking on the second image feature can refer to the method of masking the first image feature described above. Of course, the parameters of the Bernoulli distribution used when performing random masking on the second image feature, i.e., the probabilities of the appearance of 0 elements and 1 elements in the mask, can be different from the parameters of the Bernoulli distribution used when masking the first image feature.

[0101] Thus, according to one second image feature, several different second masked features can be obtained.

[0102] Step S407B: inputting the n second masked features into the prompt generator respectively to generate the mean of n second prompt distributions corresponding to the to-be-classified image as n second prompts corresponding to the to-be-classified image.

[0103] Specifically, for any second masked feature, refer to step S207 to determine the second prompt corresponding to the second masked feature, thereby determining n second prompts.

[0104] Step S409B: concatenating the n second prompts with the embedding representation of the to-be-classified image respectively to obtain n second comprehensive embedding representations.

[0105] Specifically, for any second prompt, the second comprehensive embedding representation corresponding to the second prompt can be determined by referring to step S209, and n second comprehensive embedding representations are determined.

[0106] Step S411B: determining n second classification result distributions respectively corresponding to the n second comprehensive embedding representations.

[0107] Specifically, for any second comprehensive embedding representation, the second classification result distribution corresponding to the second comprehensive embedding representation can be determined by referring to step S211.

[0108] Step S413B: determining the classification result of the image to be classified according to the n second classification result distributions.

[0109] After determining each second classification result distribution, the total probability and the highest classification category can be determined as the classification result of the image to be classified according to the probability of each classification category in each second classification result distribution.

[0110] Since each second prompt is generated according to different mask features, and further, step S411B determines the second classification result distribution corresponding to each second prompt, the influence of a specific position in the image to be classified can be reduced, and on the other hand, the effect of repeated experiments can be achieved, thereby improving the accuracy of the classification result of the image to be classified.

[0111] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0112] The controller can be implemented in any suitable way, for example, the controller can take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code form, the controller can perfectly well be implemented to perform the same functions in the form of logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of logical programming of the method steps. Such a controller can thus be considered a hardware component, and the means comprised therein for performing the various functions can also be considered structures within the hardware component. Alternatively, or even, the means for performing the various functions can be considered both software modules implementing the method and structures within the hardware component.

[0113] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not rule out that with the development of future computer technology, computers implementing the functions of the above embodiments can be personal computers, laptop computers, vehicle human-computer interaction devices, cellular phones, camera phones, smart phones, personal digital assistants, media players, navigation devices, email devices, game consoles, tablet computers, wearable devices, or combinations of any of these devices.

[0114] Although the method operations of the embodiments of the present disclosure are described in a particular, sequential order, one or more of the method operations can be omitted, or the method operations can be performed in an order other than the described order. Additionally, one or more of the method operations can be performed concurrently, or with partial concurrence. Furthermore, one or more of the method operations can be performed by different entities, or over different time periods. The term "including" as used herein is intended to mean "comprising," such that the process, method, article, or apparatus that includes elements in addition to those specified. As used in this description, the term "coupled" means a direct or indirect connection, which can be physical or logical. The term "coupled" does not relate to a direct connection or wiring.

[0115] For the sake of description, the above-described apparatus is described as various modules to describe the functions of the apparatus. Of course, when implementing one or more embodiments of the present disclosure, the functions of the modules can be implemented in one or more software and / or hardware, or the modules implementing the same functions can be combined into a plurality of sub-modules or sub-units. The apparatus embodiments described above are merely illustrative, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0116] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The term "coupled" is intended to mean a direct or indirect connection, which can be physical or logical. Figure 1 The term "coupled" is intended to mean a direct or indirect connection, which can be physical or logical.

[0117] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0119] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0120] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0121] Computer readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, graphene memory or other magnetic storage devices, or any other non-transmission medium that can be used to store information that is accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0122] Those skilled in the art will appreciate that the one or more embodiments described herein can be provided as a method, system or computer program product. Accordingly, the one or more embodiments described herein can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the one or more embodiments described herein can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable code.

[0123] The one or more embodiments described herein can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The one or more embodiments described herein can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0124] The various embodiments described in this specification are described in the context of progressive embodiments, with each embodiment building on the previous one. The same or similar parts between embodiments are cross-referenced as appropriate. Each embodiment focuses on the differences between that embodiment and the previous one. In particular, the system embodiments are described relatively simply, as they are substantially similar to the method embodiments. In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific feature, structure, material or characteristic being described in connection with that embodiment or example is included in at least one embodiment or example in the specification. Illustrative representations of the above terms in the specification are not necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics being described can be combined in any suitable manner in one or more embodiments or examples. Furthermore, the skilled person can combine and combine features of different embodiments or examples and characteristics of different embodiments or examples in a manner that is not mutually contradictory.

[0125] The above only describes embodiments of the one or more embodiments described in this specification and is not intended to limit the one or more embodiments described in this specification. The one or more embodiments described in this specification can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the specification shall be included in the scope of the claims.

Claims

1. A method for fine-tuning an image classification model, the method being executed using a cue generator and a pre-trained classification model, the pre-trained classification model comprising an encoding layer and an output layer, the method comprising: Obtain training samples, which include sample images and classification labels corresponding to the sample images; Using the cue generator, a first cue distribution is generated based on the sample image; Randomly sample from the first prompt distribution to determine the first prompt; The embedded representation of the sample image is concatenated with the first prompt to obtain the first comprehensive embedded representation; Using the coding layer, the first classification feature corresponding to the first comprehensive embedding representation is determined; Using the output layer, the distribution of the first classification result corresponding to the first classification feature is determined; Based on the distribution of the first classification results and the classification labels, at least the prompt generator is trained.

2. The method as described in claim 1, wherein the cue generator generates a first cue distribution based on the sample image, specifically comprising: The sample image is encoded to determine the first image feature corresponding to the sample image; Randomly mask the first image features to determine the first mask features corresponding to the first image features; The first mask feature is input into the cue generator to generate a first cue distribution corresponding to the sample image.

3. The method as described in claim 1, wherein the first prompt distribution is a Gaussian distribution; The generation of the first hint distribution specifically includes: Generate the mean and variance of the first hint distribution.

4. The method as described in claim 1, wherein generating a first cue distribution based on the sample image specifically includes: Several initial cue distributions are generated based on the sample images; Based on the aforementioned initial prompt distributions, several first prompt distributions are determined; Randomly sampling from the first prompt distribution to determine the first prompt specifically includes: Randomly sample from each first prompt distribution to determine the initial prompt corresponding to each first prompt distribution; Based on the initial prompts, determine the first prompt.

5. The method as described in claim 1, wherein concatenating the embedded representation of the sample image with the first prompt, specifically includes: The embedded representation of the sample image, the first prompt, and the classifier are concatenated.

6. The method as described in claim 1, wherein training the prompt generator based on the first classification result distribution and the classification label, specifically includes: The prompt generator and the output layer are trained based on the distribution of the first classification result and the classification label.

7. The method of claim 3, wherein training the prompt generator based on the first classification result distribution and the classification label, specifically includes: Based on the first classification result distribution and the classification label, the prompt generator is trained at least to maximize the preset objective function, wherein the objective function represents the lower bound of evidence in the observed data, and the observed data consists of the sample image and the classification label.

8. The method of claim 3, further comprising: Obtain the image to be classified; Using the aforementioned cue generator, the mean of a second cue distribution is generated based on the image to be classified, and this mean is used as the second cue corresponding to the image to be classified. The classification result of the image to be classified is determined based on the embedding representation of the image to be classified and the second cue.

9. The method of claim 8, wherein the cue generator generates the mean of a second cue distribution based on the image to be classified, as a second cue corresponding to the image to be classified, specifically comprising: The image to be classified is encoded to determine the second image feature corresponding to the image to be classified. The second image feature is randomly masked n times to determine n second mask features corresponding to the image feature; The n second mask features are respectively input into the prompt generator to generate the mean of the n second prompt distributions corresponding to the image to be classified, which are used as the n second prompts corresponding to the image to be classified. Based on the embedding representation of the image to be classified and the second cue, the classification result of the image to be classified is determined, specifically including: The n second hints are concatenated with the embedding representation of the image to be classified to obtain n second comprehensive embedding representations; Determine the distribution of the n second classification results corresponding to the n second comprehensive embedding representations; Based on the distribution of the n second classification results, the classification result of the image to be classified is determined.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images