An image recognition method based on debiased variational inference hint tuning

Abstract

The present invention discloses an image recognition method based on debiased variational inference prompt tuning, belonging to the field of artificial intelligence and computer vision. The present invention uses CLIP as a benchmark framework, which includes an image branch and a text branch; based on the image branch, the present invention designs an image encoder based on style perturbation to obtain debiased visual features; based on the debiased visual features and the text branch, a text encoder based on debiased variational inference is designed to obtain debiased text features; and through a designed loss function based on variational inference, the training set data is optimized by minimizing the model parameters through the loss function based on variational inference to obtain a model with optimal parameters; the test image data is input into the model with optimal parameters to obtain the image recognition result. The present invention provides a general small sample image recognition method for data scarce or limited scenarios, which can still maintain a high recognition accuracy and generalization performance under limited data.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision, and in particular relates to an image recognition method based on debiased variational inference prompt tuning. Background Art

[0002] Vision-Language Pre-trained Models (VLPMs) have strong performance in open-set visual concept understanding and zero-shot learning, and have been widely used in downstream scenarios with scarce or limited data. Among them, Contrastive Language-Image Pre-Training (CLIP) is one of the basic frameworks of mainstream VLPMs. However, since CLIP is based on a large number of public natural scene datasets to implement model training, mainstream models face domain shift problems when applied in specific fields, that is, there is a distribution shift between downstream scene data and pre-training data, which limits the generalization performance of VLPMs. Prompt Tuning (PT), as an efficient method to guide the model to adapt to specific tasks by adjusting or learning the input prompt, has become one of the mainstream means to migrate CLIP to downstream scenarios. That is: without changing the parameters of VLPMs, the performance of CLIP on specific tasks can be improved by adjusting or learning the prompt. Although PT has achieved remarkable results in small-sample downstream tasks, its generalization performance is still limited by two aspects:

[0003] First, the rich contextual information of the image forces the learnable text prompt to learn a specific task description, which hinders the generalization ability of the visual language model to new categories. For example, when the training data uses image data of airplanes parked on the ground, CLIP can easily learn the specific prompt of "This is an airplane parked on the grass." However, when the test data contains data of airplanes operating in the air, the model is prone to poor generalization performance due to contextual semantic deviations from the training data. Second, the prompt tuning of the mainstream visual language model is based on biased point estimation to optimize model parameters, which hinders the generalization ability of the visual language model to unseen prompts. Specifically, existing methods usually regard PT as a point estimation problem and rely on empirical risk minimization to optimize parameters, which leads to overfitting of the model to task-specific descriptions, ignoring the bias and uncertainty in the data, resulting in limited generalization ability of the model in scenarios with distribution shift.

[0004] In view of the above problems, the present invention invents an image recognition method based on debiased variational inference prompt tuning, aiming to provide a set of universal small sample learning methods for scenarios such as data scarcity or data limitation, and to improve the generalization performance of visual language models. Summary of the invention

[0005] The present invention provides an image recognition method based on debiased variational inference prompt tuning to solve the technical problem of poor generalization performance of CLIP in the prior art.

[0006] The present invention proposes an image recognition method based on debiased variational inference prompt tuning, which specifically includes the following steps:

[0007] Step S1: Split the paired image-text FGVC-Aircraft dataset into a training set and a test set, where the training set is randomly selected from each category The remaining paired image-text labels are used as the test set;

[0008] Step S2: using the contrastive language-image pre-training model as a benchmark framework, the benchmark framework includes an image branch and a text branch, the image branch includes an image block encoding module, an image encoder, and an ImageProj function; the text branch includes a text word embedding module, a text encoder, and a TextProj function;

[0009] Step S3: Based on the image branch of the reference framework, a style perturbation-based image encoder is designed, and two image data in the training set of step S1 are selected and then input into the style perturbation-based image encoder, where one image data is used as training image data and the other image data is used as auxiliary image data to obtain debiased visual features;

[0010] Step S4: Based on the text branch of the reference framework, a text encoder based on debiased variational inference is designed; text labels paired with the training image data selected in step S3 are selected from the training set, and the debiased visual features obtained in step S3 and the paired text labels are input into the text encoder based on debiased variational inference to obtain debiased text features;

[0011] Step S5: Randomly select The training set data of categories, select one corresponding image and text label for each category, and execute steps S3 and S4 to obtain The debiased visual features and debiased text features of the group are combined; a loss function based on variational inference is designed, and positive sample pairs and negative sample pairs are selected to input the loss function based on variational inference to complete the update of network parameters; after all samples in the training set constructed in step S1 participate in the network parameter update, the optimal model parameters and the debiased text features of each category are output to obtain a model with optimal parameters;

[0012] Step S6: Input the test set image data into the model with the optimal parameters after optimization in step S5, obtain the debiased visual features through the image encoder based on style perturbation, and obtain the image recognition results by calculating the highest cosine similarity between the debiased visual features and the debiased text features of all categories.

[0013] Furthermore, the style perturbation-based image encoder is constructed by performing a forward pass on the image encoder of the reference frame image branch. The style perturbation operation is performed on the layer.

[0014] Further, step S3 includes the following steps:

[0015] Step S31: compress the training image data to a length of , width The compressed image data is then cropped into The size is The image block, , and The image blocks are input into the image block encoding module to obtain the 0th layer visual token of the training image data. , ; Among them, the visual token has a total of Visual sub-tokens, Indicates the number of dimensions of each visual sub-token, , Indicates each The number of channels of the image block; similarly, the auxiliary image data is compressed and cropped to obtain the 0th layer visual token of the auxiliary image data , ;

[0016] Step S32: Visual tokens at level 0 of the training image data Introducing a learnable class word token , get the visual token with category name of the training image data layer 0 [ c 0 , E 0 train ] ;

[0017] Step S33: Visual tokens with category names in layer 0 of the training image data [ c 0 , E 0 train ] The first layer of the Transformer network layer inputs the image encoder and outputs the first layer of visual tokens with category names for the training image data, represented as [ c 1 , E 1 train ] . is the first-level category word token, Visual tokens for the first layer of training image data; ; Add auxiliary image data layer 0 visual token Input image encoder layer 1 Transformer network layer, output auxiliary image data layer 1 visual token ;

[0018] Step S34: The front of the image encoder The visual tokens of the training image data in the visual tokens with category names output by each layer of the Transformer network layer are subjected to style perturbation operations to obtain visual tokens after the perturbation of the training image data;

[0019] Step S35: Image encoder To The training image data output by each layer of the Transformer network layer carries the visual token of the category name, which is directly input into the next Transformer network layer to obtain the training image data of the next layer with the visual token of the category name; finally, the training image data of the first layer is obtained. Layers with class name visual tokens ;

[0020] Step S36: training image data Layers with class-named visual tokens Medium Class Token Input the ImageProj function in the benchmark frame of step S2 to obtain the debiased visual features , Indicates the dimension size of the debiased visual features.

[0021] Furthermore, the style perturbation operation includes the following steps:

[0022] Step S341: The image encoder The training image data output by the Transformer network layer Layer Visual Token , flattened into training image data Layer Flatten Visual Tokens , ;

[0023] Step S342: Calculate the training image data Layer Flatten Visual Tokens The style features include the mean of the training image data and the training image data variance ;

[0024] Step S343: The auxiliary image data Layer Visual Token After the same flattening and style feature calculation operations as the training image data, the mean value of the auxiliary image data is obtained. and auxiliary image data variance ;

[0025] Step S344: average the training image data , training image data variance , auxiliary image data mean and auxiliary image data variance Perform random convex weight combination calculation to obtain the mixed mean after style perturbation and the mixed variance after style perturbation ;

[0026] Step S345: Obtain the flattened visual token after disturbance using the style transformation formula , ;

[0027] Step S346: Flatten the disturbed visual tokens The dimension is reset to obtain the training image data Visual tokens after layer perturbation ;in, ;

[0028] Step S347: Visual tokens after layer perturbation With Layer Classifier Token Combined to get the training image data Layer visual tokens with class names [ c i , E i Perturbation ] ;

[0029] Step S348: The training image data Layer visual tokens with class names [ c i , E i Perturbation ] Input image encoder Transformer network layer, output training image data Layers with class-named visual tokens [ c i + 1 , E i + 1 train ] ; For the training image data Layer Visual Tokens, For the The class word token of the layer, Represents the image encoder Transformer network layer; the auxiliary image data Layer Visual Token Input image encoder Transformer network layer, get auxiliary image data Layer visual tokens, represented as .

[0030] Furthermore, the text encoder based on debiased variational inference obtains debiased visual cue tokens and probability distribution tokens by inputting the debiased visual features into the meta-learning network, fuses the debiased visual cue tokens and probability distribution tokens with the text tokens output by the text word embedding module, and then executes them according to the text branch of the benchmark framework.

[0031] Further, step S4 includes the following steps:

[0032] Step S41: Select the text label in the training set that is paired with the training image data selected in step S3, and input the text label into the text word embedding module to obtain the 0th layer text token ;

[0033] Step S42: Debiasing the visual features obtained in step S3 Input the meta-learning network to get the debiased visual cue token and probability distribution tokens ;

[0034] Step S43: Debiasing visual cue tokens and probability distribution tokens Generate latent variables using reparameterization techniques ;

[0035] Step S44: The layer 0 text token obtained in step S41 and the hidden variable obtained in step S43 Fusion obtains text tokens with introduced probability distribution ;

[0036] Step S45: The text tokens obtained in step S44 are introduced into the probability distribution Input the text encoder to get the final text representation ; Represent the text The last subtext representation in Input the TextProj function in the reference frame text branch to obtain the debiased text features .

[0037] Furthermore, the latent variables are generated using the reparameterization technique The expression is:

[0038]

[0039] in, is the random noise obtained by random sampling from a standard normal distribution.

[0040] Further, step S5 includes the following steps:

[0041] Step S51: Randomly select The training set data of each category is selected, and one corresponding image and text label are selected. Steps S3 and S4 are executed to obtain Group paired debiased visual features and debiased text features;

[0042] Step S52: Design a loss function based on variational inference. The specific expression of the loss function is:

[0043] L o s s = − (  [ l o g p ( y | F Debiasing visual features , η ) ] − K L ( ℕ ( W ˜ 0 , r ( F Debiasing visual features ) 2 ) | | ℕ ( 0 , 1 ) ) )

[0044] in,  [ l o g p ( y | F Debiasing visual features , η ) ] Indicates the debiased visual features and hidden variables Predict the outcome in a given situation The log-likelihood of ; In step S42, the debiased visual cue token is the mean, probability distribution token is a Gaussian distribution with variance and standard Gaussian distribution The KL divergence between

[0045] Step S53: From A set of debiased visual features and paired debiased text features are selected as positive sample pairs; the selected debiased visual features are compared with the rest The debiased text features are used as negative sample pairs, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference.

[0046] Step S54: From A set of debiased visual features and debiased text features that are not selected as positive sample pairs are selected from the paired debiased visual features and debiased text features as positive sample pairs; the selected debiased visual features are combined with the remaining The debiased text features are used as negative sample pairs, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference. Each group of paired debiased visual features and debiased text features is selected as a positive sample pair, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference;

[0047] Step S55: After all samples in the training set constructed in step S1 participate in the network parameter update, the optimal model parameters and the debiased text features of each category are output to obtain a model with optimal parameters.

[0048] Furthermore, the image recognition cosine similarity is calculated as follows:

[0049]

[0050] in, Represented as the predicted category for a given debiased visual feature The probability of represents the total number of categories, Indicates categories, Represents the temperature parameter.

[0051] Compared with the prior art, the present invention has the following beneficial technical effects: the present invention adopts the mainstream CLIP as the benchmark model, and improves the image encoder and text encoder of CLIP respectively, including: designing a style-perturbed image encoder to reduce the contrastive language-image pre-training model's dependence on the image-specific context; based on the output of the CLIP text encoder and the style-perturbed image encoder, designing a text encoder based on the image recognition method of debiased variational inference prompt tuning to alleviate the overfitting of the contrastive language-image pre-training model to the task-specific description; designing a debiased variational inference loss function to optimize the model parameters. The present invention provides a general small sample image recognition method for data-scarce or restricted scenarios, significantly reducing the need for manually annotated data, thereby reducing the data annotation costs of enterprises. In addition, the present invention can still maintain a high recognition accuracy and excellent generalization performance under limited data, which provides enterprises with an efficient and flexible intelligent solution in data-scarce environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0053] Figure 1 The present invention is a training flowchart of an image recognition method based on debiased variational inference prompt tuning according to an embodiment of the present invention.

[0054] Figure 2 This is a flow chart of an image encoder based on style perturbation according to an embodiment of the present invention.

[0055] Figure 3 This is a flowchart of a text encoder based on debiased variational inference according to an embodiment of the present invention.

[0056] Figure 4 This is a test flow chart of an image recognition method based on debiased variational inference prompt tuning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The present invention proposes an image recognition method based on debiased variational inference prompt tuning. The embodiment of the present invention takes the fine-grained dataset FGVC-Aircraft as an example, wherein the FGVC-Aircraft dataset is a fine-grained visual classification task dedicated to distinguishing different types of aircraft. The overall flow chart of the training process of the image recognition method based on debiased variational inference prompt tuning proposed by the present invention is as follows: Figure 1 As shown, the method comprises the following steps:

[0059] Step S1: Split the paired image-text FGVC-Aircraft dataset into a training set and a test set, where the training set is randomly selected from each category The remaining paired image-text labels are used as the test set.

[0060] Step S2: The Contrastive Language-Image Pre-Training (CLIP) model is used as a benchmark framework, which includes an image branch and a text branch. The image branch includes an image block encoding module, an image encoder, and an ImageProj function; the text branch includes a text word embedding module, a text encoder, and a TextProj function.

[0061] Step S3: Based on the image branch of the benchmark framework, design a style perturbation-based image encoder, select 2 image data from the training set of step S1, and then input the style perturbation-based image encoder, where 1 image data is used as training image data and the other image data is used as auxiliary image data to obtain debiased visual features.

[0062] Step S4: Based on the text branch of the benchmark framework, design a text encoder based on debiased variational inference; select text labels in the training set that are paired with the training image data selected in step S3, and input the debiased visual features obtained in step S3 and the paired text labels into the text encoder based on debiased variational inference to obtain debiased text features.

[0063] Step S5: Randomly select The training set data of categories, select one corresponding image and text label for each category, and execute steps S3 and S4 to obtain The debiased visual features and debiased text features of the group are combined; a loss function based on variational inference is designed, and positive sample pairs and negative sample pairs are selected to input the loss function based on variational inference to complete the update of network parameters; after all samples in the training set constructed in step S1 participate in the network parameter update, the optimal model parameters and the debiased text features of each category are output to obtain a model with optimal parameters.

[0064] Step S6: Input the test set image data into the model with the optimal parameters after optimization in step S5, obtain the debiased visual features through the image encoder based on style perturbation, and obtain the image recognition results by calculating the highest cosine similarity between the debiased visual features and the debiased text features of all categories.

[0065] The specific construction steps of an image recognition method based on debiased variational inference prompt tuning in an embodiment of the present invention are as follows:

[0066] Step S1: Include The FGVC-Aircraft dataset of categories is divided into training set and test set. Each category is randomly selected samples as training set, and the remaining samples as test set, where each category contains paired image data and corresponding text labels. is the number of data set categories. In this embodiment , .

[0067] Step S2: The contrastive language-image pre-training model is used as a benchmark framework, which includes an image branch and a text branch. The image branch includes an image block encoding module, an image encoder, and an ImageProj function; the text branch includes a text word embedding module, a text encoder, and a TextProj function; wherein the image encoder and the text encoder are both composed of Transformer network layers, represented as and , Represents the image encoder Transformer network layers, Represents the text encoder Transformer network layers, in this embodiment .

[0068] Step S3: Based on the image branch of the benchmark framework, design a style perturbation-based image encoder, select 2 image data from the training set of step S1, input the style perturbation-based image encoder, where 1 image data is used as training image data and the other image data is used as auxiliary image data to obtain debiased visual features.

[0069] The style perturbation-based image encoder is a forward-looking representation of the image encoder of the reference frame image branch. The style perturbation operation is performed on the layer to obtain a style perturbation-based image encoder, which aims to reduce the dependence of the contrast language-image pre-training model on the image specific context by randomly perturbing the visual tokens.

[0070] Specifically, Figure 2 As shown, step S3 includes the following steps:

[0071] Step S31: compress the training image data to a length of , width The compressed image data is then cropped into The size is The image block, , and The image blocks are input into the image block encoding module to obtain the 0th layer visual token of the training image data. , , among which, the visual tokens have Visual sub-tokens, Indicates the number of dimensions of each visual sub-token, , Indicates each The number of channels of the image block; in this embodiment , , , , , Similarly, the auxiliary image data is compressed and cropped to obtain the auxiliary image data layer 0 visual token , .

[0072] Step S32: Visual tokens at level 0 of the training image data Introducing a learnable class word token , get the visual token with category name of the training image data layer 0 [ c 0 , E 0 train ] , The dimension is 768

[0073] Step S33: Visual tokens with category names in layer 0 of the training image data [ c 0 , E 0 train ] The first layer of the Transformer network layer inputs the image encoder and outputs the first layer of visual tokens with category names for the training image data, represented as [ c 1 , E 1 train ] . is the first-level category word token, Layer 1 visual tokens for the training image data. . Add auxiliary image data layer 0 visual token Input image encoder layer 1 Transformer network layer, output auxiliary image data layer 1 visual token .

[0074] Step S34: The front of the image encoder The visual tokens of the training image data in the visual tokens with category names output by each layer of the Transformer network layer are subjected to style perturbation operations to obtain visual tokens after the perturbation of the training image data. The visual tokens with category names output by the layer are represented as [ c i , E i train ] , ;in, For the Layer class word tokens, The dimension is 768, For the training image data Layer Visual Tokens, , after the style perturbation operation, the training image data is obtained Visual tokens after layer perturbation The number of layers that perform style perturbations on visual tokens It can be reasonably set as needed. .

[0075] Specifically, the style perturbation operation includes the following steps:

[0076] Step S341: The image encoder The training image data output by the Transformer network layer Layer Visual Token , flattened into training image data Layer Flatten Visual Tokens , ;

[0077] Step S342: Calculate the training image data Layer Flatten Visual Tokens The style features include the mean of the training image data and the training image data variance , calculated as follows:

[0078]

[0079]

[0080] in, Represents the training image data Each visual sub-token of the layer is The matrix obtained by performing mean operation on each channel; Represents the training image data Each visual sub-token of the layer is The channel coordinates are The corresponding numerical matrix; and Respectively represent the training image data Each visual sub-token of the layer is The height and width corresponding to each channel; Represents the training image data Each visual sub-token of the layer is The matrix obtained by performing standard deviation operation on each channel; represents the smoothing term, ;

[0081] Step S343: The auxiliary image data Layer Visual Token After the same flattening and style feature calculation operations as the training image data, the mean value of the auxiliary image data is obtained. and auxiliary image data variance ;

[0082] Step S344: average the training image data , training image data variance , auxiliary image data mean and auxiliary image data variance Perform random convex weight combination calculation to obtain the mixed mean after style perturbation and the mixed variance after style perturbation , the calculation formula for random convex weight combination is:

[0083]

[0084]

[0085] in, is the harmonizing factor, ;

[0086] Step S345: Obtain the flattened visual token after disturbance using the style transformation formula ;in, , the style transformation formula is as follows:

[0087]

[0088] in, Represents the mean of training image data The mean matrix of training image data after dimensional expansion using PyTorch broadcasting mechanism; Represents the variance of training image data The variance matrix of the training image data after expanding the dimension using the PyTorch broadcast mechanism; Represents the mixed mean after style perturbation The style perturbation mixture mean matrix after expanding the dimension using PyTorch broadcast mechanism; Represents the mixed variance after style perturbation The style perturbation mixture variance matrix after dimensionality expansion using PyTorch broadcasting mechanism.

[0089] Step S346: Flatten the disturbed visual tokens The dimension is reset to obtain the training image data Visual tokens after layer perturbation ;in, , .

[0090] Step S347: Visual tokens after layer perturbation With Layer Classifier Token Combined to get the training image data Layer visual tokens with class names [ c i , E i Perturbation ] .

[0091] Step S348: The training image data Layer visual tokens with class names [ c i , E i Perturbation ] Input image encoder Transformer network layer, output training image data Layers with class-named visual tokens [ c i + 1 , E i + 1 train ] ; [ c i + 1 , E i + 1 train ] = V i + 1 ([ c i , E i Perturbation ]), i = 1 , 2 , ⋅⋅⋅ , j ; For the training image data Layer Visual Tokens, For the The class word token of the layer, Represents the image encoder Transformer network layer; the auxiliary image data Layer Visual Token Input image encoder Transformer network layer, get auxiliary image data Layer visual tokens, represented as .

[0092] Step S35: Image encoder To The training image data output by each layer of the Transformer network layer carries the visual token of the category name, which is directly input into the next Transformer network layer to obtain the training image data of the next layer with the visual token of the category name; finally, the training image data of the first layer is obtained. Layers with class-named visual tokens .

[0093] Specifically, the training image data The visual tokens with class names are represented as [ c k , E k train ], k = j + 1 ,..., K − 1 , then after the Transformer network layer, output training image data Layers with class-named visual tokens [ c k + 1 , E k + 1 train ] ; [ c k + 1 , E k + 1 train ] = V k + 1 ([ c k , E k train ] .

[0094] Step S36: The training image data Layers with class-named visual tokens Medium Class Token Input the ImageProj function in the benchmark frame of step S2 to obtain the debiased visual features , Indicates the dimension size of the debiased visual feature, .

[0095] Step S4: Based on the text branch of the benchmark framework, design a text encoder based on debiased variational inference; select text labels in the training set that are paired with the training image data selected in step S3, and input the text labels and the debiased visual features obtained in step S3 into the text encoder based on debiased variational inference to obtain debiased text features.

[0096] The present invention obtains debiased visual cue tokens and probability distribution tokens by inputting debiased visual features into a meta-learning network, fuses the debiased visual cue tokens and probability distribution tokens with text tokens output by a text word embedding module, and then executes according to the text branch of the reference framework to obtain a text encoder based on debiased variational inference. The present invention can alleviate the overfitting of the contrast language-image pre-training model to task-specific descriptions through the text encoder based on debiased variational inference.

[0097] Specifically, Figure 3 As shown, step S4 includes the following steps:

[0098] Step S41: Select the text label in the training set that is paired with the training image data selected in step S3, and input the text label into the text word embedding module to obtain the 0th layer text token , denoted as W 0 = [ w 0 1 ,... w 0 i , ⋅⋅⋅ , w 0 N ] ∈ R N × d l , Indicates the 0th layer text sub-tokens, Indicates the size of the text sub-token dimension, Indicates the number of text sub-tokens; in this embodiment , .

[0099] Step S42: Debiasing the visual features obtained in step S3 Input the meta-learning network to get the debiased visual cue token and probability distribution tokens , further generating tokens to debias visual cues is the mean, probability distribution token is a Gaussian distribution with variance ; Debias visual cue tokens Expressed as W ˜ = [ w ˜ 1 ,..., w ˜ i ,..., w ˜ N ] ∈ R N × d l , Indicates debiased visual cue sub-tokens; probability distribution tokens Expressed as r ( F Debiasing visual features ) = [ r 1 ,..., r i ,..., r N ] ∈ R N × d l , Indicates The meta-learning network consists of two fully connected neural network layers. The first layer of the meta-learning network has a neural network dimension of 1024, and the second layer of the meta-learning network has a neural network dimension of 512.

[0100] Step S43: Debiasing visual cue tokens and probability distribution tokens Generate latent variables using reparameterization techniques , using the reparameterization technique to generate latent variables The expression is:

[0101]

[0102] in, is the random noise obtained by random sampling from the standard normal distribution; the latent variable is expressed as η = [ τ 1 ,..., τ i ,..., τ N ] ∈ R N × d l , Indicates sub-latent variable tokens.

[0103] Step S44: The layer 0 text token obtained in step S41 and hidden variables Fusion obtains text tokens with introduced probability distribution , denoted as p r = [ w 0 1 + τ 1 ,..., w 0 i + τ i ,..., w 0 N + τ N ] .

[0104] Step S45: The text tokens obtained in step S44 are introduced into the probability distribution Input the text encoder to get the final text representation , expressed as W K = [ w K 1 ,... w K i , ⋅⋅⋅ , w K N ] ∈ R N × d l ;in, Indicates Sub-text representation, then the text representation The last subtext representation in Input the TextProj function in the reference frame text branch to obtain the debiased text features , denoted as ,in, Represents the TextProj function, , .

[0105] Step S5: Randomly select The training set data of categories, select one corresponding image and text label for each category, and execute steps S3 and S4 to obtain The debiased visual features and debiased text features of the group are combined; a loss function based on variational inference is designed, and positive sample pairs and negative sample pairs are selected to input the loss function based on variational inference to complete the update of network parameters; after all samples in the training set constructed in step S1 participate in the network parameter update, the optimal model parameters and the debiased text features of each category are output to obtain a model with optimal parameters.

[0106] Positive sample pairs and negative samples are selected in the following way: A set of debiased visual features and paired debiased text features are used as positive sample pairs; the selected debiased visual features are compared with the rest debiased text features as negative sample pairs; until Each pair of debiased visual features and debiased text features is selected as a positive sample pair.

[0107] Specifically, step S5 includes the following steps:

[0108] Step S51: Randomly select The training set data of categories, select one corresponding image and text label for each category, and execute steps S3 and S4 to obtain Group paired debiased visual features and debiased textual features.

[0109] Step S52: Design a loss function based on variational inference. The specific expression of the loss function is:

[0110] L o s s = − (  [ l o g p ( y | F Debiasing visual features , η ) ] − K L ( ℕ ( W ˜ 0 , r ( F Debiasing visual features ) 2 ) | | ℕ ( 0 , 1 ) ) )

[0111] in,  [ l o g p ( y | F Debiasing visual features , η ) ] Indicates the debiased visual features and hidden variables Predict the outcome in a given situation The log-likelihood of ; In step S42, the debiased visual cue token is the mean, probability distribution token is a Gaussian distribution with variance and standard Gaussian distribution The KL divergence between .

[0112] In debiasing visual features and hidden variables Predict the outcome in a given situation The log-likelihood of  [ l o g p ( y | F Debiasing visual features , η ) ] Calculated as follows:

[0113]  [ l o g p ( y | F Debiasing visual features , η ) ] = 1 X ∑ i = 1 X l o g p ( y i | F Debiasing visual features , η )

[0114] in, Indicates The text labels for the categories, Given a debiased visual feature and hidden variables In this case, the predicted category is The probability of The expression is:

[0115]

[0116] in, Indicates categories, Indicates The debiased text features of categories, represents the temperature parameter, , represents the debiased visual features and The cosine similarity between the debiased text features of the categories, The expression is:

[0117]

[0118] in, express Norm.

[0119] To debias visual cue tokens is the mean, probability distribution token is a Gaussian distribution with variance and standard Gaussian distribution The specific expression of KL divergence between is:

[0120]

[0121] Step S53: From A set of debiased visual features and paired debiased text features are selected as positive sample pairs; the selected debiased visual features are compared with the rest The debiased text features are used as negative sample pairs, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference.

[0122] Step S54: From A set of debiased visual features and debiased text features that are not selected as positive sample pairs are selected from the paired debiased visual features and debiased text features as positive sample pairs; the selected debiased visual features are combined with the remaining The debiased text features are used as negative sample pairs, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference. Each group of paired debiased visual features and debiased text features is selected as a positive sample pair, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference.

[0123] Step S55: After all samples in the training set constructed in step S1 participate in the network parameter update, the optimal model parameters and the debiased text features of each category are output to obtain a model with optimal parameters.

[0124] Step S6: Input the test set image data into the model with the optimal parameters after optimization in step S5, obtain the debiased visual features through the image encoder based on style perturbation, and obtain the image recognition results by calculating the highest cosine similarity between the debiased visual features and the debiased text features of all categories.

[0125] Image recognition cosine similarity is calculated as follows:

[0126]

[0127] in, Represented as the predicted category for a given debiased visual feature The probability of represents the total number of categories, , Indicates categories, represents the temperature parameter, .

[0128] like Figure 4 As shown in the figure, a 737-200 aircraft image data is input, and the final model gives "This is an image of a 737-200".

[0129] The present invention designs an image encoder based on style perturbation, which reduces the contrastive language-image pre-training model's dependence on image-specific context by randomly perturbing visual tokens; the design of a text encoder based on debiased variational inference can alleviate the overfitting of the contrastive language-image pre-training model to task-specific descriptions. It provides a general small-sample image recognition method for scenarios with scarce or restricted data, significantly reducing the need for manually annotated data, thereby reducing the data annotation costs of enterprises. In addition, the present invention can still maintain a high recognition accuracy and excellent generalization performance under limited data, especially in the recognition of new or rare categories, which provides enterprises with an efficient, flexible and intelligent solution in data-scarce environments.

[0130] The present invention provides a prompt tuning method for an image recognition method based on debiased variational inference prompt tuning. The above-mentioned open source methods, image processing tools, parameter values, etc. are all intended to help readers understand the principles of the present invention, and the protection scope of the present invention is not limited to such special statements and embodiments. Ordinary technicians in this field can also make improvements and deformations that do not deviate from the principles of the present invention, and these improvements and deformations still belong to the protection scope of the present invention.

Claims

1. An image recognition method based on debiased variational inference prompt tuning, characterized in that: The following steps are involved: Step S1: Split the paired image-text FGVC-Aircraft dataset into a training set and a test set, where the training set is randomly selected from each category The remaining paired image-text labels are used as the test set; Step S2: using the contrastive language-image pre-training model as a benchmark framework, the benchmark framework includes an image branch and a text branch, the image branch includes an image block encoding module, an image encoder, and an ImageProj function; the text branch includes a text word embedding module, a text encoder, and a TextProj function; Step S3: Based on the image branch of the reference framework, a style perturbation-based image encoder is designed, and two image data in the training set of step S1 are selected and then input into the style perturbation-based image encoder, where one image data is used as training image data and the other image data is used as auxiliary image data to obtain debiased visual features; Step S4: Based on the text branch of the reference framework, a text encoder based on debiased variational inference is designed; text labels paired with the training image data selected in step S3 are selected from the training set, and the debiased visual features obtained in step S3 and the paired text labels are input into the text encoder based on debiased variational inference to obtain debiased text features; Step S5: Randomly select The training set data of each category is selected, and one corresponding image and text label are selected. Steps S3 and S4 are executed to obtain Group paired debiased visual features and debiased text features; Design a loss function based on variational inference, select positive sample pairs and negative sample pairs and input the loss function based on variational inference to complete the update of network parameters; until all samples in the training set constructed in step S1 participate in the network parameter update, output the optimal model parameters and the debiased text features of each category, and obtain the model with the optimal parameters; Step S6: input the test set image data into the model with the optimal parameters after optimization in step S5, obtain the debiased visual features through the image encoder based on style perturbation, and obtain the image recognition result by calculating the highest cosine similarity between the debiased visual features and the debiased text features of all categories; Step S3 includes the following steps: Step S31: compress the training image data to a length of , width The compressed image data is then cropped into The size is The image block, , and The image blocks are input into the image block encoding module to obtain the 0th layer visual token of the training image data. , ; Among them, the visual token has a total of Visual sub-tokens, Indicates the number of dimensions of each visual sub-token, , Indicates each The number of channels of the image block; similarly, the auxiliary image data is compressed and cropped to obtain the 0th layer visual token of the auxiliary image data , ; Step S32: Visual tokens at level 0 of training image data Introducing a learnable class word token , get the visual token with category name of the training image data layer 0 ; Step S33: Visual tokens with category names in layer 0 of the training image data The first layer of the Transformer network layer inputs the image encoder and outputs the first layer of visual tokens with category names for the training image data, represented as ; is the first-level category word token, Visual tokens for the first layer of training image data; ; Add auxiliary image data layer 0 visual token Input image encoder layer 1 Transformer network layer, output auxiliary image data layer 1 visual token ; Step S34: The front of the image encoder The visual tokens of the training image data in the visual tokens with category names output by each layer of the Transformer network layer are subjected to style perturbation operations to obtain visual tokens after the perturbation of the training image data; Step S35: Image encoder To The training image data output by each layer of the Transformer network layer carries the visual token of the category name, which is directly input into the next Transformer network layer to obtain the training image data of the next layer with the visual token of the category name; finally, the training image data of the first layer is obtained. Layers with class-named visual tokens ; Step S36: training image data Layers with class-named visual tokens Medium Class Token Input the ImageProj function in the benchmark frame of step S2 to obtain the debiased visual features , Indicates the dimension size of the debiased visual feature; The style perturbation operation includes the following steps: Step S341: The image encoder The training image data output by the Transformer network layer Layer Visual Token , flattened into training image data Layer Flatten Visual Tokens , ; Step S342: Calculate the training image data Layer Flatten Visual Tokens The style features include the mean of the training image data and the training image data variance ; Step S343: The auxiliary image data Layer Visual Token After the same flattening and style feature calculation operations as the training image data, the mean value of the auxiliary image data is obtained. and auxiliary image data variance ; Step S344: average the training image data , training image data variance , auxiliary image data mean and auxiliary image data variance Perform random convex weight combination calculation to obtain the mixed mean after style perturbation and the mixed variance after style perturbation ; Step S345: Obtain the flattened visual token after disturbance using the style transformation formula , ; Step S346: Flatten the disturbed visual tokens The dimension is reset to obtain the training image data Visual tokens after layer perturbation ;in, ; Step S347: Visual tokens after layer perturbation With Layer Classifier Token Combined to get the training image data Layer visual tokens with class names ; Step S348: The training image data Layer visual tokens with class names Input image encoder Transformer network layer, output training image data Layers with class-named visual tokens ; For the training image data Layer Visual Tokens, For the The class word token of the layer, Represents the image encoder Transformer network layer; the auxiliary image data Layer Visual Token Input image encoder Transformer network layer, get auxiliary image data Layer visual tokens, represented as ; Step S4 includes the following steps: Step S41: Select the text label in the training set that is paired with the training image data selected in step S3, and input the text label into the text word embedding module to obtain the 0th layer text token ; Step S42: Debiasing the visual features obtained in step S3 Input the meta-learning network to get the debiased visual cue token and probability distribution tokens ; Step S43: Debiasing visual cue tokens and probability distribution tokens Generate latent variables using reparameterization techniques ; Step S44: The layer 0 text token obtained in step S41 and the hidden variable obtained in step S43 Fusion obtains text tokens with introduced probability distribution ; Step S45: The text tokens obtained in step S44 are introduced into the probability distribution Input the text encoder to get the final text representation ; Represent the text The last subtext representation in Input the TextProj function in the reference frame text branch to obtain the debiased text features .

2. The image recognition method based on debiased variational inference prompt tuning according to claim 1, characterized in that: The style perturbation-based image encoder is constructed by The style perturbation operation is performed on the layer.

3. The image recognition method based on debiased variational inference prompt tuning according to claim 1, characterized in that: The text encoder based on debiased variational inference obtains the debiased visual cue tokens and probability distribution tokens by inputting the debiased visual features into the meta-learning network, fuses the debiased visual cue tokens and probability distribution tokens with the text tokens output by the text word embedding module, and then executes them according to the text branch of the benchmark framework.

4. The image recognition method based on debiased variational inference prompt tuning according to claim 1, characterized in that: Generate latent variables using reparameterization techniques The expression is: in, is the random noise obtained by random sampling from a standard normal distribution.

5. The image recognition method based on debiased variational inference prompt tuning according to claim 1, characterized in that: Step S5 includes the following steps: Step S51: Randomly select The training set data of each category is selected, and one corresponding image and text label are selected. Steps S3 and S4 are executed to obtain Group paired debiased visual features and debiased text features; Step S52: Design a loss function based on variational inference. The specific expression of the loss function is: in, Indicates the debiased visual features and hidden variables Predict the outcome in a given situation The log-likelihood of ; In step S42, the debiased visual cue token is the mean, probability distribution token is a Gaussian distribution with variance and standard Gaussian distribution The KL divergence between Step S53: From A set of debiased visual features and paired debiased text features are selected as positive sample pairs; the selected debiased visual features are compared with the rest The debiased text features are used as negative sample pairs, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference. Step S54: From A set of debiased visual features and debiased text features that are not selected as positive sample pairs are selected from the paired debiased visual features and debiased text features as positive sample pairs; the selected debiased visual features are combined with the remaining The debiased text features are used as negative sample pairs, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference. Each group of paired debiased visual features and debiased text features is selected as a positive sample pair, and the loss function based on variational inference is input. The network parameters are updated by minimizing the loss function based on variational inference; Step S55: After all samples in the training set constructed in step S1 participate in the network parameter update, the optimal model parameters and the debiased text features of each category are output to obtain a model with optimal parameters.

6. The image recognition method based on debiased variational inference prompt tuning according to claim 1, characterized in that: Image recognition cosine similarity is calculated as follows: in, Represented as the predicted category for a given debiased visual feature The probability of represents the total number of categories, Indicates categories, Represents the temperature parameter.

Images