Diversity induction bias learning method for generalizable large model

Abstract

The invention provides a generalizable large model-oriented diversity induction bias learning method. The method comprises the following steps of: constructing an input pair image of complete prompt description according to a training sample set; extracting image features and text features of the input pair by using the large generalization model; and for the same pair of inputs, performing supervised training by using cosine similarity loss. In order to meet the semantic gap between a language and a visual mode, the invention provides a text-level inductive deviation, and detailed information of each category is provided by supplementing a plurality of descriptions generated by LLM in a prompt text; in order to enable the model to well capture inductive deviation, designing a phrase adapter for a text encoder to explicitly explore connection between adjacent words; designing a space adapter for the image encoder to allow the model to see more local relationships and details; according to the method, overfitting is reduced through optimization level inductive bias; this is accomplished by a dynamic training strategy that enables the model to learn fitting states of varying degrees.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a diversity inductive bias learning method for a generalizable large model. Background Art

[0002] Deep neural networks are powerful tools for image understanding. Among them, the most successful paradigm for vision-related tasks is to first pre-train a base deep neural network on some large-scale datasets, and then fine-tune the pre-training on a specific task. This is largely due to the availability of crowd-sourced labeled datasets such as ImageNet, PLACES, and MSCOCO. However, these datasets and their high-quality labels are expensive to collect.

[0003] The advent of Visual Language Models (VLMs) reduces the effort of manually collecting such high-quality annotations. VLMs are pre-trained on large amounts of web search data, such as the 400 million image-text pairs used in the Contrastive Language-Image Pretraining (CLIP) model. Through extensive pre-training, VLMs help to fully understand the complex relationship between images and language and demonstrate good generalization capabilities across a variety of tasks. Although fine-tuning a pre-trained VLM is the most straightforward strategy, the large number of parameters of VLMs poses challenges for fine-tuning them for different downstream tasks, especially when data and labels are limited. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a diverse inductive bias learning method for a generalizable large model, and the present invention can achieve the learning of three types of inductive biases.

[0005] The technical solution of the present invention is: a diversity inductive bias learning method for a generalizable large model, comprising the following steps:

[0006] S1), constructing the input pair image with complete prompt description according to the training sample set;

[0007] S2), using the generalized large model to extract image features and text features of the input pair;

[0008] S3), for the same pair of inputs, supervised training is performed using cosine similarity loss.

[0009] Preferably, in step S1), for each image-label input pair, the corresponding category label CN ​​is extracted, and the category description T is constructed according to the manual prompt description form. hc (CN):=one <cn>image.

[0010] Preferably, in step S1), the ChatGPT3 large language model is used to generate K custom prompt descriptions T for the corresponding category word CN. ct (CN) i .

[0011] Preferably, in step S1), the category description T constructed in the form of manual prompt description is hc (CN) with custom prompt description T ct (CN) i Use the splicing operator to splice and integrate to get the complete prompt description T(CN) corresponding to the input image i ,Right now:

[0012] T(CN) i =[T hc (CN); T ct (CN) i ].

[0013] Preferably, in step S2), the generalized large model includes a text encoder and an image encoder; the text encoder and the image encoder have similar structures, but do not share parameters with each other; and the text encoder and the image encoder each include multiple encoding layers.

[0014] Preferably, in step S2), the generalized large model includes three inductive biases, namely, text-level inductive bias; using the customized text generated by LLM to supplement "a <cn>photos;

[0015] Model-level inductive biases: Through the adapter, some hard inductive biases are learned for tuning the text and image encoders;

[0016] Optimizing the inductive bias, in the adapter, the task-specific knowledge is controlled by the hyperparameter α; during training, DY(α) is dynamically changed to allow the model to learn different degrees of task-specific knowledge.

[0017] Preferably, in step S2), the encoding layer includes a multi-head attention module and a feed-forward layer, and the model is enriched by using layer normalization and an inductive bias after the first fully connected layer in the multi-head attention module and the feed-forward layer.

[0018] Preferably, in step S2), a corresponding phrase adapter is inserted into each encoding layer of the text encoder; and a corresponding space adapter is inserted into each encoding layer of the image encoder.

[0019] Preferably, in step S2), the extraction of text features is specifically as follows:

[0020] S21), converting the complete prompt word description of each input pair into a text embedding feature W, and then sending the text embedding feature W to the L encoding layers of the text autoencoder, and performing feature extraction on the text embedding feature W through the L encoding layers and the phrase adapter.

[0021] Preferably, in step S21), the expression of the text embedding feature W is:

[0022]

[0023] In the formula, w represents each token in the text, M is the total number of tokens, and D is the feature dimension of the token.

[0024] Preferably, in step S21), in each encoding layer of the text autoencoder, the text embedding feature W is first processed by the LayerNorm activation layer of the encoding layer, and then sent to the phrase adapter PA, that is:

[0025] PA(LN(W))=W+DY(α)·DWConv1D(W);

[0026] Where DWConv1D refers to a one-dimensional depthwise separable convolution with a kernel size of 3×1×D; DY(α) is a piecewise function with hyperparameters α and s, PA represents the phrase adapter PA, and LN(W) represents the output processed by the LayerNorm activation layer.

[0027] Preferably, in step S21), the output of the phrase adapter PA is subjected to a standard multi-head self-attention operation Attention, and then updated to the original text embedding feature W in the form of a residual, that is:

[0028] W′=W+Attention(PA(LN(W))).

[0029] Preferably, in step S21), the updated feature W′ is further extracted through the LayerNorm activation layer, the fully connected layer, the phrase adapter and the fully connected layer, and updated to the original text embedding feature W in the form of residual learning; that is:

[0030] W″=W+FC2(PA(FC1(LN(W))));

[0031] Where FC1 and FC2 represent the first and second fully connected layers, respectively.

[0032] Preferably, in step S21), the piecewise function DY(α) is expressed as:

[0033]

[0034] Where prob is randomly sampled from a uniform distribution [0, 1], the hyperparameter α is the proportional factor, and s is the scale factor.

[0035] Preferably, in step S2), the extraction of image features is specifically as follows:

[0036] S22), for each input pair, discretize the image and convert it into a series of image-embedded password features And on this basis, add a category embedded password feature Then concatenate the two to get the complete image embedding feature Then it is sent to the L encoding layers of the image autoencoder, and the image embedding feature Y is extracted through the L encoding layers and the spatial adapter SA.

[0037] Preferably, in step S22), similar to the text autoencoder, the image embedding feature Y passes through the LayerNorm activation layer, the spatial adapter SA, the multi-head self-attention mechanism, and then the parameters are updated in the form of residuals, that is:

[0038] Y′=Y+Attention(SA(LN(Y))).

[0039] Preferably, in step S22), Y′ updated in the form of residual is extracted through the LayerNorm activation layer, the fully connected layer, the spatial adapter SA and the fully connected layer, and updated in the form of residual learning; that is:

[0040] Y″=Y+FC2(PA(FC1(LN(Y))));

[0041] Where FC1 and FC2 represent the first and second fully connected layers, respectively.

[0042] Preferably, in step S22), the operation flow of the spatial adapter SA is as follows: first, the image-embedded password feature X is extracted from Y, and then the image-embedded password feature sequence is restored to a two-dimensional image shape, the shape is Denoted as:

[0043] X = Seq2Im(X);

[0044] Where the Seq2Im(X) operation represents the reshaping of a one-dimensional sequence into a two-dimensional feature map.

[0045] Preferably, in step S22), a two-dimensional depth-separable convolution is performed on the image-embedded password feature X, and a piecewise function DY(α) is introduced to modulate the convolution result, and finally the extracted feature is updated to X in the form of a residual, that is:

[0046] X′=X+DY(α)·DWConv2D(X);

[0047] Among them, DWCon v 2D refers to a convolution kernel of size 3×3×1×D.

[0048] Preferably, in step S22), the updated image embedded password feature X′ is discretized into a password sequence, and concatenated with the category password embedded feature to obtain a complete spatial adapter output Y, that is:

[0049] Y = [c; Im2Seq(X)];

[0050] Where Im2Seq(X) represents the reshaping of the two-dimensional feature map back to a one-dimensional sequence.

[0051] Preferably, in step S3), supervised learning is performed based on the obtained text features W″ and image features Y″ using cosine similarity loss.

[0052] The beneficial effects of the present invention are:

[0053] 1. In order to bridge the semantic gap between language and visual patterns, the present invention proposes a text-level inductive bias to provide detailed information for each category by supplementing the prompt text with many LLM-generated descriptions;

[0054] 2. In order to enable the model to capture inductive bias well, the present invention proposes two types of adapters for text and image encoders respectively. For the text encoder, a phrase adapter is designed to explicitly explore the connection between adjacent words; for the image encoder, a spatial adapter is designed to allow the model to see more local relationships and details;

[0055] 3. The present invention reduces overfitting by optimizing the level inductive bias; this is achieved through a new dynamic training strategy that enables the model to learn different degrees of fitting states. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the method of the present invention;

[0057] Figure 2 It is a schematic diagram of the framework of the generalized large model of the present invention;

[0058] Figure 3 It is a schematic diagram of the model-level inductive bias integration of the present invention;

[0059] Figure 4 It is a schematic diagram of the structure of the spatial adapter of the image encoder of the present invention. DETAILED DESCRIPTION

[0061] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0062] This embodiment provides a method for learning a diversity inductive bias for a generalizable large model, including the following steps:

[0063] S1), constructing the input pair image with complete prompt description according to the training sample set;

[0064] In this embodiment, constructing an input pair image with a complete prompt description specifically includes the following steps:

[0065] S11), for each image-label input pair, extract its corresponding category label CN, and construct the category description T according to the manual prompt description form hc (CN):=one <cn>image;

[0066] S12), Generate K custom prompt descriptions T for the corresponding category word CN through the ChatGPT3 large language model ct (CN) i ;

[0067] S13), by constructing the category description T in the form of manual prompt description hc (CN) with custom prompt description T ct (CN) i Use the splicing operator to splice and integrate to get the complete prompt description T(CN) corresponding to the input image i ,Right now:

[0068] T(CN) i =[T hc (CN); T ct (CN) i ].

[0069] S2), using the generalized large model to extract image features and text features of the input pair;

[0070] In this embodiment, the framework diagram of the generalized large model is shown in Figure 2 As shown; the generalized large model includes a text encoder and an image encoder; the text encoder and the image encoder have similar structures, but do not share parameters with each other; and the text encoder and the image encoder each include multiple encoding layers, and the text encoder also includes a text embedding layer and a text mapping layer; the image encoder also includes a tile embedding layer and an image mapping layer.

[0071] As preferred in this embodiment, the generalized large model includes three inductive biases, namely, text-level inductive bias; using the customized text generated by LLM to supplement "a <cn>photos;

[0072] Model-level inductive biases: learning hard inductive biases for tuning text and image encoders through designed adapters;

[0073] Optimizing the inductive bias, in the adapter, the task-specific knowledge is controlled by the hyperparameter α; during training, DY(α) is dynamically changed to allow the model to learn different degrees of task-specific knowledge.

[0074] As a preferred embodiment of the present invention, the encoding layer includes a multi-head attention module and a feed-forward layer, and the model is enriched by using layer normalization in the multi-head attention module and the feed-forward layer and an inductive bias after the first fully connected layer.

[0075] As a preferred embodiment of the present invention, a corresponding phrase adapter is inserted into each coding layer of the text encoder; a corresponding space adapter is inserted into each coding layer of the image encoder, wherein the results of the phrase adapter and the space adapter can be seen in Figure 4 shown.

[0076] As preferred in this embodiment, Figure 3 As shown, the extraction of text features specifically includes the following steps:

[0077] S211), converting the complete prompt word description of each input pair into a text embedding feature W; the expression of the text embedding feature W is:

[0078]

[0079] In the formula, w represents each token in the text, M is the total number of tokens; D is the feature dimension of the token;

[0080] S212), then the text embedding feature W is sent to the L encoding layers of the text autoencoder. In each encoding layer of the text autoencoder, the text embedding feature W is first processed by the LayerNorm activation layer of the encoding layer, and then sent to the phrase adapter PA, that is:

[0081] PA(LN(W))=W+DY(α)·DWConv1D(W);

[0082] Where DWConv1D refers to a one-dimensional depth-wise separable convolution with a kernel size of 3×1×D; DY(α) is a piecewise function with hyperparameters α and s involved, PA represents the phrase adapter PA, and LN(W) represents the output processed by the LayerNorm activation layer;

[0083] Wherein, the piecewise function DY(α) is expressed as:

[0084]

[0085] Where prob is randomly sampled from a uniform distribution [0, 1];

[0086] S213), perform a standard multi-head self-attention operation on the output of the phrase adapter PA n , and then updated to the original text embedding feature W in the form of residual, that is:

[0087] W′=W+Attention(PA(LN(W)));

[0088] S214), the updated feature W′ in step S213) is further extracted through the LayerNorm activation layer, the fully connected layer, the phrase adapter and the fully connected layer, and updated to the original text embedding feature W in the form of residual learning; that is:

[0089] W″=W+FC2(PA(FC1(LN(W))));

[0090] Where FC1 and FC2 represent the first and second fully connected layers respectively;

[0091] S215) Repeat steps S212) to S214) until all L encoding layers of the text autoencoder are passed.

[0092] As a preferred embodiment of the present invention, the extraction of image features specifically includes the following steps:

[0093] S221), for each input pair, discretize the image and password into a series of image-embedded password features And on this basis, add a category embedded password feature Then concatenate the two to get the complete image embedding feature

[0094] S222), the complete image embedding feature Y is sent to the L encoding layers of the image autoencoder. In each encoding layer of the image autoencoder, the complete image embedding feature Y passes through the LayerNorm activation layer, the spatial adapter SA, and the multi-head self-attention mechanism, and then the parameters are updated in the form of residuals, that is:

[0095] Y′=Y+Attention(SA(LN(Y)));

[0096] S223), extract features from Y′ after updating in the form of residual through LayerNorm activation layer, fully connected layer, spatial adapter SA and fully connected layer, and update in the form of residual learning; that is:

[0097] Y″=Y+FC2(PA(FC1(LN(Y))));

[0098] Where FC1 and FC2 represent the first and second fully connected layers, respectively.

[0099] As a preferred embodiment of the present invention, in step S222), the operation flow of the spatial adapter SA is as follows: firstly, the image-embedded password feature X is extracted from Y, and then the image-embedded password feature sequence is restored to a two-dimensional image shape, the shape is Denoted as:

[0100] X = Seq2Im(X);

[0101] Where the Seq2Im(X) operation represents the reshaping of a one-dimensional sequence into a two-dimensional feature map.

[0102] As a preferred embodiment of the present invention, in step S222), a two-dimensional depth-separable convolution is performed on the image-embedded password feature X, and a piecewise function DY(α) is introduced to modulate the convolution result, and finally the extracted feature is updated to X in the form of a residual, that is:

[0103] X′=X+DY(α)·DWConv2D(X);

[0104] Among them, DWCon v 2D refers to the convolution kernel of size 3×3×1×D;

[0105] As a preferred embodiment of this invention, in step S222), the updated image embedded password feature X′ is discretized into a password sequence, and is concatenated with the category password embedded feature to obtain a complete spatial adapter output Y, that is:

[0106] Y = [c; Im2Seq(X)];

[0107] Where Im2Seq(X) represents the reshaping of the two-dimensional feature map back to a one-dimensional sequence.

[0108] S3), for the same pair of inputs, supervised training is performed using cosine similarity loss.

[0109] This embodiment performs supervised learning based on the obtained text feature W″ and image feature Y″ and uses cosine similarity loss.

[0110] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.< / cn> < / cn> < / cn> < / cn>

Claims

1. A diverse inductive bias learning method for generalizable large models, characterized in that: The following steps are involved: S1), constructing the input pair image with complete prompt description according to the training sample set; S2), using the generalized large model to extract image features and text features of the input pair; S3), for the same pair of inputs, supervised training is performed using cosine similarity loss.

2. The method for learning a diverse inductive bias for a generalizable large model according to claim 1, characterized in that: Step S1) specifically includes the following steps: S11), for each image-label input pair, extract its corresponding category label CN, and construct the category description T according to the manual prompt description form hc (CN):=one <cn> image;< / cn> S12), Generate K custom prompt descriptions T for the corresponding category word CN through the ChatGPT3 large language model ct (CN) i ; S13), by constructing the category description T in the form of manual prompt description hc (CN) and custom prompt description T ct (CN) i Use the splicing operator to splice and integrate to get the complete prompt description T(CN) corresponding to the input image i ,Right now: T(CN) i =[T hc (CN);T ct (CN) i ]。 3. The method for learning diverse inductive biases for generalizable large models according to claim 1, characterized in that: In step S2), the generalized large model includes a text encoder and an image encoder; the text encoder and the image encoder have similar structures, but do not share parameters with each other; and the text encoder and the image encoder both include multiple encoding layers, and the text encoder also includes a text embedding layer and a text mapping layer; the image encoder also includes a tile embedding layer and an image mapping layer.

4. The method for learning a diverse inductive bias for a generalizable large model according to claim 3, characterized in that: In step S2), the generalized large model includes three inductive biases, namely text-level inductive bias, model-level inductive bias, and optimization-level inductive bias.

5. The method for learning a diverse inductive bias for a generalizable large model according to claim 3, characterized in that: In step S2), a corresponding phrase adapter is inserted into each encoding layer of the text encoder; and a corresponding space adapter is inserted into each encoding layer of the image encoder.

6. The method for learning diverse inductive biases for generalizable large models according to claim 5, characterized in that: In step S2), the text feature extraction is as follows: the complete prompt word description of each input pair is converted into a text embedding feature W, and then the text embedding feature W is sent to the L encoding layers of the text autoencoder, and the text embedding feature W is extracted through the L encoding layers and the phrase adapter.

7. The method for learning diverse inductive biases for generalizable large models according to claim 6, characterized in that: In step S2), the extraction of text features specifically includes the following steps: S211), converting the complete prompt word description of each input pair into a text embedding feature W, wherein the expression of the text embedding feature W is: In the formula, w represents each token in the text, M is the total number of tokens; D is the feature dimension of the token; S212), in each encoding layer of the text autoencoder, the text embedding feature W is first processed by the LayerNorm activation layer of the encoding layer, and then sent to the phrase adapter PA, that is: PA(LN(W))=W+DY(α)·DWConv1D(W); Where DWConv1D refers to a one-dimensional depth-separable convolution with a convolution kernel size of 3×1×D; DY(α) is a piecewise function with hyperparameters α and s, where the hyperparameter α is the scale factor; s is the scale factor; PA represents the phrase adapter PA, and LN(W) represents the output processed by the LayerNorm activation layer; S213), perform a standard multi-head self-attention operation Attention on the output of the phrase adapter PA, and then update it to the original text embedding feature W in the form of a residual, that is: W′=W+Attention(PA(LN(W))); S214), the updated feature W′ is further extracted through the LayerNorm activation layer, the fully connected layer, the phrase adapter and the fully connected layer, and updated to the original text embedding feature W in the form of residual learning; that is: W″=W+FC2(PA(FC1(LN(W)))); Where FC1 and FC2 represent the first and second fully connected layers respectively; S215) Repeat steps S212) to S214) until all L encoding layers of the text autoencoder are passed.

8. The method for learning diverse inductive biases for generalizable large models according to claim 7, characterized in that: In step S212), the piecewise function DY(α) is expressed as: Where prob is randomly sampled from a uniform distribution [0, 1].

9. The method for learning diverse inductive biases for generalizable large models according to claim 8, characterized in that: In step S2), the extraction of image features specifically includes the following steps: S221), for each input pair, discretize the image and convert it into a series of image-embedded password features And on this basis, add a category embedded password feature Then concatenate the two to get the complete image embedding feature: S222), the complete image embedding feature Y is sent to the L encoding layers of the image autoencoder. In each encoding layer of the image autoencoder, the complete image embedding feature Y passes through the LayerNorm activation layer, the spatial adapter SA, and the multi-head self-attention mechanism, and then the parameters are updated in the form of residuals, that is: Y′=Y+Attention(SA(LN(Y))); S223), extract features from Y′ after updating in the form of residual through LayerNorm activation layer, fully connected layer, spatial adapter SA and fully connected layer, and update in the form of residual learning; that is: Y″=Y+FC2(PA(FC1(LN(Y)))); Where FC1 and FC2 represent the first and second fully connected layers, respectively.

10. The method for learning diverse inductive biases for generalizable large models according to claim 9, characterized in that: In step S222), the operation flow of the spatial adapter SA is as follows: first, the image-embedded password feature X is extracted from Y, and then the image-embedded password feature sequence is restored to a two-dimensional image shape, the shape is Denoted as: X = Seq2Im(X); In the formula, the Seq2Im(X) operation represents the reshaping of a one-dimensional sequence into a two-dimensional feature map; Perform a two-dimensional depth-separable convolution on the image embedded password feature X, and introduce a piecewise function DY(α) to modulate the convolution result. Finally, the extracted features are updated to X in the form of residuals, that is: X′=X+DY(α)·DWConv2D(X); Among them, DWConv2D refers to the convolution kernel of size 3×3×1×D; The updated image embedded password feature X′ is discretized into a password sequence and concatenated with the category password embedded feature to obtain the complete spatial adapter output Y, that is: Y = [c; Im2Seq(X)]; Where Im2Seq(X) represents the reshaping of the two-dimensional feature map back to a one-dimensional sequence.

Images