A Diversity Inductive Bias Learning Method for Generalizable Large Models

Abstract

The application provides a diversity inductive bias learning method for a generalizable large model, comprising constructing an input pair of images according to a complete prompt description of a training sample set; using a generalizable large model to extract image features and text features of the input pair; and using cosine similarity loss to supervise training for the same input pair. In order to bridge the semantic gap between language and visual patterns, the application proposes a text-level inductive bias, provides detailed information for each category by supplementing many LLM-generated descriptions in the prompt text; in order to enable the model to well capture the inductive bias, a phrase adapter is designed for the text encoder to explicitly explore the connection between adjacent words; a space adapter is designed for the image encoder to allow the model to see more local relationships and details; the application reduces overfitting by optimizing the level inductive bias; this is achieved through a dynamic training strategy, which enables the model to learn different degrees of fitting state.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a diversity inductive bias learning method for generalizable large models. Background Technology

[0002] Deep neural networks are powerful tools for image understanding. The most successful paradigm for vision-related tasks is to first pre-train a base deep neural network on some large-scale datasets, and then fine-tune it on task-specific pre-training. This is largely due to the availability of crowdsourced labeled datasets such as ImageNet, PLACES, and MSCOCO. However, collecting these datasets and their high-quality labels is very costly.

[0003] The emergence of Visual Language Models (VLMs) has reduced the workload of manually collecting such high-quality annotations. VLMs are pre-trained on massive amounts of web search data, such as the 400 million image-text pairs used in the Contrastive Language-Image Pre-trained Model (CLIP). Through extensive pre-training, VLMs contribute to a comprehensive understanding of the complex relationships between images and language and demonstrate good generalization ability across various tasks. While fine-tuning a pre-trained VLM is the most straightforward strategy, the large number of parameters in a VLM presents a challenge for fine-tuning it for different downstream tasks, especially when data and labels are limited. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a diversity inductive bias learning method for generalizable large models, which can achieve the learning of three inductive biases.

[0005] The technical solution of this invention is: a diversity inductive bias learning method for generalizable large models, comprising the following steps:

[0006] S1) Construct an input pair image with a complete prompt description based on the training sample set;

[0007] S2) Extract image and text features from the input pairs using a generalized large model;

[0008] S3) For the same pair of inputs, supervised training is performed using cosine similarity loss.

[0009] Preferably, in step S1), for each image-label input pair, the corresponding category label CN ​​is extracted, and a category description T is constructed based on the manual prompt description format. hc (CN): = One sheet <cn>image.

[0010] Preferably, in step S1), K custom prompt descriptions T are generated for the corresponding category word CN using the ChatGPT3 large language model. ct (CN) i .

[0011] Preferably, in step S1), the category description T is constructed by means of a manual prompt description. hc (CN) and Custom Hint Description T ct (CN) i By using a concatenation operator to concatenate and integrate the input images, a complete prompt description T(CN) corresponding to the input image is obtained. i ,Right now:

[0012] T(CN) i =[T hc (CN); T ct (CN) i ].

[0013] Preferably, in step S2), the generalized large model includes a text encoder and an image encoder; the text encoder and the image encoder have similar structures, but their parameters are not shared; and both the text encoder and the image encoder include multiple encoding layers.

[0014] Preferably, in step S2), the generalized large model includes three inductive biases: text-level inductive bias; and custom text generated by LLM to supplement "a..." <cn>Photos;

[0015] Model-level inductive bias; through an adapter, learn some hard inductive biases to adjust the text and image encoders;

[0016] To optimize inductive bias, in the adapter, the hyperparameter α controls the acquisition of task-specific knowledge; during training, DY(α) is dynamically changed to allow the model to learn different levels of task-specific knowledge.

[0017] Preferably, in step S2), the encoding layer includes a multi-head attention module and a feedforward layer, and the model is enriched by layer normalization and inductive bias after the first fully connected layer in the multi-head attention module and the feedforward layer.

[0018] Preferably, in step S2), a corresponding phrase adapter is inserted into each encoding layer of the text encoder; and a corresponding spatial adapter is inserted into each encoding layer of the image encoder.

[0019] Preferably, in step S2), the extraction of text features specifically includes:

[0020] S21) The complete prompt word description of each input pair is transformed into text embedding feature W. Then, the text embedding feature W is fed into L encoding layers of a text autoencoder. The text embedding feature W is extracted through L encoding layers and a phrase adapter.

[0021] Preferably, in step S21), the expression for the text embedding feature W is:

[0022]

[0023] In the formula, w represents each token in the text, M is the total number of tokens, and D is the feature dimension of the token.

[0024] Preferably, in step S21), in each coding layer of the text autoencoder, the text embedding feature W is first processed by the LayerNorm activation layer of the coding layer, and then sent to the phrase adapter PA, that is:

[0025] PA(LN(W))=W+DY(α)·DWConv1D(W);

[0026] In the formula, DWConv1D refers to a one-dimensional depthwise separable convolution with a kernel size of 3×1×D; DY(α) is a piecewise function with hyperparameters α and s; PA represents the phrase adapter PA; and LN(W) represents the output of the LayerNorm activation layer.

[0027] Preferably, in step S21), the output of the phrase adapter PA is subjected to a standard multi-head self-attention operation, Attention, and then the residual is used to update the original text embedding features W, i.e.:

[0028] W′=W+Attention(PA(LN(W))).

[0029] Preferably, in step S21), the updated feature W′ is further processed through a LayerNorm activation layer, a fully connected layer, a phrase adapter, and a fully connected layer to extract features, and then updated to the original text embedding feature W using residual learning; that is:

[0030] W″=W+FC2(PA(FC1(LN(W))));

[0031] In the formula, FC1 and FC2 represent the first and second fully connected layers, respectively.

[0032] Preferably, in step S21), the piecewise function DY(α) is expressed as:

[0033]

[0034] In the formula, prob is obtained by random sampling from a uniform distribution [0,1], the hyperparameter α is the scaling factor, and s is the scale factor.

[0035] Preferably, in step S2), the extraction of image features specifically includes:

[0036] S22) For each input pair, the image is discretized and password-encoded into a series of image-embedded password features. In addition, a category-embedded password feature is added. Then the two are concatenated to obtain the complete image embedding features. The image is then fed into L coding layers of an image autoencoder, where the image embedding features Y are extracted using the L coding layers and the spatial adapter SA.

[0037] Preferably, in step S22), similar to the text autoencoder, the image embedding feature Y passes through a LayerNorm activation layer, a spatial adapter SA, and a multi-head self-attention mechanism, and then the parameters are updated in the form of residuals, i.e.:

[0038] Y′=Y+Attention(SA(LN(Y))).

[0039] Preferably, in step S22), the updated Y′ in residual form is processed by extracting features through a LayerNorm activation layer, a fully connected layer, a spatial adapter SA, and a fully connected layer, and then updated using residual learning; that is:

[0040] Y″=Y+FC2(PA(FC1(LN(Y))));

[0041] In the formula, FC1 and FC2 represent the first and second fully connected layers, respectively.

[0042] Preferably, in step S22), the operation flow of the spatial adapter SA is as follows: first, the image-embedded password feature X is extracted from Y; then, the image-embedded password feature sequence is restored to a two-dimensional image shape, the shape being... Recorded as:

[0043] X = Seq2Im(X);

[0044] In the formula, the Seq2Im(X) operation represents reshaping a one-dimensional sequence into a two-dimensional feature map.

[0045] Preferably, in step S22), a two-dimensional depthwise separable convolution is performed on the image-embedded password feature X, and a piecewise function DY(α) is introduced to modulate the convolution result. Finally, the extracted features are updated into X in the form of residuals, i.e.:

[0046] X′=X+DY(α)·DWConv2D(X);

[0047] Here, DWConv2D refers to a convolution kernel of size 3×3×1×D.

[0048] Preferably, in step S22), the updated image-embedded password feature X′ is discretized into a password sequence, and concatenated with the category-based password embedding feature to obtain the complete spatial adapter output Y, i.e.:

[0049] Y = [c; Im2Seq(X)];

[0050] In the formula, Im2Seq(X) represents reshaping a two-dimensional feature map back into a one-dimensional sequence.

[0051] Preferably, in step S3), supervised learning is performed based on the obtained text features W″ and image features Y″, using cosine similarity loss.

[0052] The beneficial effects of this invention are as follows:

[0053] 1. In order to bridge the semantic gap between language and visual patterns, this invention proposes a text-level inductive bias by supplementing the prompt text with numerous LLM-generated descriptions to provide detailed information for each category;

[0054] 2. In order to enable the model to capture inductive bias well, this invention proposes two types of adapters for text and image encoders respectively. For text encoders, a phrase adapter is designed to explicitly explore the connections between adjacent words; for image encoders, a spatial adapter is designed to allow the model to see more local relationships and details.

[0055] 3. This invention reduces overfitting by optimizing the inductive bias; this is achieved through a novel dynamic training strategy that enables the model to learn different degrees of fit. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the framework of the generalization model of this invention;

[0057] Figure 2 This is a schematic diagram of the model-level inductive bias integration of the present invention;

[0058] Figure 3 This is a schematic diagram of the phrase adapter of the text encoder of the present invention;

[0059] Figure 4 This is a schematic diagram of the spatial adapter of the image encoder of the present invention. Detailed Implementation

[0060] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0061] This embodiment provides a diversity inductive bias learning method for generalizable large models, including the following steps:

[0062] S1) Construct an input pair image with a complete prompt description based on the training sample set;

[0063] In this embodiment, constructing an input pair image with a complete prompt description includes the following steps:

[0064] S11) For each image-label input pair, extract its corresponding category label CN, and construct the category description T according to the hand-provided description format. hc (CN): = One sheet <cn>image;

[0065] S12) Generate K custom prompt descriptions T for the corresponding category word CN using the ChatGPT3 large language model. ct (CN) i ;

[0066] S13), Category descriptions T constructed by hand-drawn prompts hc (CN) and Custom Hint Description T ct (CN) i By using a concatenation operator to concatenate and integrate the input images, a complete prompt description T(CN) corresponding to the input image is obtained. i ,Right now:

[0067] T(CN) i =[T hc (CN); T ct (CN) i ].

[0068] S2) Extract image and text features from the input pairs using a generalized large model;

[0069] In this embodiment, the framework diagram of the generalized large model is shown below. Figure 1 As shown; the generalized large model includes a text encoder and an image encoder; the text encoder and the image encoder have similar structures, but their parameters are not shared; and both the text encoder and the image encoder include multiple encoding layers, and the text encoder also includes a text embedding layer and a text mapping layer; the image encoder also includes a tile embedding layer and an image mapping layer.

[0070] In this preferred embodiment, the generalized large model includes three inductive biases: text-level inductive bias; and custom text generated by LLM to supplement "a...". <cn>Photos;

[0071] Model-level inductive bias; learns to adjust hard inductive bias for text and image encoders through a designed adapter;

[0072] To optimize inductive bias, in the adapter, the hyperparameter α controls the acquisition of task-specific knowledge; during training, DY(α) is dynamically changed to allow the model to learn different levels of task-specific knowledge.

[0073] In a preferred embodiment, the coding layer includes a multi-head attention module and a feedforward layer, and the model is enriched by layer normalization and inductive bias after the first fully connected layer in the multi-head attention module and the feedforward layer.

[0074] In a preferred embodiment, a corresponding phrase adapter is inserted in each coding layer of the text encoder; a corresponding spatial adapter is inserted in each coding layer of the image encoder. The results of the phrase adapter and the spatial adapter can be found in [reference needed]. Figure 3 and 4 As shown.

[0075] As a preferred embodiment, such as Figure 1 As shown, the extraction of text features specifically includes the following steps:

[0076] S211) Transform the complete prompt word description of each input pair into a text embedding feature W; the expression of the text embedding feature W is:

[0077]

[0078] In the formula, w represents each token in the text, M is the total number of tokens, and D is the feature dimension of the token;

[0079] (S212) Subsequently, the text embedding feature W is fed into L coding layers of the text autoencoder. In each coding layer of the text autoencoder, the text embedding feature W is first processed by the LayerNorm activation layer of the coding layer, and then fed into the phrase adapter PA, that is:

[0080] PA(LN(W))=W+DY(α)·DWConv1D(W);

[0081] In the formula, DWConv1D refers to a one-dimensional depthwise separable convolution with a kernel size of 3×1×D; DY(α) is a piecewise function with hyperparameters α and s; PA represents the phrase adapter PA; and LN(W) represents the output of the LayerNorm activation layer.

[0082] The piecewise function DY(α) is expressed as:

[0083]

[0084] In the formula, prob is obtained by random sampling from a uniform distribution [0,1].

[0085] S213) Perform standard multi-head self-attention operation on the output of the phrase adapter PA, and then update the original text embedding features W as residuals, i.e.:

[0086] W′=W+Attention(PA(LN(W)));

[0087] S214) The updated feature W′ from step S213) is further processed through LayerNorm activation layer, fully connected layer, phrase adapter, and fully connected layer to extract features, and then updated to the original text embedding feature W using residual learning; that is:

[0088] W″=W+FC2(PA(FC1(LN(W))));

[0089] In the formula, FC1 and FC2 represent the first and second fully connected layers, respectively;

[0090] (S215) Repeat steps S212) to S214) until all L encoding layers of the text autoencoder have been passed.

[0091] In a preferred embodiment, the extraction of image features specifically includes the following steps:

[0092] S221) For each input pair, the image is discretized and password-encoded into a series of image-embedded password features. In addition, a category-embedded password feature is added. Then the two are concatenated to obtain the complete image embedding features.

[0093] S222) The complete image embedding feature Y is fed into L coding layers of the image autoencoder. In each coding layer of the image autoencoder, the complete image embedding feature Y passes through a LayerNorm activation layer, a spatial adapter (SA), and a multi-head self-attention mechanism, and then the parameters are updated in the form of residuals, i.e.:

[0094] Y′=Y+Attention(SA(LN(Y)));

[0095] S223) After updating Y′ in residual form, features are extracted through LayerNorm activation layer, fully connected layer, spatial adapter SA, and fully connected layer, and updated in the form of residual learning; that is:

[0096] Y″=Y+FC2(PA(FC1(LN(Y))));

[0097] In the formula, FC1 and FC2 represent the first and second fully connected layers, respectively.

[0098] In a preferred embodiment, in step S222), the operation flow of the spatial adapter SA is as follows: first, the image-embedded password feature X is extracted from Y; then, the image-embedded password feature sequence is restored to a two-dimensional image shape, the shape being... Recorded as:

[0099] X = Seq2Im(X);

[0100] In the formula, the Seq2Im(X) operation represents reshaping a one-dimensional sequence into a two-dimensional feature map.

[0101] In a preferred embodiment, in step S222), a two-dimensional depthwise separable convolution is performed on the image-embedded password feature X, and a piecewise function DY(α) is introduced to modulate the convolution result. Finally, the extracted features are updated into X in the form of residuals, i.e.:

[0102] X′=X+DY(α)·DWConv2D(X);

[0103] Wherein, DWConv2D refers to a convolution kernel of size 3×3×1×D;

[0104] In a preferred embodiment, in step S222), the updated image-embedded password feature X′ is discretized into a password sequence, and concatenated with the category-based password embedding feature to obtain the complete spatial adapter output Y, i.e.:

[0105] Y = [c; Im2Seq(X)];

[0106] In the formula, Im2Seq(X) represents reshaping a two-dimensional feature map back into a one-dimensional sequence.

[0107] S3) For the same pair of inputs, supervised training is performed using cosine similarity loss.

[0108] This embodiment uses the obtained text features W″ and image features Y″, and performs supervised learning using cosine similarity loss.

[0109] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.< / cn> < / cn> < / cn> < / cn>

Claims

1. A diversity induction bias learning method for a generalizable large model, characterized in that, The method comprises the following steps: S1), constructing an input pair of complete prompt description according to a training sample set; S2), extracting image features and text features of the input pair by using a general large model; The general large model comprises three kinds of inductive bias, namely text-level inductive bias, model-level inductive bias and optimization-level inductive bias; Among them, the text-level inductive bias supplements the category description with the customized text generated by the LLM; The model-level inductive bias learns the hard inductive bias for adjusting the text and image encoders through the designed adapter; Optimizing the bias-variance trade-off for adapting the model in the adapter by hyper-parameters controlling the fusion ratio of task-specific knowledge and original model knowledge; during the training process, by dynamically changing to let the model learn different degrees of task-specific knowledge; wherein, for a hyper-parameter , s is a hyper-parameter is a scale factor; A corresponding phrase adapter is inserted into each encoding layer of the text encoder; and a corresponding space adapter is inserted into each encoding layer of the image encoder; S3), for the same input pair, supervised training is performed using cosine similarity loss.

2. The diversity inductive bias learning method for a generalizable large model according to claim 1, characterized in that: In step S1), the following steps are specifically included: S11), for each image-label input pair, extract its corresponding class label and construct class descriptions according to the hand-crafted description forms ; S12)、 through ChatGPT3 large language model for corresponding category word generate Custom prompt description ; S13), the category description is constructed by the form of manual prompt description with the custom prompt description The complete prompt description corresponding to the input image is obtained by splicing integration with the splicing operator That is: 。 3. The diversity inductive bias learning method for a generalizable large model according to claim 1, wherein: In step S2), the general large model comprises a text encoder and an image encoder; the text encoder and the image encoder are similar in structure, but the parameters are not shared with each other; and the text encoder and the image encoder each comprise a plurality of encoding layers, and the text encoder further comprises a text embedding layer and a text mapping layer; the image encoder further comprises a tile embedding layer and an image mapping layer.

4. The diversity inductive bias learning method for a generalizable large model according to claim 1, wherein: In step S2), the text feature extraction is to convert the complete prompt word description of each input pair into a text embedding feature , and then send the text embedding feature into an encoding layer of a text auto-encoder, and perform feature extraction on the text embedding feature through the encoding layer and a phrase adapter.

5. The diversity inductive bias learning method for a generalizable large model according to claim 4, characterized in that: In step S2), the extraction of the text features specifically comprises the following steps: S211) Transform the complete prompt word description of each input pair into text embedding features. The text embedding features mentioned above The expression is: ； In the formula, wherein, M represents each token in the text, and D represents the total number of tokens and the feature dimension of the token. S212)、in each encoding layer of the text autoencoder, the text embedding feature The LayerNorm activation layer of the encoding layer is processed first, and then sent to the phrase adapter In the middle, namely: ； In the formula, is a one-dimensional depth separable convolution with a kernel size of ; is a piecewise function with a hyperparameter , s, wherein the hyperparameter is a scale factor; s is a scale factor; represents a phrase adapter , represents the output processed by the LayerNorm activation layer. S213), performing a standard multi-head self-attention operation on the output of the phrase adapter , followed by updating in residual form to the original text embedding features , i.e.: ​ ； S214), updating the updated features Further, the features are extracted through a LayerNorm activation layer, a fully connected layer, a phrase adapter, and a fully connected layer, and are updated to the original text embedding features in the form of residual learning In the middle; that is: ； In the formula, FC1 and FC2 represent the first and second fully connected layers, respectively; S215), repeat steps S212) ~ S214) until all the text auto-encoders of encoding layers are passed through.

6. The diversity inductive bias learning method for a generalizable large model according to claim 5, characterized in that: In step S212, the is expressed as: ； wherein from a uniform distribution obtained by random sampling.

7. The diversity inductive bias learning method for a generalizable large model according to claim 6, characterized in that: In step S2), the extraction of the image features specifically comprises the following steps: S221)、For each input pair, discretize and hash the images into a series of image embedding hash features ; and on this basis, add a category embedding hash feature ; and then splice the two to obtain the complete image embedding feature: ； S222), embedding the complete image into features into the image autoencoder into the image autoencoder, in each encoding layer of the image autoencoder, the complete image is embedded into features a LayerNorm activation layer, a spatial adapter SA, a multi-head self-attention mechanism, and then updated in a residual form, that is: ； S223), update the residual form The features are extracted by the LayerNorm activation layer, the fully connected layer, the spatial adapter SA, and the fully connected layer, and are updated in the form of residual learning; that is: ； In the formula, FC3 and FC4 represent the third and fourth fully connected layers, respectively.

8. The diversity inductive bias learning method for a generalizable large model according to claim 7, characterized in that: In step S222, the operation flow of the space adapter SA is: firstly, the image is embedded into the entry feature From , and then the image embedded into the entry feature sequence is restored to a two-dimensional image shape, which is , denoted as: ； In the formula, Operation denotes reshaping a one-dimensional sequence into a two-dimensional feature map; A two-dimensional depth separable convolution is performed on the image embedding entry feature X, and a piecewise function is introduced The convolution result is modulated, and finally the extracted feature is updated to in the form of a residual, that is: ； wherein is a convolution kernel of size ; Embedding updated image into password feature Discretize into password sequence, concatenate with category password embedding feature to get complete spatial adapter output Y, i.e.: ； In the formula, represents reshaping the two-dimensional feature map back to a one-dimensional sequence.

Images