Knowledge distillation method and device for image mask pre-training

Abstract

The present invention discloses a knowledge distillation method and apparatus for image mask pre-training. The method comprises the following steps: S01. Obtaining a sample set of image data to be learned, cutting it into uniform, non-overlapping image blocks, and extracting a specified proportion as visible blocks; S02. Inputting the visible blocks into a student model to generate a low-level semantic representation of the visible blocks and a high-level semantic representation of the entire image; S03. Generating a first image tag based on the high-level semantic representation, and calculating the high-level semantic representation loss compared with the second image tag generated by the first teacher model; S04. Generating a low-level semantic representation of the visible blocks by the second teacher model, and calculating the low-level semantic representation loss compared with the low-level semantic representation generated by the student model; S05. Establishing an objective function for image mask knowledge distillation based on the low-level semantic representation loss and the high-level semantic representation loss. The present invention has the advantages of simple implementation, low computational complexity and latency, and the ability to simultaneously learn low-level and high-level semantic representations.

Description

Technical Field

[0001] The present invention relates to the field of image representation learning technology, and in particular to a knowledge distillation method and device for image mask pre-training. Background Art

[0002] Pretraining techniques are the primary method for learning prior knowledge about images. Mask-based pretraining is a novel approach for learning image representations. Compared to autoencoders, mask pretraining offers stronger data representation. Contrastive learning relies heavily on data augmentation strategies, large numbers of negative samples, and low-scalability clustering methods. Masked image modeling better meets the label-free and independent requirements of unsupervised learning, thus facilitating image representation learning during the pretraining phase.

[0003] In the existing technology, image mask pre-training mainly adopts the mask image modeling method based on pixel reconstruction and the mask image modeling method based on token reconstruction. The mask image modeling method based on pixel reconstruction uses the reconstructed pixel values ​​as the pre-training target and introduces an asymmetric encoder-decoder structure, in which the encoder only learns the visible blocks. The visible block representation output by the encoder and the initialized mask block representation are input to the decoder together to recover the image content of the masked area. The mask image modeling method based on token reconstruction has a different pre-training target from the mask image modeling method based on pixel reconstruction. It infers the token of the mask block by obtaining the token of the visible block through the variational autoencoder (VAE).

[0004] However, the above-mentioned mask image modeling method in the prior art has the following problems:

[0005] 1. Limited learning capabilities. Pixel reconstruction-based mask image modeling fits the pixel values ​​between the reconstructed image blocks and the original image blocks, so the learned image representation is low-level semantic. Token reconstruction-based mask image modeling fits the image tokens generated by the strain autoencoder between the reconstructed image blocks and the real image blocks, so the learned image representation is high-level semantic. Neither approach can simultaneously learn both low-level and high-level semantic representations.

[0006] 2. The model size, latency, and computational complexity are large. During the pre-training process, in order to understand the data characteristics and learn effective data representations, and then build a deeper and wider network architecture, higher latency and computational complexity are inevitable. Knowledge distillation can effectively reduce the model size and latency, but the image knowledge distillation method in the existing technology requires the input of a complete image, while the mask pre-training model only inputs partial images during the training process, and there is redundancy between images, resulting in similarities and differences between the learned image block representations. Therefore, traditional image knowledge distillation methods are not suitable for distilling the knowledge of mask pre-training models. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a knowledge distillation method and device for image mask pre-training, which has a simple implementation method, low computational complexity and delay, and can simultaneously learn low-level semantic representation and high-level semantic representation of images.

[0008] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0009] A knowledge distillation method for image mask pre-training, comprising the following steps:

[0010] S01 obtains a sample set of image data to be learned, cuts each image data sample in the sample set of image data to be learned into image blocks evenly without overlapping, and extracts a specified proportion of image blocks as visible blocks;

[0011] S02. Input the extracted visible blocks into the student model to generate a low-level semantic representation of the visible blocks and a high-level semantic representation of the entire image;

[0012] S03. Generating a first image tag according to the high-level semantic representation, and generating a second image tag for the entire image using a first teacher model, and calculating a high-level semantic representation loss according to the first image tag and the second image tag, wherein the first teacher model is an image mask model based on Token Reconstruction;

[0013] S04. Input the visible block into a second teacher model to generate a low-level semantic representation of the visible block, and calculate the low-level semantic representation loss based on the low-level semantic representation generated by the second teacher model and the low-level semantic representation generated by the student model in step S02, where the second teacher model is an image mask model based on pixel reconstruction;

[0014] S05. Based on the low-level semantic representation loss and the high-level semantic representation loss, establish an objective function of image mask knowledge distillation to complete image mask knowledge distillation.

[0015] Furthermore, the step S02 includes:

[0016] S201. Convert the extracted visible blocks into a two-dimensional feature matrix, and expand the feature dimension of each image block through a linear mapping layer to obtain visible block features;

[0017] S202. Initialize an additional cls token and concatenate it with the visible block features into a feature matrix;

[0018] S203. The combined feature matrix and the corresponding position information are used as input to the student model;

[0019] S204. The student model generates a visible block representation according to the input, where the visible block representation includes a low-level semantic representation and a high-level semantic representation of the entire image.

[0020] Furthermore, the student model is a Transformer-Encoder module, and the visible block representation is generated by forward propagation of the Transformer-Encoder module, including:

[0021] Extract potential features through multi-head attention layer and Add&Norm layer:

[0022] e′ l =LayerNorm(e l +MultiHeadAttention(e l ))

[0023] Among them, e l is the input of the l-th layer Transformer-Encoder module, e′ l is the potential feature extracted from the lth layer, MultiHeadAttention(·) is the multi-head attention layer, LayerNorm(·) is the Layer Normalization layer in the Add&Norm layer used to normalize the hidden layer in the neural network to a standard normal distribution, e l +MultiHeadAttention(e l ) corresponds to the Add layer in the Add&Norm layer, and the multi-head attention layer function MultiHeadAttention(e l ) multiple self-attention functions Self-Attention(e l )’s output features are spliced ​​together;

[0024] The potential feature e′ extracted from the lth layer l Input to the Feed Forward layer and Add&Norm layer to extract semantic features:

[0025] e l+1 =LayerNorm(e′ l +FeedForward(e′ l ))

[0026] FeedForward(·) is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. It is expressed by the following formula:

[0027] max(0,e′ l W l1 +b l1 )W l2 +b l2

[0028] Among them, W l1 and W l2 is the feature matrix to be trained; b l1 and b l2 is the feature bias to be trained;

[0029] After completing the feature extraction of all visible blocks, we can finally obtain the low-level semantic information about each visible block and the high-level semantic information of the entire image.

[0030] Furthermore, each of the self-attention functions Self-Attention(e l )include:

[0031] Q l =e l *W l,Q

[0032] K l =e l *W l,K

[0033] V l =e l *W l,V

[0034]

[0035] Among them, W l,Q 、W l,K 、W l,V are the three feature matrices to be trained in the lth Encoder module, used to train the feature e lPerform linear mapping operation, d is the matrix Q l and K l The number of columns, different self-attention functions Self-Attention(e l ) corresponds to different linear mapping matrices W l,Q 、W l,K 、W l,V .

[0036] Furthermore, in step S03, an image token modeling head is used to generate the first image token of the high-level semantic representation. The image token modeling head includes a linear layer and a softmax layer, and the expression is:

[0037]

[0038] in, s H v [0]+b s The corresponding linear layer, s is the feature matrix to be trained, b s is the feature bias to be trained.

[0039] 6. The knowledge distillation method for image mask pre-training according to claim 5, characterized in that in step S03, a high-level semantic representation loss function is constructed using a cross entropy function to calculate the high-level semantic representation loss, and the high-level semantic representation loss function is expressed as:

[0040]

[0041] Among them, loss s () is the cross entropy function, represents the first image tag generated according to the high-level semantic representation, Y s Represents the second image labeling generated by the first teacher model.

[0042] Furthermore, in step S04, a low-level semantic representation loss function is constructed using a mean square error (MSE) function to calculate the low-level semantic representation loss. The low-level semantic representation loss function is expressed as:

[0043]

[0044] in, represents the low-level semantic representation generated by the student model, Y l Represents the low-level semantic representation generated by the second teacher model, loss l () is the mean square error function, Y li ∈Yl represents the low-level feature representation of each visible block i, H v [] represents the information of the i-th visible block, H v [ ] includes high-level semantic representation as well as low-level semantic representation.

[0045] Furthermore, in step S05, the low-level semantic representation loss and the high-level semantic representation loss are added together to establish an objective function for image mask knowledge distillation, namely:

[0046]

[0047] in, represents the high-level semantic representation loss, represents the first image tag generated according to the high-level semantic representation, Y s represents the second image tag generated by the first teacher model, Represents the low-level semantic representation loss, loss s () is the cross entropy function, represents the low-level semantic representation generated by the student model, Y l Represents the low-level semantic representation generated by the second teacher model, loss l () is the mean square error function.

[0048] Furthermore, step S01 also includes converting, cropping, and coloring the original image data sample set to expand the sample size, and then converting each sample to a uniform size to obtain the image data sample set to be learned, and cutting each image data sample in the image data sample set to be learned into image blocks of the same size and number without overlap.

[0049] A knowledge distillation device for image mask pre-training, comprising a processor and a memory, wherein the memory is used to store a computer program, and wherein the processor is used to execute the computer program to perform the above-mentioned method.

[0050] Compared with the prior art, the advantages of the present invention are: the present invention combines two mask image modeling methods based on Pixel Reconstruction and based on Token Reconstruction, and realizes image mask pre-training based on the knowledge distillation architecture consisting of two teacher models and one student model. First, the image dataset samples are cut into picture blocks by uniform non-overlapping, and a part of the image blocks are extracted as visible blocks according to a certain proportion. The visible blocks are then used as the input of the student model to generate low-level semantic representations of the visible blocks and high-level semantic representations of the entire image. The high-level semantic representation is then labeled by the image labeling modeling head, and a high-level semantic representation loss is established with the label generated by the image mask model based on Token Reconstruction. At the same time, a low-level semantic representation loss is established with the low-level semantic representation generated by the image mask model based on Pixel Reconstruction. Finally, the low-level semantic representation loss and the high-level semantic representation loss are combined to jointly establish the objective function of mask knowledge distillation, which can fully combine the image mask model based on Token Reconstruction with the image mask model based on Pixel Reconstruction based on the knowledge distillation architecture. The advantages of the two models of image mask model and reconstruction enable the simultaneous learning of low-level and high-level semantic representations, while effectively reducing the model size and latency and reducing the amount of computation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the implementation process of the knowledge distillation method for image mask pre-training in this embodiment.

[0052] Figure 2 Schematic diagram of the network architecture for masked knowledge distillation for low-level and high-level semantic learning in this embodiment.

[0053] Figure 3 It is a structural diagram of the Student model used in this embodiment.

[0054] Figure 4 Schematic diagram of the network architecture of the first teacher model (VAE) used in this embodiment.

[0055] Figure 5 Schematic diagram of the network architecture of the second teacher model (MAE) used in this embodiment. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0057] As shown in the present disclosure, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not specifically refer to the singular and may also include the plural. The words "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprising" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0058] The traditional mask image modeling method based on pixel reconstruction can learn low-level semantic representations, but cannot learn high-level semantic representations, while the mask image modeling method based on token reconstruction can learn high-level semantic representations but cannot learn low-level semantic representations. The present invention combines the advantages of the two mask image modeling methods based on pixel reconstruction and token reconstruction, and implements image mask pre-training based on a knowledge distillation architecture consisting of two teacher models and one student model, so that it can learn both the low-level semantic representation of the image and the high-level semantic representation of the image. At the same time, the knowledge distillation architecture can also effectively reduce the model size, computational complexity and latency.

[0059] like Figure 1 、 2 As shown, the steps of the knowledge distillation method for image mask pre-training in this embodiment include:

[0060] S01. Obtain a sample set of image data to be learned, cut each image data sample in the sample set into image blocks evenly and without overlap, and extract a specified proportion of image blocks as visible blocks.

[0061] In this embodiment, after obtaining the original image dataset ImageNet-1K, the original image data samples are first preprocessed by rotation, cropping, coloring, etc. to increase the sample size of the dataset, and then converted into a uniform size (such as 224×224) to obtain the ImageNet-1K-1 dataset, an image data sample set to be learned.

[0062] Each image block representation learned by mask image modeling is highly unique and contains part of the global information of the image. In knowledge distillation, if the input of the student model is an image block, the semantic representation of the image block can be learned. In this embodiment, each image data sample X in the image dataset ImageNet-1K-1 is cut into uniformly sized patches by uniform non-overlapping, and a portion of the image blocks is randomly extracted at a specified ratio as the visible block X. v , X v ∈ X. For example, each sample X of the learning image dataset ImageNet-1K-1 can be treated by uniformly non-overlapping cutting into the same size 16×16 and the same number of picture blocks.

[0063] S02. Input the extracted visible blocks into the student model to generate a low-level semantic representation of the visible blocks and a high-level semantic representation of the entire image.

[0064] In this embodiment, the visible blocks are first v The image is converted into a two-dimensional feature matrix, and then the linear mapping layer is used to expand the feature dimension of each image block to obtain the visible block feature. An additional cls token is then initialized and added to the input feature. After concatenating it with the visible block feature and adding the corresponding position information, it is used as the input feature of the student model. After the student model, the output includes a low-level semantic representation and a high-level semantic representation of the entire image. The specific steps are as follows:

[0065] S201. Visible blocks v Each image block i ∈X v The RGB three channels are expanded to 1024 dimensions through the linear layer to obtain the visible block feature x′ i , that is, x′ i =x i E;

[0066] S202. Initialize additional cls token x cla , and the visible block feature x′ i Splice into a feature matrix, that is, [x cla ,x′1,…,x′ i ,…,x′ I ];

[0067] S203. The feature matrix [x cla ,x′1,…,x′ i ,…,x′ I] plus the corresponding position information as the input of the student model, that is, e0 = [x cla ,x′1,…,x′ i ,…,x′ I ]+E pos ;

[0068] S204. The obtained e0 is respectively passed through the student model f e Generate the representation H corresponding to the visible block v ,Right now:

[0069] H v =f e (X v ) (1)

[0070] Among them, H v [0] belongs to high-level semantic representation, which represents the semantic information of the entire image. v [1:I] belongs to the low-level semantic representation, H v [] represents the information of the i-th visible block.

[0071] In this embodiment, the student model adopts the Transformer-Encoder module. Figure 3 As shown in Figure 1, the Transformer-Encoder module extracts the semantic features of the visible block through the Add&Norm layer, the Feed Forward layer, and the multi-head attention layer, with the initial input being e0. The specific steps of generating the visible block representation through the forward propagation of the Transformer-Encoder module include:

[0072] (1) Extracting latent features through the multi-head attention layer and Add&Norm layer:

[0073] e′ l =LayerNorm(e l +MultiHeadAttention(e l )) (2)

[0074] Among them, e l is the input of the l-th layer Transformer-Encoder module, e′ l is the potential feature extracted from the lth layer, MultiHeadAttention(·) is the multi-head attention layer, LayerNorm(·) is the LayerNormalization layer in the Add&Norm layer used to normalize the hidden layer in the neural network to the standard normal distribution, e l +MultiHeadAttention(e l) corresponds to the Add layer in the Add&Norm layer. Add is a residual connection. The multi-head attention layer function MultiHeadAttention(e l ) multiple self-attention functions Self-Attention(e l )’s output features are concatenated together.

[0075] Each self-attention function Self-Attention(e l ) specifically include:

[0076]

[0077]

[0078] Among them, W l,Q 、W l,K 、W l,V are the three feature matrices to be trained in the lth Encoder module, used to train the feature e l Perform linear mapping operation, d is the matrix Q l and K l The number of columns, different self-attention functions Self-Attention(e l ) corresponds to different linear mapping matrices W l,Q 、W l,K 、W l,V , to indicate that the model judges the degree of correlation between image features from different perspectives.

[0079] (2) The potential feature e′ extracted from the lth layer l Input to the Feed Forward layer and Add&Norm layer to extract semantic features:

[0080] e l+1 =LayerNorm(e′ l +FeedForward(e′ l )) (5)

[0081] FeedForward(·) is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. It is expressed by the following formula:

[0082] max(0,e′ l W l1 +b l1 )W l2 +b l2 (6)

[0083] Among them, W l1 and W l2is the feature matrix to be trained; b l1 and b l2 is the feature bias to be trained.

[0084] (3) Repeat steps (1) to (3) for a total of Num times, where Num is the set number of times, and finally obtain the low-level semantic information about each visible block and the high-level semantic information of the entire image.

[0085] This embodiment uses the Transformer-Encoder module architecture as a student model to perform feature extraction on visible blocks, and can efficiently extract low-level semantic information of the visible blocks and high-level semantic information of the entire image.

[0086] S03. Generate a first image tag based on the high-level semantic representation, and generate a second image tag for the entire image using the first teacher model, and calculate the high-level semantic representation loss based on the first image tag and the second image tag. The first teacher model is an image mask model based on Token Reconstruction.

[0087] The high-level semantics is represented by H v [0] Generate the token matrix corresponding to the entire image through the image tag modeling head This is the first image tag generated based on the high-level semantic representation. The image tag modeling head specifically includes a linear layer and a Softmax layer, and the expression is:

[0088]

[0089] in, s H v [0]+b s The corresponding linear layer, s is the feature matrix to be trained, b s is the feature bias to be trained.

[0090] The original image is input into the trained first teacher model (VAE model Teacher 1). The first teacher model is an image mask model based on Token Reconstruction, that is, the token of the visible block is obtained by the variational autoencoder VAE to infer the token of the mask block, such as Figure 4 As shown, the first teacher model generates the token matrix Y corresponding to the entire image s = es (), which is the second image label generated using the first teacher model.

[0091] In a specific application embodiment, the original image is input into the trained Token Reconstruction-based image mask model Teacher 1 (VAE model) to generate a 14×14 token matrix, where each matrix element represents a high-level semantic representation of the 16×16 image block at the corresponding position, and each matrix element value is a number between [1,8192].

[0092] In this embodiment, a high-level semantic representation loss function is constructed using a cross entropy function to calculate the high-level semantic representation loss. The high-level semantic representation loss function is expressed as:

[0093]

[0094] Among them, loss s () is the cross entropy function, represents the first image tag generated according to the high-level semantic representation, Y s Represents the second image labeling generated by the first teacher model.

[0095] S04. Input the visible block into the second teacher model to generate a low-level semantic representation of the visible block, and calculate the low-level semantic representation loss based on the low-level semantic representation generated by the second teacher model and the low-level semantic representation generated by the student model in step S02. The second teacher model is an image mask model based on Pixel Reconstruction.

[0096] Make visible blocks v Input into the trained Pixel Reconstruction-based image mask model Teacher2 (second teacher model) to generate a low-level feature representation Y for each visible block li ∈Y l = el ( v ).like Figure 5 As shown, the image mask model based on Pixel Reconstruction takes the reconstructed pixel value as the pre-training target and introduces an asymmetric encoder-decoder structure, in which the encoder only learns the visible blocks, and the visible block representation output by the encoder and the initialized mask block representation are input together into the decoder to restore the image content of the masked area. In this embodiment, the image mask model based on Pixel Reconstruction specifically adopts the Masked AutoEncoders (MAE) model.

[0097] In this embodiment, the mean square error function is used to construct a high-level semantic representation loss function, namely:

[0098]

[0099] in, represents the low-level semantic representation generated by the student model, Y l Represents the low-level semantic representation generated by the second teacher model, loss l () is the mean square error function.

[0100] S05. Based on the low-level semantic representation loss and the high-level semantic representation loss, an objective function of image mask knowledge distillation is established to complete image mask knowledge distillation.

[0101] The low-level semantic representation loss and the high-level semantic representation loss are added together to establish the objective function of image mask knowledge distillation, namely:

[0102]

[0103] This embodiment establishes the objective function of image mask knowledge distillation by fusing the low-level semantic representation loss and the high-level semantic representation loss, which can fully integrate the advantages of the two image mask models, making it possible to learn the low-level semantic representation and high-level semantic representation of the image, effectively improving the image learning ability.

[0104] After completing the pre-training of the model, keep the Student model f e The network parameters in the

[15] are used as input for all image patches to obtain a high-level semantic representation of the entire image and a low-level semantic representation of each patch within the image, which serves as the image's semantic information. When performing downstream tasks, the image semantic features are further fine-tuned using linear probes or fine probes based on the downstream task objectives. The performance of visible patches at different scales is then evaluated based on the results of the downstream tasks.

[0105] The present invention cuts the image dataset samples into image blocks (patches) uniformly and non-overlappingly, extracts a part of the image blocks as visible blocks in a certain proportion, and then uses the visible blocks as the input of the student model to generate a low-level semantic representation of the visible blocks and a high-level semantic representation of the entire image. The high-level semantic representation is then tokenized through the image tag modeling head, and a high-level semantic representation loss is established with the token generated by the image mask model based on Token Reconstruction (the first teacher model). At the same time, the low-level semantic representation is established with the low-level semantic representation generated by the image mask model based on Pixel Reconstruction (the second teacher model). Finally, the low-level semantic representation loss and the high-level semantic representation loss are combined to jointly establish the objective function of mask knowledge distillation. Based on the knowledge distillation architecture, the advantages of the image mask model based on Token Reconstruction and the image mask model based on Pixel Reconstruction can be fully combined, so that low-level and high-level semantic representations can be learned simultaneously, while effectively reducing the model scale and latency as well as the amount of computation.

[0106] This embodiment also provides a knowledge distillation device for image mask pre-training, including a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0107] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A knowledge distillation method for image mask pre-training, characterized in that the steps include: S01 obtains a sample set of image data to be learned, cuts each image data sample in the sample set of image data to be learned into image blocks evenly without overlapping, and extracts a specified proportion of image blocks as visible blocks; S02. Input the extracted visible blocks into the student model to generate a low-level semantic representation of the visible blocks and a high-level semantic representation of the entire image; S03. Generating a first image tag according to the high-level semantic representation, and generating a second image tag for the entire image using a first teacher model, and calculating a high-level semantic representation loss according to the first image tag and the second image tag, wherein the first teacher model is an image mask model based on Token Reconstruction; S04. Input the visible block into a second teacher model to generate a low-level semantic representation of the visible block, and calculate the low-level semantic representation loss based on the low-level semantic representation generated by the second teacher model and the low-level semantic representation generated by the student model in step S02, where the second teacher model is an image mask model based on pixel reconstruction; S05. Based on the low-level semantic representation loss and the high-level semantic representation loss, establish an objective function of image mask knowledge distillation to complete image mask knowledge distillation.

2. The knowledge distillation method for image mask pre-training according to claim 1, characterized in that The step S02 includes: S201. Convert the extracted visible blocks into a two-dimensional feature matrix, and expand the feature dimension of each image block through a linear mapping layer to obtain visible block features; S202. Initialize an additional cls token and concatenate it with the visible block features into a feature matrix; S203. The combined feature matrix and the corresponding position information are used as input to the student model; S204. The student model generates a visible block representation according to the input, where the visible block representation includes a low-level semantic representation and a high-level semantic representation of the entire image.

3. The knowledge distillation method for image mask pre-training according to claim 1, characterized in that The student model is a Transformer-Encoder module, and the visible block representation is generated by forward propagation of the Transformer-Encoder module, including: Extract potential features through multi-head attention layer and Add&Norm layer: in l ′=LayerNorm(e l +MultiHeadAttention(e l )) Among them, e l is the input of the l-th layer Transformer-Encoder module, e l ′ is the potential feature extracted from the lth layer, MultiHeadAttention(·) is the multi-head attention layer, LayerNorm(·) is the LayerNormalization layer in the Add&Norm layer used to normalize the hidden layer in the neural network to a standard normal distribution, e l +MultiHeadAttention(e l ) corresponds to the Add layer in the Add&Norm layer, and the multi-head attention layer function MultiHeadAttention(e l ) multiple self-attention functions Self-Attention(e l )’s output features are spliced ​​together; The latent feature e extracted from the lth layer l ’ is input to the Feed Forward layer and Add&Norm layer to extract semantic features: in l+1 =LayerNorm(e l ′+FeedForward(e l ′)) FeedForward(·) is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. It is expressed by the following formula: max(0,e l ′W l1 +b l1 )W l2 +b l2 Among them, W l1 and W l2 is the feature matrix to be trained; b l1 and b l2 is the feature bias to be trained; After completing the feature extraction of all visible blocks, we can finally obtain the low-level semantic information about each visible block and the high-level semantic information of the entire image.

4. The knowledge distillation method for image mask pre-training according to claim 3, characterized in that Each of the self-attention functions Self-Attention(e l )include: Q l =e l *W l,Q K l =e l *IN l,K V l s.e l *W l,V Among them, W l,Q 、W l,K 、W l,V are the three feature matrices to be trained in the lth Encoder module, used to train the feature e l Perform linear mapping operation, d is the matrix Q l and K l The number of columns, different self-attention functions Self-Attention(e l ) corresponds to different linear mapping matrices W l,Q 、W l,K 、W l,V .

5. The knowledge distillation method for image mask pre-training according to claim 1, characterized in that In step S03, the first image tag of the high-level semantic representation is generated using an image tag modeling head, wherein the image tag modeling head includes a linear layer and a softmax layer, and the expression is: Among them, W s H v [0]+b s Corresponding to the linear layer, W s is the feature matrix to be trained, b s is the feature bias to be trained, H v [0] indicates a high-level semantic representation.

6. The knowledge distillation method for image mask pre-training according to claim 5, characterized in that In step S03, a high-level semantic representation loss function is constructed using a cross entropy function to calculate the high-level semantic representation loss. The high-level semantic representation loss function is expressed as: Among them, loss s () is the cross entropy function, represents the first image tag generated according to the high-level semantic representation, Y s Represents the second image labeling generated by the first teacher model.

7. The knowledge distillation method for image mask pre-training according to any one of claims 1 to 6, characterized in that: In step S04, a low-level semantic representation loss function is constructed using a mean square error function to calculate the low-level semantic representation loss. The low-level semantic representation loss function is expressed as: in, represents the low-level semantic representation generated by the student model, Y l Represents the low-level semantic representation generated by the second teacher model, loss l () is the mean square error function, Y li ∈Y l represents the low-level feature representation of each visible block i, H v [i] represents the information of the i-th visible block, H v [i] Medium H v [1:I] represents the low-level semantic representation.

8. The knowledge distillation method for image mask pre-training according to any one of claims 1 to 6, characterized in that: In step S05, the low-level semantic representation loss and the high-level semantic representation loss are added together to establish an objective function for image mask knowledge distillation, namely: in, represents the high-level semantic representation loss, represents the first image tag generated according to the high-level semantic representation, Y s represents the second image tag generated by the first teacher model, Represents the low-level semantic representation loss, loss s () is the cross entropy function, represents the low-level semantic representation generated by the student model, Y l Represents the low-level semantic representation generated by the second teacher model, loss l () is the mean square error function.

9. The knowledge distillation method for image mask pre-training according to any one of claims 1 to 6, characterized in that: The step S01 also includes converting, cropping, and coloring the original image data sample set to expand the sample size, and then converting each sample to a uniform size to obtain the image data sample set to be learned, and cutting each image data sample in the image data sample set to be learned into image blocks of the same size and number evenly and without overlapping.

10. A knowledge distillation device for image mask pre-training, comprising a processor and a memory, wherein the memory is used to store a computer program, characterized in that: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

Images