An end-to-end image mask pre-training method and device

Abstract

The application discloses an end-to-end image mask pre-training method and device, and the method comprises the following steps: S1, preprocessing each data sample in an original image data set; S2, non-overlapping cutting each data sample in a to-be-processed image data set, and randomly sampling image blocks of a specified proportion as visible blocks and the rest as mask blocks; S3, inputting the visible blocks into two different encoders respectively, generating visible block representations by one of the encoders, and using the other encoder to infer mask block representations; S4, matching and sorting the mask block representations and mask block positions according to the mask blocks, and combining the visible block representations and the mask block representations to form complete original image representations after matching and sorting, and inputting the complete original image representations into a decoder to reconstruct original images. The application has the advantages of simple implementation method, strong image representation ability of the generated images, and strong learning and inference ability of the whole model.

Description

Technical Field

[0001] The present invention relates to the technical field of image representation learning, and in particular to an end-to-end image mask pre-training method and device. Background Art

[0002] Pre-training technology is the primary method for learning prior knowledge about images. Mask-based pre-training, as an image representation learning method, offers stronger data representation than autoencoders. Compared to clustering methods, which rely heavily on data augmentation strategies, large numbers of negative samples, and low scalability, its masked image modeling better meets the label-free and independent requirements of unsupervised learning, facilitating image representation learning during the pre-training phase.

[0003] Image mask pre-training in the existing art mainly adopts two methods: mask image modeling based on pixel reconstruction and mask image modeling based on token reconstruction. The mask image modeling method based on pixel reconstruction uses reconstructed pixel values ​​as the pre-training target and introduces an asymmetric encoder-decoder structure, in which the encoder only learns the visible blocks. The visible block representation output by the encoder and the initialized mask block representation are input into the decoder to recover the image content of the masked area. The mask image modeling method based on token reconstruction has a different pre-training target from the mask image modeling method based on pixel reconstruction. It infers the mask block token by obtaining the visible block token through a variational autoencoder (VAE). The VAE aims to map a complete 224×224 image into a 14×14 token matrix, where each matrix element corresponds to a 16×16 image block. In summary, the mask image modeling based on pixel reconstruction is to fit the pixel values ​​between the reconstructed image block and the original image block, while the mask image modeling based on token reconstruction is to fit the image tokens generated by the strain autoencoder corresponding to the reconstructed image block and the real image block.

[0004] However, both the mask image modeling method based on pixel reconstruction and the mask image modeling method based on token reconstruction have certain limitations, which will cause the learned image representation to be of low-level semantics or have insufficient semantics. At the same time, the mask image modeling method based on token reconstruction also needs to be highly dependent on the trained VAE model to provide pre-training targets, which limits the scalability of model learning. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides an end-to-end image mask pre-training method and device with simple implementation method, high efficiency and strong flexibility, which can enhance the semantic representation ability of the image and the overall learning and reasoning ability of the model without relying on the model to provide high-level semantic information.

[0006] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0007] An end-to-end image mask pre-training method, when the dimension of the image block representation is not 1, the pre-training method comprises the following steps:

[0008] S1. Perform preprocessing operations on each data sample in the original image dataset to obtain a dataset of images to be processed;

[0009] S2. performing non-overlapping cuts on each data sample in the image dataset to be processed, and randomly sampling a specified proportion of image blocks from each cut image block as visible blocks and the remaining image blocks as mask blocks;

[0010] S3. Input the visible blocks into two different encoders, one of which generates a visible block representation, and the other encoder is used to infer the mask block representation based on the visible block content;

[0011] S4. Match and sort the mask block representation and the mask block position according to the mask block. After matching and sorting, combine the visible block representation and the mask block representation to form a complete original image representation, which is input into the decoder for original image reconstruction.

[0012] Furthermore, in step S2, each sample X in the image data sample to be processed is cut uniformly and non-overlappingly to form image blocks of the same size and number, and a set of visible blocks X is randomly extracted according to a specified ratio. v , take the remaining image blocks as mask blocks X u , where X = X v ∪X u .

[0013] Furthermore, the steps of step S3 include:

[0014] S301. Expand the RGB channels of each image block in the visible block into a specified dimension through a linear mapping layer to obtain visible block features;

[0015] S302. Initialize an additional cls token x cla And concatenate with the visible block features to form a feature matrix, where x cla Global features used to aggregate features of all visible patches as input;

[0016] S303. The feature matrix and the corresponding position information are used as input features and are input into two encoders respectively to obtain corresponding visible block representations and mask block representations.

[0017] Furthermore, both encoders are based on the Transformer-Encoder module, and the high-level semantic features of the image blocks are extracted through the Add&Norm layer, Feed Forward layer and multi-head attention layer in the Transformer-Encoder module. The Add&Norm layer includes the Add layer and the Layer Normalization layer, wherein the Add layer is used for residual connection, and the Layer Normalization layer is used to normalize the hidden layer in the neural network to a standard normal distribution.

[0018] Furthermore, in step S3, the two encoders generate visible block representation and mask block representation based on the Transformer-Encoder module, and the steps include:

[0019] The input e of the l-th layer Transformer-Encoder module l As the input of the l+1th layer Transformer-Encoder, high-level features e are extracted through the multi-head attention layer and Add&Norm layer l ′:

[0020] e′ l =LayerNorm(e l +MultiHeadAttention(e l ))

[0021] Among them, MultiHeadAttention(·) is the multi-head attention layer, LayerNorm(·) is the LayerNormalization layer, e l +MultiHeadAttention(el ) corresponds to the Add layer;

[0022] The extracted high-level features e′ l Input to the Feed Forward layer and Add&Norm layer to extract high-level semantic features:

[0023] e l+1 =LayerNorm(e′ l +FeedForward(e′ l ))

[0024] FeedForward(·) is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. It is expressed by the following formula:

[0025] max(0,e′ l W l1 +b l1 )W l2 +b l2

[0026] Among them, W l1 and W l2 is the feature matrix to be trained; b l1 and b l2 is the feature bias to be trained;

[0027] The high-level semantic features extracted from each image block ultimately obtain the high-level semantic representation corresponding to each visible block and mask block, thereby obtaining the visible block representation and the mask block representation.

[0028] Furthermore, the multi-head attention layer function MultiHeadAttention(e l ) includes several self-attention functions Self-Attention(e l ), the multi-head attention layer MultiHeadAttention(e l ) all self-attention functions Self-Attention(e l )’s output features are spliced ​​together, and the self-attention function Self-Attention (e l )include:

[0029] Q l =e l *W l,Q

[0030] K l =e l *W l,K

[0031] V l =e l *W l,V

[0032]

[0033] Among them, W l,Q 、W l,K 、W l,V are the three feature matrices to be trained in the lth Encoder module, used to train the feature e l Perform linear mapping operation, each self-attention function Self-Attention(e l ) corresponds to the linear mapping matrix W l,Q 、W l,K 、W l,V Different, d is the matrix Q l and K l The number of columns.

[0034] Furthermore, in step S4, the mask block representation and the mask block position are matched and sorted according to the mask block by a masked embeddings matching head. The specific execution steps of the masked embeddings matching head include:

[0035] Input the mask block into the third encoder to obtain the mask block embedding space And a representation space (Embedding space) is constructed by one-to-one correspondence with the position information of the corresponding mask block. The third encoder is obtained by performing gradient sliding average on the first encoder during the pre-training process. The first encoder is the encoder used to generate the visible block representation among the two encoders. The mask block representation H u Generated by the second encoder of the two encoders;

[0036] Use mask blocks to represent H u and the mask block embedding space The elements in the corresponding two sets A are constructed u and And by transforming the set A u The elements in the two sets A are sorted so that u and The KL (Kullback-Leibler divergence) divergence between elements is minimal;

[0037] According to the sorted A u The element value corresponds to The elements in the constructed representation space search for the position information of the corresponding matching mask representation, and represent H according to the mask block obtained by the search u The position information of the mask block represents H u Sort and get the sorted mask block representation

[0038] Furthermore, in step S4, the pre-training objective function of the model is constructed according to the following formula:

[0039]

[0040] in, is the combined complete image embedding, H v is the visible block representation, g(·) represents the decoder, represents the image generated based on the learned image embedding, and MSE represents the mean squared error function used to calculate the generated image and the pixel loss between the original image X.

[0041] Furthermore, when the dimension of the image block representation is 1, the steps of the pre-training method include:

[0042] S1. Perform preprocessing operations on each data sample in the original image dataset to obtain a dataset of images to be processed;

[0043] S2. performing non-overlapping cuts on each data sample in the image dataset to be processed, and randomly dividing the cut image blocks into sets A and B;

[0044] S3. Input set A as visible blocks into the model and generate feature representation H of visible block set A. A , and infer the feature representation H of the set B as the mask block A→B ; Input set B as the visible block into the model and generate the feature representation H of the visible block set B B , and infer the feature representation H as the set A as the mask block B→A ;

[0045] S4. Matching head to feature representation H through mask representation A→B Sort so that the feature representation H B And the feature representation H A→B The KL divergence between them is minimized, and the mask representation is matched to the feature representation H B→A Sort so that the feature representation H B→A And the feature representation H A The KL divergence between them is the smallest;

[0046] S5. Represent the sorted features as H A→BWith the feature representation H B , the sorted feature representation H B→A With the feature representation H A After being combined, they serve as a high-level semantic representation of the target image.

[0047] A computer device includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0048] Compared with the prior art, the advantages of the present invention are: by making full use of the characteristics of mask image representation, the present invention uses a learning mechanism with the original image as the pre-training target to perform image mask pre-training, uses two encoders to respectively learn the semantic information of the visible block and infer the semantic information of the mask block based on the content of the visible block, and then combines the semantic information of the visible block and the mask block to form a complete image representation for image reconstruction, which can effectively improve the ability to capture various image features. For image features that are difficult to learn or capture, they can also be inferred through visible block information, effectively compensating for and enhancing the image representation, so that the learned image representations can promote and compensate each other in downstream tasks, effectively enhancing the image's high-level semantic representation ability, image representation ability and the overall learning and reasoning ability of the model, without relying on the model to provide high-level semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the implementation process of the end-to-end image mask pre-training method in Example 1 of the present invention.

[0050] Figure 2 This is a schematic diagram of the network architecture principle for implementing end-to-end image mask pre-training in Example 1 of the present invention.

[0051] Figure 3 It is a schematic diagram of the structure of the encoder used in a specific application embodiment of the present invention.

[0052] Figure 4 It is a schematic diagram of the implementation process of the end-to-end image mask pre-training method in Example 2 of the present invention (the dimension of the image block representation is 1). DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0054] As shown in the present disclosure, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not specifically refer to the singular and may also include the plural. The words "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprising" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0055] The traditional mask image modeling method based on pixel reconstruction and the mask image modeling method based on token reconstruction both have certain limitations, resulting in the learned image representation belonging to low-level semantics or lacking semantics. At the same time, the mask image modeling method based on token reconstruction needs to be highly dependent on the trained VAE model to provide pre-training targets. The present invention combines the advantages of the traditional mask image modeling method based on pixel reconstruction and the mask image modeling method based on token reconstruction to establish a mask image modeling method that uses image pixels as pre-training targets while achieving semantic complementarity and enhancing image representation. First, a certain proportion of image blocks are randomly sampled from the complete image as visible blocks and the remaining image blocks as mask blocks. Then, the visible blocks are respectively extracted through two different encoders to extract visible block representations and the mask block representations are inferred through the visible block content. Finally, the combined visible block representation and the mask block representation are input into the decoder as the implicit embedding of the entire image to reconstruct the original image.

[0056] The present invention makes full use of the characteristics of mask image representation, uses a learning mechanism with the original image as the pre-training target to effectively enhance and promote image representation, and uses two encoders to learn the semantic information of visible blocks and infer the semantic information of mask blocks based on the content of visible blocks. It can effectively improve the ability to capture various image features. For image features that are difficult to learn or capture, they can also be inferred through visible block information, which can greatly compensate for and enhance image representation. The encoder used for inference has stronger reasoning ability, which further enhances the overall learning and reasoning ability of the model. Therefore, the image representations learned in the above manner can promote and compensate each other in downstream tasks, effectively enhancing the high-level semantic representation ability of the image without relying on the model to provide high-level semantic information.

[0057] The end-to-end image mask pre-training method of this embodiment is as follows Figure 1 、 2 As shown, where the dimension of the image block representation is not 1, the steps of the pre-training method include:

[0058] S1. Perform preprocessing operations on each data sample in the original image dataset to obtain the image dataset to be processed.

[0059] In this embodiment, each data sample in the original image dataset ImageNet-1K is first preprocessed by rotation, cropping, coloring, etc. to expand the sample size of the dataset ImageNet-1K, and then converted into a uniform size of 224×224 to form the image dataset ImageNet-1K-1 to be processed.

[0060] S2. Perform non-overlapping cuts on each data sample in the image dataset to be processed, and randomly sample a specified proportion of image blocks from each cut image block as visible blocks and the remaining image blocks as mask blocks.

[0061] In this embodiment, each sample X of the image dataset ImageNet-1K-1 to be processed is cut into patches of the same size 16×16 and the same number of patches evenly and without overlap, and a set is randomly extracted according to a certain ratio, namely X v , the remaining image blocks are used as mask blocks X u , then X=X v ∪X u The extraction ratio used in this embodiment is specifically 50%, which can be configured according to actual needs.

[0062] S3. The visible blocks are input into two different encoders respectively, one of which generates the visible block representation, and the other encoder is used to infer the mask block representation based on the visible block content.

[0063] In this embodiment, the visible block X v Convert it into a two-dimensional feature matrix, pass the linear mapping layer to expand the feature dimension of each image block, then initialize an additional cls token to add it to the 0 position of the input feature, concatenate it with the image block feature and add the corresponding position information as the input feature of the encoder.

[0064] S301. Each image block x in the visible block i ∈X v Expand the RGB three channels into a specified dimension (such as 1024 dimensions) through a linear layer, i.e. x′ i =x i E;

[0065] S302. Initialize an additional cls tokenx cla , and the visible block feature x′ i Splice into a feature matrix, that is, [xcla ,x′1,…,x′ i ,…,x′ I ], where x cla Global features used to aggregate features of all visible patches as input;

[0066] S303. The feature matrix [x cla ,x′1,…,x′ i ,…,x′ I ] plus the corresponding location information E pos Later, they are used together as the input feature e0 of the encoder, that is, e0 = [x cla ,x′1,…,x′ i ,…,x′ I ′]+E pos ;

[0067] S304. Input the obtained input features e0 to the first encoder f θ Generate tokens corresponding to visible blocks Right now and input to the second encoder f θ Infer the token corresponding to the mask block Right now That is, the corresponding visible block representation and mask block representation are obtained.

[0068] In this embodiment, two encoders (the first encoder f θ , the second encoder f θ ) are all based on the Transformer-Encoder module, such as Figure 3 As shown in the figure, the Transformer-Encoder module includes an Add&Norm layer, a Feed Forward layer, and a multi-head attention layer to extract image blocks. The high-level semantic features of the image blocks are extracted through the Add&Norm layer, the Feed Forward layer, and the multi-head attention layer. The Add&Norm layer includes an Add layer and a Layer Normalization layer, where the Add layer is used for residual connection and the Layer Normalization layer is used to normalize the hidden layer in the neural network to a standard normal distribution.

[0069] In this embodiment, two encoders (the first encoder f θ , the second encoder f θ ) The steps of generating visible block representation and mask block representation based on the forward propagation of the Transformer-Encoder module include:

[0070] Step 1) Transformer-Encoder module input e l As the input of the l+1th layer Transformer-Encoder, the high-level feature e′ is extracted through the multi-head attention layer and the Add&Norm layer l :

[0071]

[0072] Among them, MultiHeadAttention(·) is the multi-head attention layer, LayerNorm(·) is the LayerNormalization layer, e l +MultiHeadAttention(e l ) corresponds to the Add layer;

[0073] Step 2) Extract the high-level features e′ l Input to the Feed Forward layer and Add&Norm layer to extract high-level semantic features:

[0074] e l+1 =LayerNorm(e′ l +FeedForward(e′ l )) (2)

[0075] FeedForward(·) is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. It is expressed by the following formula:

[0076] max(0,e′ l W l1 +b l1 )W l2 +b l2 (3)

[0077] Among them, W l1 and W l2 is the feature matrix to be trained; b l1 and b l2 is the feature bias to be trained;

[0078] Step 3) Repeat steps 1), 2), and 3) for a total of Num times, where Num is a set number of times (specifically set to 12 times in this embodiment) to finally obtain a high-level semantic representation corresponding to each image block, that is, a visible block representation and a mask block representation.

[0079] The above multi-head attention layer function MultiHeadAttention(e l) specifically includes several self-attention functions Self-Attention(e l ), MultiHeadAttention(e l ) all self-attention functions Self-Attention(e l )’s output features are spliced ​​together, and the self-attention function Self-Attention (e l ):

[0080]

[0081]

[0082] Among them, W l,Q 、W l,K 、W l,V are the three feature matrices to be trained in the lth Encoder module, used to train the feature e l Perform linear mapping operation, each self-attention function Self-Attention(e l ) corresponds to the linear mapping matrix W l,Q 、W l,K 、W l,V Different, d is the matrix Q l and K l The number of columns.

[0083] S4. Match the mask block representation with the mask block position according to the mask block, combine the matched visible block representation and the mask block representation to form a complete original image representation, and input it into the decoder for original image reconstruction.

[0084] The first encoder f θ The generated image representation has a one-to-one correspondence with the visible blocks, but the second encoder f θ Since the inferred image representation has no position information to match it with the mask block, the learned semantic representation cannot be matched with the corresponding mask block. Therefore, it is necessary to match the mask block representation with the mask block position. This embodiment further introduces a mask representation matching head. Based on the mask representation matching head, the embedding space is H u The location information provided is used to u Perform position matching and sorting. The mask indicates that the specific execution steps of the matching head include:

[0085] S401. Input the mask block to the third encoder To obtain the mask block embedding space And the corresponding mask block position information is combined to construct a representation space, the third encoder is the first encoder f in the pre-training process θ Perform gradient sliding average to obtain;

[0086] S402. Use mask block to represent H u and the mask block embedding space The elements in the corresponding two sets A are constructed u and And by transforming the set A u The elements in the two sets A are sorted so that u and The KL divergence between elements is minimal;

[0087] S403. According to the sorted A u The element value corresponds to The elements in the constructed representation space look for the corresponding matching mask representation position information, and represent H based on the mask block obtained by the search u The position information of the mask block represents H u Sort and get the sorted mask block representation

[0088] In this embodiment, the pre-training objective function of the model is constructed according to the following formula:

[0089]

[0090] in, is the combined complete image embedding, H v is the visible block representation, g(·) represents the decoder, and the network structure is similar to the encoder. represents the image generated based on the learned image embedding, and MSE (Mean-Square Error, MSE) represents the mean square error function used to calculate the generated image and the pixel loss between the original image X.

[0091] It should be noted that when calculating the KL divergence of any two image block representations, due to the characteristics of image block representation, there will not be an image block representation with a higher similarity j with other image block representations. Therefore, the sorted A u It is unique and clear.

[0092] Example 2:

[0093] This embodiment performs image mask pre-training for the special case where the dimension of the corresponding image block representation is 1. When the dimension of the image block representation is 1, each image block is mapped to a numerical value, namely a token, and then combined into a token matrix as the potential semantic embedding of the entire image. Unlike high-dimensional embedding, this embodiment uses a pre-training method similar to the mask image modeling method based on Token Reconstruction to learn a 14×14 matrix to represent the semantic information of the entire image.

[0094] The following differences exist between this embodiment and embodiment 1 in the processing of multi-dimensional image blocks:

[0095] (1) The encoder used consists of a Transformer-Encoder module, followed by a linear layer and a Softmax layer.

[0096] The specific process of the above-mentioned linear layer and softmax layer can be expressed as follows:

[0097] Y=Softmax(W r H+b r )

[0098] Among them, H is the feature output of the last layer Transformer-Encoder, which represents the high-level semantic features of the image block. θ In the figure, the feature representation H corresponding to the visible block is represented v and in the encoder f θ In the figure, the feature representation H corresponding to the mask block is represented u ; Y is the token generated based on the image block features; the linear layer is represented by W in the formula r H+b r , where W r is the feature matrix to be trained; b r is the feature bias to be trained.

[0099] (2) The MSE function is used to calculate the minimum distance between set elements.

[0100] In this embodiment, after completing the pre-training of the model, the first encoder f in the model is retained. θ and the second encoder f θ The network parameters in the image are used to obtain the high-level semantic representation corresponding to each image block in the entire image. θ The generated image block representation corresponds to the representation of the image block; the second encoder f θ The second encoder f needs to be fed through a mask block θ and the third encoder fθ The image block representation generated in the image block representation obtains the position information of each mask block representation, and the matching process is similar to the pre-training stage, so as to further infer the corresponding mask block representation.

[0101] The specific steps of the pre-training method in this embodiment include:

[0102] S1. Perform preprocessing operations on each data sample in the original image dataset to obtain a dataset of images to be processed;

[0103] S2. Perform non-overlapping cuts on each data sample in the image data set to be processed, and randomly divides the cut image blocks into sets A and B.

[0104] For example, the target image is converted to a size of 224×224, cut into patches of the same size of 16×16 and the same number of patches evenly and non-overlappingly, and then randomly divided into two sets A and B at a ratio of 50%.

[0105] S3. Input set A as visible blocks into the model and generate feature representation H of visible block set A. A , and infer the feature representation H of the set B as the mask block A→B ; Input set B as the visible block into the model and generate the feature representation H of the visible block set B B , and infer the feature representation H as the set A as the mask block B→A ;

[0106] S4. Matching head to feature representation H through mask representation A→B Sort so that the feature representation H B And the feature representation H A→B The KL divergence between them is minimized, and the mask representation is matched to the feature representation H B→A Sort so that the feature representation H B→A And the feature representation H A The KL divergence between them is the smallest;

[0107] S5. Represent the sorted features as H A→B With the feature representation H B , the sorted feature representation H B→A With the feature representation H A After being combined, they serve as a high-level semantic representation of the target image.

[0108] Through the above steps, this embodiment performs further linear probe or fine probe fine-tuning on the high-level semantic information of the target image by initializing a new linear layer after the Transformer-Encoder module when performing downstream tasks to fit the downstream task objectives. This can solve the problem that traditional mask image modeling methods based on Token Reconstruction need to rely on the model to provide high-level semantic information.

[0109] The computer device of this embodiment includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0110] The present invention realizes end-to-end image mask pre-training by fusing image semantic representations, which can solve the problem of insufficient image representation generated by traditional mask image modeling methods based on pixel reconstruction, and effectively improve the high-level semantic representation capability of images.

[0111] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. An end-to-end image mask pre-training method, characterized in that: When the dimension represented by the image block is not 1, the steps of the pre-training method include: S1. Perform preprocessing operations on each data sample in the original image dataset to obtain a dataset of images to be processed; S2. performing non-overlapping cuts on each data sample in the image dataset to be processed, and randomly sampling a specified proportion of image blocks from each cut image block as visible blocks and the remaining image blocks as mask blocks; S3. Input the visible blocks into two different encoders, one of which generates a visible block representation, and the other encoder is used to infer the mask block representation based on the visible block content; S4. Matching and sorting the mask block representation and the mask block position according to the mask block, and after matching and sorting, combining the visible block representation and the mask block representation to form a complete original image representation, which is input into the decoder for original image reconstruction; In step S2, each sample of the image data to be processed is processed Perform uniform non-overlapping cutting to form image blocks of the same size and number, and randomly extract a visible block from the set according to the specified ratio , use the remaining image blocks as mask blocks ,in .

2. The end-to-end image mask pre-training method according to claim 1, characterized in that The steps of step S3 include: S301. Expand the RGB channels of each image block in the visible block into a specified dimension through a linear mapping layer to obtain visible block features; S302. Initialize an additional cls token And concatenate with the visible block features to form a feature matrix, where Global features used to aggregate features of all visible patches as input; S303. The feature matrix and the corresponding position information are used as input features and are input into two encoders respectively to obtain corresponding visible block representations and mask block representations.

3. The end-to-end image mask pre-training method according to claim 1, characterized in that Both encoders are based on the Transformer-Encoder module. The high-level semantic features of the image blocks are extracted through the Add & Norm layer, Feed Forward layer and multi-head attention layer in the Transformer-Encoder module. The Add & Norm layer includes the Add layer and the Layer Normalization layer, where the Add layer is used for residual connection, and the Layer Normalization layer is used to normalize the hidden layer in the neural network to a standard normal distribution.

4. The end-to-end image mask pre-training method according to claim 3, characterized in that In step S3, the two encoders generate visible block representation and mask block representation based on the Transformer-Encoder module, and the steps include: The first l Input to the Transformer-Encoder module As the first l +1 layer of Transformer-Encoder input, extracting high-level features through multi-head attention layer and Add & Norm layer : in, is a multi-head attention layer, It is the LayerNormalization layer. Corresponding to the Add layer; The high-level features extracted Input to the Feed Forward layer and Add & Norm layer to extract high-level semantic features: in, It is a two-layer fully connected layer. The activation function of the first layer is ReLU, and the second layer does not use an activation function. It is expressed by the following formula: in, and is the feature matrix to be trained; and is the feature bias to be trained; The high-level semantic features extracted from each image block ultimately obtain the high-level semantic representation corresponding to each visible block and mask block, thereby obtaining the visible block representation and the mask block representation.

5. The end-to-end image mask pre-training method according to claim 4, characterized in that The multi-head attention layer function There are several self-attention functions included , the multi-head attention layer All self-attention functions The output features of the self-attention function are spliced ​​together. include: in, 、 、 It is l The three feature matrices to be trained in the Encoder module are used to train the features Perform linear mapping operation, each self-attention function The corresponding linear mapping matrix 、 、 different, is a matrix and The number of columns.

6. The end-to-end image mask pre-training method according to any one of claims 1 to 5, characterized in that: In step S4, the mask block representation and the mask block position are matched and sorted according to the mask block by the mask representation matching head. The specific execution steps of the mask representation matching head include: Input the mask block into the third encoder to obtain the mask block embedding space , and the position information of the corresponding mask block is combined to construct a representation space, the third encoder is obtained by performing gradient sliding average on the first encoder during the pre-training process, the first encoder is the encoder for generating the visible block representation among the two encoders, and the mask block representation Generated by the second encoder of the two encoders; Using mask blocks and the mask block embedding space The elements in the corresponding two sets are constructed and , and by adding the collection The elements are sorted so that the two sets and The KL divergence between elements is minimal; According to the sorted The element value corresponds to The elements in the constructed representation space search for the position information of the corresponding matching mask representation, and represent the mask block obtained according to the search The position information of the mask block represents Sort and get the sorted mask block representation .

7. The end-to-end image mask pre-training method according to claim 6, characterized in that In step S4, the pre-training objective function of the model is constructed according to the following formula: in, is the combined complete image embedding, For visible blocks, Describes the decoder, represents the image generated based on the learned image embedding, and MSE represents the mean squared error function used to calculate the generated image and the pixel loss between the original image X.

8. The end-to-end image mask pre-training method according to claim 1, characterized in that When the dimension of the image block representation is 1, the steps of the pre-training method include: S1. Perform preprocessing operations on each data sample in the original image dataset to obtain a dataset of images to be processed; S2. performing non-overlapping cuts on each data sample in the image dataset to be processed, and randomly dividing the cut image blocks into sets A and B; S3. Input set A as visible blocks into the model to generate feature representation of visible block set A , and infer that set B is used as the feature representation of the mask block ; Input set B as the visible block into the model to generate the feature representation of visible block set B , and infer the feature representation of set A as the mask block ; S4. Matching head-to-feature representations via mask representations Sort so that the feature representation and feature representation The KL divergence between them is minimized, and the head pair feature representation is matched by mask representation Sort so that the feature representation and feature representation The KL divergence between them is the smallest; S5. Represent the sorted features respectively and feature representation , feature representation after sorting and feature representation After being combined, they serve as a high-level semantic representation of the target image.

9. A computer device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

Images