Convolutional network masking method, system and device and storage medium

Abstract

The invention provides a convolutional network masking method, system and device and a storage medium, and the method comprises the steps: introducing a grouping matrix with the same width and height as an input feature map for a convolution layer with a convolution kernel greater than a set threshold value in a convolution module, and setting a grouping index for the grouping matrix; and when each output position executes convolution, values are taken from the region with the same grouping index on the same position of the grouping matrix and calculation is carried out, and a convolution calculation result with a mask is obtained. According to the convolution calculation method with the mask, convolution is integrated into a mask image modeling self-supervision pre-training method, and meanwhile efficient parallel calculation is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a convolutional network masking method, system, device and storage medium. Background Art

[0002] In recent years, research in the field of vision based on deep learning has made great progress. Early convolutional neural networks collected local features in a hierarchical manner through convolution operations to obtain better image representations, and had good performance in computer vision tasks such as image classification, object detection, and instance segmentation. The current computer vision field has ushered in a new era of large-scale representation learning, and the Transformer architecture originally designed for the field of natural language processing has become the mainstream architecture of large-scale visual models.

[0003] Since it is expensive and time-consuming to obtain a large number of high-quality samples, and self-supervised learning can capture deeper structures and features within the data, self-supervised learning has become the mainstream paradigm for large-scale model pre-training. In the field of computer vision, Masked Image Modeling (MIM) is a common self-supervised pre-training method. The core idea of ​​masked image modeling is to mask image blocks during training, and then use the unmasked visible parts of the image to predict the masked original pixels or potential features, so as to learn more general and rich visual representations. Self-supervised pre-training frameworks based on masked image modeling include Masked Autoencoder (MAE), DATA2VEC, etc.

[0004] MAE trains the model through pixel-level image reconstruction tasks. The structure is as follows Figure 1 As shown. MAE first divides the image into a large number of image blocks, and randomly samples a part of the image blocks based on uniform distribution for masking. Each masked image block is replaced by a mask token, which is a shared and learnable vector. The encoder of MAE uses the ViT model and only processes visible image blocks; the decoder is a lightweight module that outputs the pixel values ​​of the predicted masked image blocks, where input is the input masked image, encoder is the encoder, decoder is the decoder, and target is the original image.

[0005] DATA2VEC trains the model by predicting the representation space. The structure is shown in the attached figure. Figure 2As shown. Among them, Original is the original input, Masked is the masked input, Images is the image, Speech is the speech, Language is the language, I like tea with milk is I like milk tea, Model in teacher-mode is the teacher model, Predictmodel representation of original input is the potential representation of the predicted original input, and Teacher tracksstudent parameters is the teacher model tracking student model parameters. DATA2VEC generates masks in the same way as MAE, that is, a certain proportion of image blocks are randomly selected for masking. The training process of DATA2VEC involves two models: the teacher model and the student model. The teacher model encodes the complete input data (unmasked image blocks) and generates a potential representation of the data. The student model predicts these potential representations based on the visible image blocks instead of directly predicting the pixel values ​​of the image blocks. This method enables the model to learn more general and robust feature representations.

[0006] The above self-supervised pre-training framework usually uses the Vision Transformer network architecture to encode images. Although the Vision Transformer can capture complex spatial transformations and long-distance feature dependencies due to its self-attention mechanism and multi-layer perceptron (MLP) structure to obtain global feature representation, the Vision Transformer ignores the details of local features. Therefore, in order to improve the performance of the model, some methods consider the advantages of convolutional neural networks in local feature extraction and introduce them into the model architecture of self-supervised pre-training to enhance the learning of feature representation. However, one problem faced by this is how to combine local convolution operations and global Transformer operations into the self-supervised pre-training framework for mask image modeling. The reason is that the masking strategy based on mask image modeling cannot be directly applied to the encoder of the "convolution + Transformer" structure - if the convolution operation is performed before the image block mask, because when the input is convolved, the spatial information of the visible image block and the masked image block may be fused, resulting in the encoder's final output token containing some visible information, thus affecting the reconstruction of the original image; if the convolution operation is performed after the image block mask, image blocks at different positions will be scattered in all the output tokens, and these tokens need to be retained in the later Transformer stage, resulting in excessively high computational costs for pre-training large models.

[0007] To address this problem, ConvMAE adopts a block masking strategy, such as Figure 3 As shown. Encoder is an encoder, which includes three stages: Stage1, Stage2, and Stage3. The size of the image input to the encoder is H*W*3, where H represents the height and W represents the width. The sizes of the feature maps output by the three stages are H / 4*W / 4*C1, H / 8*W / 8*C2, and H / 16*W / 16*C3, respectively, where C1, C2, and C3 represent different numbers of channels. The first two stages of the encoder include Patch Embedding and Masked Convolution Block modules. For the masked convolution module, the input FeatureEmbedding is a feature embedding, which is masked (Mask) and then passed through the depth convolution module (DepthWise Conv), the linear module (Linear), and then the residual connection, and finally the feedforward neural network (FFN) and the residual connection to obtain the final result. The third stage includes Patch Embedding and Transformer modules. Block-wise Masking is a convolutional masking strategy for block masking. This strategy first generates a random mask in the last stage, and upsamples the generated mask by 2 and 4 times to mask the input of the convolution in the second and first stages. The convolution modules in both stages use non-overlapping 2×2 convolutions to project the input to the next stage. In this way, the masked token and the visible token can be completely separated in the Transformer stage. The output E1 of the first stage of the Encoder is subjected to convolution and straightening operations (StrideConv+Flatten) and masking operations with a stride of 4, and the output E2 of the second stage is subjected to convolution and straightening operations (StrideConv+Flatten) and masking operations with a stride of 2. Multi-Scale Fusion is performed with the output E3 of the third stage as the input of the decoder. The decoder contains a linear layer for fusing multi-granular features, and then adds the masked token to obtain the result after the Transformer module. Although the mask strategy used by ConvMAE can integrate convolution into the "convolution + Transformer" architecture while inheriting the computational efficiency of MAE, the training of the model is still time-consuming. Summary of the invention

[0008] In order to integrate convolution into the structure of mask image modeling self-supervised pre-training and solve the problem of time-consuming model training, the present invention proposes a convolutional network masking method, including:

[0009] For the convolutional layers in the convolutional module whose convolution kernels are larger than the set threshold, a grouping matrix with the same width and height as the input feature map is introduced, and a grouping index is set for the grouping matrix;

[0010] When performing convolution at each output position, a value is taken from an area with the same grouping index at the same position of the grouping matrix and calculated to obtain a masked convolution calculation result.

[0011] Optionally, when performing convolution at each output position, a value is taken from an area with the same grouping index at the same position of the grouping matrix and calculated to obtain a masked convolution calculation result, including:

[0012] When executing the convolution, compare the grouping indexes of different positions participating in the convolution calculation with the grouping indexes of the grouping matrix to see if they are the same, and obtain the convolution mask matrix;

[0013] The convolution mask matrix is ​​multiplied by the input feature matrix after sliding cropping to obtain a masked input feature matrix;

[0014] Matrix multiplication is used to multiply the masked input feature matrix with the weight matrix for convolution calculation, and the masked convolution calculation result is obtained after adding the bias.

[0015] Optionally, the step of comparing the grouping indexes of different positions participating in the convolution calculation with the grouping indexes of the grouping matrix to see whether they are the same when performing the convolution to obtain the convolution mask matrix includes:

[0016] Perform sliding cropping function operation on the input convolution feature map and random grouping matrix;

[0017] The random grouping matrix after sliding cropping is compared with the original random grouping matrix to see if the grouping index is the same to generate the convolution mask matrix.

[0018] Optionally, for the convolution layer in the convolution module whose convolution kernel is larger than a set threshold, introducing a grouping matrix having the same width and height as the input feature map includes:

[0019] Obtaining an input image and a mask ratio, wherein the input image includes a plurality of image blocks;

[0020] Determine the number of groups based on the mask ratio;

[0021] The image blocks are randomly and evenly grouped based on the number of groups to obtain a grouping matrix with the same width and height as the input feature map.

[0022] On the other hand, the present invention also provides a convolutional network mask system, comprising:

[0023] A setting module, used for introducing a grouping matrix with the same width and height as the input feature map for a convolution layer whose convolution kernel is greater than a set threshold in the convolution module, and setting a grouping index for the grouping matrix;

[0024] The mask calculation module is used for taking and calculating values ​​from an area with the same grouping index at the same position of the grouping matrix at each output position when performing convolution, so as to obtain a convolution calculation result with a mask.

[0025] Optionally, the setting module is specifically used to:

[0026] Obtaining an input image and a mask ratio, wherein the input image includes a plurality of image blocks;

[0027] Determine the number of groups based on the mask ratio;

[0028] The image blocks are randomly and evenly grouped based on the number of groups to obtain a grouping matrix with the same width and height as the input feature map.

[0029] Optionally, the mask calculation module includes:

[0030] A first calculation submodule is used to compare whether the grouping indexes of different positions participating in the convolution calculation are the same as the grouping indexes of the grouping matrix when performing the convolution, so as to obtain a convolution mask matrix;

[0031] A second calculation submodule is used for multiplying the convolution mask matrix with the input feature matrix after sliding cropping to obtain a masked input feature matrix;

[0032] The third computing submodule is used to perform convolution calculation by multiplying the masked input feature matrix with the weight matrix using matrix multiplication, and obtain the masked convolution calculation result after adding the bias.

[0033] Optionally, the first calculation submodule is specifically used for:

[0034] Perform sliding cropping function operation on the input convolution feature map and random grouping matrix;

[0035] The random grouping matrix after sliding cropping is compared with the original random grouping matrix to see if the grouping index is the same to generate the convolution mask matrix.

[0036] In another aspect, the present application further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0037] The memory is used to store one or more programs;

[0038] When the one or more programs are executed by the at least one processor, a convolutional network mask method as described above is implemented.

[0039] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, a convolutional network mask method as described above is implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention provides a convolution network mask method, comprising: for a convolution layer in a convolution module whose convolution kernel is greater than a set threshold, introducing a grouping matrix with the same width and height as the input feature map, and setting a grouping index for the grouping matrix; when executing convolution, each output position takes and calculates a value from an area with the same grouping index at the same position of the grouping matrix to obtain a masked convolution calculation result. The present invention proposes a masked convolution calculation method, which realizes the integration of convolution into a mask image modeling self-supervised pre-training method, and realizes efficient parallel calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the MAE structure;

[0043] Figure 2 It is a schematic diagram of the DATA2VEC structure;

[0044] Figure 3 This is a diagram of the ConvMAE structure and mask strategy;

[0045] Figure 4 A flow chart of a convolutional network masking method of the present invention;

[0046] Figure 5 A schematic diagram of a self-attention calculation method for mask image modeling according to the present invention;

[0047] Figure 6 This is an example diagram of the convolutional network mask strategy of the present invention;

[0048] Figure 7 Schematic diagram of a convolution calculation method based on a convolution mask strategy of the present invention;

[0049] Figure 8 The structure diagram of the student model encoder of Example 1 of the present invention;

[0050] Fig. 9 This is a structural diagram of a student model decoder of Example 1 of the present invention;

[0051] Fig.10 This is the overall structure diagram of Example 1 of the present invention;

[0052] Fig.11 This is a structural diagram of the encoder model of Example 2 of the present invention;

[0053] Fig.12 This is a structural diagram of the model decoder of Example 2 of the present invention;

[0054] Fig.13 This is the overall structure diagram of Example 2 of the present invention;

[0055] Fig.14 The figure is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0056] In order to use the "convolution + Transformer" architecture more efficiently with random masks to achieve a fast mask modeling solution, this paper designs a convolutional network masking method suitable for mask image modeling pre-training. This method can be integrated into the mask image modeling self-supervised pre-training method to achieve fast mask modeling under the "convolution + Transformer" architecture.

[0057] In order to better understand the present invention, the content of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0058] Embodiment 1:

[0059] A convolutional network masking method, such as Figure 4 As shown, including:

[0060] Step S1: For a convolution layer in a convolution module whose convolution kernel is larger than a set threshold, a grouping matrix having the same width and height as the input feature map is introduced, and a grouping index is set for the grouping matrix;

[0061] Step S2: When performing convolution at each output position, a value is taken from an area with the same grouping index at the same position of the grouping matrix and calculated to obtain a masked convolution calculation result.

[0062] The following is a detailed description of each step.

[0063] Step S1: For the convolution layer in the convolution module whose convolution kernel is larger than the set threshold, a grouping matrix with the same width and height as the input feature map is introduced, and a grouping index is set for the grouping matrix, including:

[0064] Obtaining an input image and a mask ratio, wherein the input image includes a plurality of image blocks;

[0065] Determine the number of groups based on the mask ratio;

[0066] The image blocks are randomly and evenly grouped based on the number of groups to obtain a grouping matrix with the same width and height as the input feature map.

[0067] Step S2: When performing convolution at each output position, a value is taken from an area with the same group index at the same position of the grouping matrix and calculated to obtain a convolution calculation result with a mask, including:

[0068] When executing the convolution, compare the grouping indexes of different positions participating in the convolution calculation with the grouping indexes of the grouping matrix to see if they are the same, and obtain the convolution mask matrix;

[0069] The convolution mask matrix is ​​multiplied by the input feature matrix after sliding cropping to obtain a masked input feature matrix;

[0070] Matrix multiplication is used to multiply the masked input feature matrix with the weight matrix for convolution calculation, and the masked convolution calculation result is obtained after adding the bias.

[0071] Furthermore, when performing the convolution, the grouping indexes of different positions participating in the convolution calculation are compared with the grouping indexes of the grouping matrix to see if they are the same, and a convolution mask matrix is ​​obtained, including:

[0072] Perform sliding cropping function operation on the input convolution feature map and random grouping matrix;

[0073] The random grouping matrix after sliding cropping is compared with the original random grouping matrix to see if the grouping index is the same to generate the convolution mask matrix.

[0074] The principle of the sliding clipping function operation involved in step S2 is as follows:

[0075] The random grouping matrix after sliding cropping has a different dimension from the original random grouping matrix. Use the "==" in pytorch to compare two tensors of different dimensions. When comparing, first expand the low-dimensional tensor to the same shape as the high-dimensional tensor, and then compare each position to obtain the mask matrix. Here pytorch is an open source Python machine learning library.

[0076] A convolutional network mask method provided by the present invention is integrated into the mask image modeling self-supervised pre-training method to realize fast mask modeling under the "convolution + Transformer" architecture. This embodiment takes the computer vision model pre-training method introduced in the patent application document with application number 202410038875.4 and name as an example, and applies a convolutional network mask method provided by the present invention to the patent application document, specifically as follows: Figure 5 shown.

[0077] When performing self-supervised pre-training for mask image modeling, given a mask ratio γ, all image blocks of each sample are randomly and evenly divided into groups, each containing the same number of image blocks. When performing self-attention calculations later, we only need to set the attention of blocks that do not belong to the same group to -∞ through the attention mask to obtain the self-attention matrix with masked samples. This self-attention calculation is equivalent to processing n g masked samples, achieving efficient parallel computation of masked samples.

[0078] The present invention improves the mask calculation method of the convolutional network so that it can be integrated into the above-mentioned mask image modeling self-supervised pre-training method. Specifically, in the convolution module, for each convolution layer with a convolution kernel greater than 1, a grouping matrix with the same width and height as the input feature map is introduced. When performing convolution, each output position only takes values ​​and calculates from the area with the same grouping index at the same position as the grouping matrix. For the output of the convolution module, each position maintains the grouping index at the same position as the grouping matrix. This masked convolution calculation method can avoid feature confusion between the mask area and the visible area in the later input to the Transformer architecture. Performing a masked convolution calculation at the same time is also equivalent to processing n g A sample with a mask. For a detailed schematic diagram of the present invention, see Figure 6 and Figure 7 .

[0079] Example 2

[0080] The following two embodiments are used to refine the content of the invention.

[0081] Example 1: DATA2VEC self-supervised pre-training method based on "convolution + Swin Transformer", such as Fig.10 As shown:

[0082] This example uses the DATA2VEC framework for pre-training. The student model is an encoder with a 3-stage "convolution + SwinTransformer" structure and a decoder with a 4-layer convolution structure. The teacher model is a 3-stage "convolution + SwinTransformer" encoder, which has the same encoder structure as the student model. The teacher model is updated using an exponential moving average. The batch size during model training is B. In practice, values ​​such as 16, 32, or 64 are often used depending on the graphics card. The input image size is 512×512×3, and the image block size is 1×1. This embodiment uses python and pytorch to implement model training, reasoning, etc., where batch size represents batch size, python is a computer language, and pytorch is an open source Python machine learning library.

[0083] 1. Student Model

[0084] 1.1 Image block division and random grouping

[0085] 1.1.1, random grouping of image blocks. Let the mask ratio γ = 75% to determine the number of random groupings of image blocks.

[0086] 1.1.1.1, calculate the size of the output of the last stage in the encoder structure, that is, the third stage, the length and width are Therefore, the number of image blocks in each group of each picture in this stage is The input image blocks of the third stage are randomly and evenly grouped according to the number of groups to ensure that the number of image blocks in each group of each picture is the same. The grouping index of each image block is recorded and the random grouping matrix G3 is output. G3 is upsampled twice to obtain the random grouping matrices G2 and G1 of the image blocks in the second and first stages of the encoder. The sizes of the random grouping matrices of image blocks in the three stages are G1 = B × 1 × 256 × 256, G2 = B × 1 × 128 × 128, and G3 = B × 1 × 64 × 64, respectively.

[0087] 1.1.1.2, based on the third random grouping matrix G3 obtained in 1.1.1.1, calculate the mask sample matrix GM3 of the four groups of samples in a batch in the third stage, and set the positions of the same grouping in the mask sample matrix to 1, and the positions of different groupings to 0. The size is GM3 = (4B) × 1 × 64 × 64.

[0088] 1.1.1.3, based on the three random grouping matrices obtained in 1.1.1.1, construct the mask matrices based on the window and the moving window of the Swin Transformer layer for window attention calculation. That is, the positions with the same grouping in the mask matrix are set to 0, and the positions with different groupings are set to -9999.0. The last two stages of the encoder are Swin Transformer layers, and the mask matrices based on the window and the moving window are recorded as (GMW2, GMSW2) and (GMW3, GMSW3), respectively, with sizes of:

[0089] GMW2=GMSW2=B×64×1×256×256

[0090] GMW3=GMSW3=B×16×1×256×256.

[0091] 1.2 Encoder

[0092] The encoder is a three-stage "convolution + Swin Transformer" structure. The output dimensions of each stage of the encoder are (128, 256, 512). The first stage is a convolutional network, and the next two stages are Swin Transformer. The number of Swin Transformer modules in each stage is (2, 10), the number of self-attention heads is (8, 16), and the window size is (16, 16). The detailed structure of the encoder can be found in Figure 8 .

[0093] 1.2.1 Phase 1

[0094] 1.2.1.1, Input training sample image I S , the sample input of each batch is B×3×512×512. The samples pass through the first stage of the convolutional network. First, PixelUnshuffle is used to downsample the input. The output after downsampling is The size is B×(3×4=12)×256×256. Then the im2col method is used to perform masked convolution calculations, with a convolution kernel size of 3×3, padding=1, and step size=1. Specifically, Perform nn.functional.unfold operation on the random grouping matrix G1, and the outputs are The size is B×(12×(3×3)=108)×(256×256=65536), then Compare with the original G1 to see if it is the same to generate the convolution mask matrix M1, M1 and Multiply to get Then use matrix multiplication to Multiplying with the weight matrix implements convolution calculation, and adding the bias to get the output result The size is B×128×256×256, and finally through LayerNorm, the output result of the first stage is obtained The size is B×128×256×256.

[0095] 1.2.2 Stage 2

[0096] 1.2.2.1, Output of 1.2.1.1 First, the image block fusion layer is passed, and the output The size is B×256×128×128, and then passes through the first stage of the Swin Transformer module. According to the window size of 16 in the first Swin layer, the input samples are divided into We first calculate the window-based Query2, Key2, and Value2 matrices, with sizes of (64B)×(16×16=256)×256. Thus we get Then add the above matrix to the window-based mask matrix GMW2 corresponding to the first Swin Transformer layer in 1.1.1.3, and then obtain the mask attention matrix A2 through Softmax, with a size of B×64×8×256×256. Finally, multiply the mask attention matrix with the Value2 matrix to obtain the window-based mask attention result with a size of (64B)×256×256. Resize the mask attention result to B×256×128×128 and Perform shortcut connection, then pass through LayerNorm layer and MLP layer in turn to get the result The size is B×256×128×128.

[0097] Output of 1.2.2.2, 1.2.2.1 After the second stage of the Swin Transformer module. According to the moving window size Resize the input to be used for moving window calculations and divide the resized input into We first calculate the Query'2, Key'2, and Value'2 matrices based on the moving window, with sizes of B×64×8×256×32, and then add the position embedding and perform the mask window self-attention calculation based on the moving window. Specifically, we first calculate the Query'2, Key'2, and Value'2 matrices based on the moving window, with sizes of B×64×8×256×32, respectively, to obtain Then add the above matrix to the moving window-based mask matrix GMSW2 corresponding to the first Swin Transformer in 1.1.1.3, and then obtain the mask attention matrix A'2 through Softmax, which is B×64×8×256×256. Finally, multiply the mask attention matrix with the Value'2 matrix to obtain the moving window-based mask attention result, which is (64B)×256×256. Resize the mask attention result to B×256×128×128 and add it to the input of 1.2.2.2 (that is, the output of 1.2.2.1). ) for shortcut connection, and then pass through the LayerNorm layer and the MLP layer in turn to get the result The size is B×256×128×128. 1.2.2.2 The overall process is similar to 1.2.2.1, the difference is that the window-based masked self-attention calculation is replaced by the moving window-based masked self-attention calculation.

[0098] 1.2.3 Phase 3

[0099] Output of 1.2.3.1, 1.2.2.2 First, the image block fusion layer is passed, and the output The size is B×512×64×64, and then it goes through the first stage of the Swin Transformer module. According to the window size of the second Swin layer 16, the input sample is divided into We first calculate the window-based Query3, Key3, and Value3 matrices, with sizes of (16B)×(16×16=256)×512. Thus we get Then add the above matrix to the window-based mask matrix GMW corresponding to the second Swin Transformer layer in 1.1.1.3, and then obtain the mask attention matrix A3 through Softmax, with a size of B×16×16×256×256. Finally, multiply the mask attention matrix with the Value3 matrix to obtain the window-based mask attention result with a size of (16B)×256×512. Resize the mask attention result to B×512×64×64 and Perform shortcut connection, then pass through LayerNorm layer and MLP layer in turn to get the result The size is B×512×64×64.

[0100] 1.2.3.2, Output of 1.2.3.1 After the second stage of the Swin Transformer module. According to the moving window size Resize the input to be used for moving window calculations and divide the resized input into We first calculate the Query'3, Key'3, and Value'3 matrices based on the moving window, which are sized B×16×16×256×32, and then add the position embedding and perform the mask window self-attention calculation based on the moving window. Specifically, we first calculate the Query'3, Key'3, and Value'3 matrices based on the moving window, which are sized B×16×16×256×32, respectively, to get Then add the above matrix to the moving window-based mask matrix GMSW3 corresponding to the second Swin Transformer in 1.1.1.3, and then obtain the mask attention matrix A'3 through Softmax, which is B×16×16×256×256. Finally, multiply the mask attention matrix with the Value'3 matrix to obtain the moving window-based mask attention result, which is (16B)×256×512. Resize the mask attention result to B×512×64×64 and add it to the input of 1.2.3.2 (that is, the output of 1.2.3.1). ) for shortcut connection, and then pass through the LayerNorm layer and the MLP layer in turn to get the result The size is B×512×64×641.2.3.2 The overall process is similar to 1.2.3.1, the difference is that the window-based masked self-attention calculation is replaced by the moving window-based masked self-attention calculation.

[0101] 1.2.3.3 Repeat the steps in 1.2.3.1 and 1.2.3.2 in sequence times, respectively Final output of the third stage The size is B×512×64×64.

[0102] 1.3 Decoder

[0103] The decoder is a 4-layer convolutional structure with an embedding dimension of 256, a convolution kernel size of 5, a step size of 1, and a padding size of 2. For the detailed structure of the decoder, see the attached Fig. 9 ConvLayer is the convolution layer, BatchNorm is batch normalization, Activation is the activation function layer, and Linear Head is the linear head. Make 4 copies, multiply them with the mask sample matrix GM3 in 1.1.1.2, and then add them to the mask token to get the decoder input D, which is (4B)×512×64×64 in size.

[0104] 1.3.1, D passes through the first convolution module of the decoder, and the result is shortcut-connected with D, outputting D1 with a size of (4B)×256×64×64. The first convolution module includes 2 layers of convolutional networks, BatchNorm (batch normalization) and activation layers.

[0105] The operations in 1.3.2 and 1.3.1 are repeated three times, and D4 is output, which has a size of (4B)×256×64×64. The three-times-repeated process is implemented by the decoder stage 2, stage 3, and stage 4 respectively. The structure of each stage is the same as that of stage 1, and will not be repeated here.

[0106] 1.3.3, D4 is projected twice linearly to obtain the final output O S , the size is (4B)×512×64×64.

[0107] 2Teacher model.

[0108] 2.1 Encoder

[0109] The encoder is a 3-stage "convolution + Swin Transformer" structure, which is the same as the encoder structure in the student model. The encoder input is I T , the size of each batch is B×3×512×512. Since the teacher model uses the original image without mask, ordinary convolution calculation is performed in the convolution stage, and general window-based self-attention and moving window-based self-attention calculation are performed in the Swin Transformer stage, which will not be repeated here. The output of the three stages is in in, is the output of the first stage of the teacher model, are the outputs of the first and second modules of the second stage of the teacher model, respectively. They are the outputs of the 1st, 2nd, 9th and 10th modules in the 3rd stage of the teacher model respectively.

[0110] 3. Model loss calculation and model training.

[0111] like Fig.10 As shown, the target predicted by the student network is the output of the teacher network The last 8 outputs These 8 outputs are first InstanceNormed (instance normalized) to obtain Calculate the mean is the result of the above output after instance normalization, i is the subscript of the element in the summation formula, and finally the output value is obtained through LayerNorm, which is recorded as O T , the size is (4B)×512×64×64.

[0112] Calculate O S The prediction results of the four groups of masked samples are similar to those of O T The actual target value corresponding to the mean square error MSE, then the loss of training a forward propagation is After calculating the loss value, the model parameters are updated to complete a training of 4 batches.

[0113] 4. Teacher model weight update

[0114] When the parameters are updated for the tth time, the parameters of the teacher model are The parameters of the student model are The teacher model is updated using exponential moving average. Specifically, the teacher model is first initialized with the parameters of the student model, i.e. After each 4-batch training, the student model updates its parameters by returning the loss gradient, while the teacher model updates its parameters by Update. α is the attenuation coefficient. As the number of iterations increases, the model parameters become more stable. In this example, when t≤500000, When t>500000, α=0.999995.

[0115] Example 2: MAE self-supervised pre-training method based on "convolution + Vision Transformer", such as Fig.13 As shown:

[0116] This example uses the MAE framework for pre-training. The model is a 2-stage "convolution + ViT" structure encoder and a 6-layer Transformer structure decoder. The batch size during model training is B. In practice, it is often 16, 32, 64, etc. depending on the graphics card. The input image size is 256×256×3, the image block size is 8×8, and the encoder embedding dimension is 128. This embodiment uses python and pytorch to implement model training, reasoning, etc.

[0117] 1. Image block division and random grouping

[0118] 1.1, Image block division. Input training sample image I, each batch of sample input is B×3×256×256. Divide the input image into non-intersecting image blocks, and linearly project them according to the encoder embedding dimension, outputting I E , the size is B×128×32×32.

[0119] 1.2, random grouping of image blocks. Let the mask ratio γ = 75% to determine the number of random groupings of image blocks.

[0120] 1.2.1, calculate the size of the output of the last stage in the encoder structure, that is, the second stage, with a length and width of Therefore, the number of image blocks in each group of each picture in this stage is The input image blocks of the second stage are randomly grouped evenly according to the number of groups to ensure that the number of image blocks in each group of each picture is the same. The grouping index of each image block is recorded and the random grouping matrix G2 is output. Since the Vision Transformer does not downsample the feature map, that is, the size of the output of the first stage is the same as the size of the output of the second stage, the random grouping matrix G1 and G2 of the image block in the first convolution stage of the encoder are the same. The size of the random grouping matrix of the image blocks in the two stages is G1 = G2 = B × 1 × 16 × 16.

[0121] 1.2.2, based on the second random grouping matrix G2 obtained in 1.2.1, calculate the mask sample matrix GM2 of the four groups of samples in a batch in the second stage. The positions of the same grouping in the mask sample matrix are set to 1, and the positions of different groupings are set to 0. The size is GM2 = (4B) × 1 × 16 × 16.

[0122] 1.2.3, based on the two random grouping matrices obtained in 1.2.1, construct the feature mask matrix of Vision Transformer for attention calculation. That is, the positions with the same grouping in the mask matrix are set to 0, and the positions with different groupings are set to -9999.0, denoted as GMF2, with a size of B×1×256×256.

[0123] 2 Encoder

[0124] The encoder is a two-stage "convolution + Swin Transformer" structure. The output dimensions of each stage of the encoder are (256, 256). The first stage is a convolutional network, and the second stage is a Vision Transformer. The number of modules contained in Vision Transformer is 10, and the number of self-attention heads is 8. The detailed structure of the encoder can be found in Fig.11 .

[0125] 2.1 Phase 1

[0126] 2.1.1, the sample passes through the first stage of the convolutional network. First, use PixelUnshuffle to downsample the input, and the output after downsampling is C ds , the size is B×(128×4=512)×16×16. Then the im2col method is used to perform masked convolution calculations, with a convolution kernel size of 3×3, padding=1, and step size=1. Specifically, for C ds Perform nn.functional.unfold operation on the random grouping matrix G1, and the outputs are The size is B×(512×(3×3)=4608)×(16×16=256), then Compare with the original G1 to see if it is the same to generate the convolution mask matrix M1, M1 and Multiply to get Then use matrix multiplication to Multiplying with the weight matrix realizes convolution calculation, and adding bias to get the output result C 1,1 , the size is B×256×16×16, and finally through LayerNorm, the output result C1 of the first stage is obtained, the size is B×256×16×16.

[0127] 2.2 Phase 2

[0128] 2.2.1, the output C1 of 2.1.1 is first added to the position embedding and the mask self-attention calculation is performed. Specifically, first calculate The matrices are of sizes Thus we get Then add the above matrix to the mask matrix GMF2 corresponding to the Vision Transformer layer in 1.2.3, and then use Softmax to get the mask attention matrix The size is B×8×256×256. Finally, the mask attention matrix is ​​combined with The matrix is ​​multiplied to obtain the masked attention result, which is B×256×256. The masked attention result is resized to B×256×16×16, and is shortcut-connected to the input C1 of 2.2.1, and then passes through the LayerNorm layer and the MLP layer in turn to obtain the result V 1,1 , the size is B×256×16×16.

[0129] 2.2.2 Repeat the operation in 2.2.1 10-1=9 times in sequence to obtain V 1,2 ,V 1,3 ……V 1,9 ,V 1,10 , the second stage finally outputs V 1,10 , the size is B×256×16×16.

[0130] 3. Decoder

[0131] The decoder is a 6-layer Transformer structure with an embedding dimension of 256 and a number of self-attention heads of 16. The detailed structure of the decoder can be found in Fig.12 .

[0132] Set the output V of 2.2.2 1,10 Make 4 copies, multiply them with the mask sample matrix GM2 in 1.2.2, and then add them to the mask token to get the decoder input D, which is 4B×256×16×16.

[0133] 3.1, D passes through the first layer Transformer module of the decoder, adds position embedding and performs self-attention calculation. Specifically, first calculate The matrices are of sizes Thus we get Then get the attention matrix through Softmax The size is (4B)×16×256×256. Finally, the attention matrix is ​​combined with The matrix is ​​multiplied to obtain the attention result, which is (4B) × 256 × 256. The masked attention result is resized to (4B) × 256 × 16 × 16, and is shortcut-connected to the input D of 3.1. Then, it passes through the LayerNorm layer and the MLP layer in sequence to obtain the result D1, which is (4B) × 256 × 16 × 16.

[0134] The operations in 3.2 and 3.1 are repeated 6-1=5 times, and the output is D6, which has a size of (4B)×256×16×16.

[0135] 3.3, D4 is subjected to a linear projection to obtain the final output O D , the size is (4B)×(3×2 2 ×8 2 =768)×16×16, and then adjusted to (4B)×3×256×256.

[0136] 4. Model loss and training

[0137] The target predicted by the model is the original image pixel corresponding to the mask position. D The mean square error (MSE) between the predicted results of the mask patch in the four groups of samples and the corresponding actual pixel values ​​in the original image is , then the loss of training one forward propagation is After calculating the loss value, the model parameters are updated to complete a training of 4 batches. Fig.10 .

[0138] Example 3

[0139] The present invention based on the same inventive concept also provides a convolutional network mask system, comprising:

[0140] A setting module, used for introducing a grouping matrix with the same width and height as the input feature map for a convolution layer whose convolution kernel is greater than a set threshold in the convolution module, and setting a grouping index for the grouping matrix;

[0141] The mask calculation module is used for taking and calculating values ​​from an area with the same grouping index at the same position of the grouping matrix at each output position when performing convolution, so as to obtain a convolution calculation result with a mask.

[0142] Optionally, the setting module is specifically used to:

[0143] Obtaining an input image and a mask ratio, wherein the input image includes a plurality of image blocks;

[0144] Determine the number of groups based on the mask ratio;

[0145] The image blocks are randomly and evenly grouped based on the number of groups to obtain a grouping matrix with the same width and height as the input feature map.

[0146] Optionally, the mask calculation module includes:

[0147] A first calculation submodule is used to compare whether the grouping indexes of different positions participating in the convolution calculation are the same as the grouping indexes of the grouping matrix when performing the convolution, so as to obtain a convolution mask matrix;

[0148] A second calculation submodule is used for multiplying the convolution mask matrix with the input feature matrix after sliding cropping to obtain a masked input feature matrix;

[0149] The third computing submodule is used to perform convolution calculation by multiplying the masked input feature matrix with the weight matrix using matrix multiplication, and obtain the masked convolution calculation result after adding the bias.

[0150] Optionally, the first calculation submodule is specifically used for:

[0151] Perform sliding cropping function operation on the input convolution feature map and random grouping matrix;

[0152] The random grouping matrix after sliding cropping is compared with the original random grouping matrix to see if the grouping index is the same to generate the convolution mask matrix.

[0153] Example 4

[0154] like Fig.14 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0155] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of a convolutional network mask method in the above embodiment.

[0156] Example 5

[0157] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a convolutional network mask method in the above embodiment.

[0158] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A convolutional network masking method, characterized in that: include: For the convolutional layers in the convolutional module whose convolution kernels are larger than the set threshold, a grouping matrix with the same width and height as the input feature map is introduced, and a grouping index is set for the grouping matrix; When performing convolution at each output position, a value is taken from an area with the same grouping index at the same position of the grouping matrix and calculated to obtain a masked convolution calculation result.

2. The method according to claim 1, characterized in that When performing convolution at each output position, a value is taken from an area with the same group index at the same position of the grouping matrix and calculated to obtain a masked convolution calculation result, including: When executing the convolution, compare the grouping indexes of different positions participating in the convolution calculation with the grouping indexes of the grouping matrix to see if they are the same, and obtain the convolution mask matrix; The convolution mask matrix is ​​multiplied by the input feature matrix after sliding cropping to obtain a masked input feature matrix; Matrix multiplication is used to multiply the masked input feature matrix with the weight matrix for convolution calculation, and the masked convolution calculation result is obtained after adding the bias.

3. The method according to claim 2, characterized in that The method of comparing the grouping indexes of different positions participating in the convolution calculation with the grouping indexes of the grouping matrix to see whether they are the same when performing the convolution, and obtaining the convolution mask matrix, includes: Perform sliding cropping function operation on the input convolution feature map and random grouping matrix; The random grouping matrix after sliding cropping is compared with the original random grouping matrix to see if the grouping index is the same to generate the convolution mask matrix.

4. The method according to claim 1, characterized in that For the convolution layer in the convolution module whose convolution kernel is larger than the set threshold, a grouping matrix with the same width and height as the input feature map is introduced, including: Obtaining an input image and a mask ratio, wherein the input image includes a plurality of image blocks; Determine the number of groups based on the mask ratio; The image blocks are randomly and evenly grouped based on the number of groups to obtain a grouping matrix with the same width and height as the input feature map.

5. A convolutional network mask system, characterized in that: include: A setting module, used for introducing a grouping matrix with the same width and height as the input feature map for a convolution layer whose convolution kernel is greater than a set threshold in the convolution module, and setting a grouping index for the grouping matrix; The mask calculation module is used for taking and calculating values ​​from an area with the same grouping index at the same position of the grouping matrix at each output position when performing convolution, so as to obtain a convolution calculation result with a mask.

6. The system according to claim 5, characterized in that The setting module is specifically used for: Obtaining an input image and a mask ratio, wherein the input image includes a plurality of image blocks; Determine the number of groups based on the mask ratio; The image blocks are randomly and evenly grouped based on the number of groups to obtain a grouping matrix with the same width and height as the input feature map.

7. The system according to claim 5, characterized in that The mask calculation module comprises: A first calculation submodule is used to compare whether the grouping indexes of different positions participating in the convolution calculation are the same as the grouping indexes of the grouping matrix when performing the convolution, so as to obtain a convolution mask matrix; A second calculation submodule is used for multiplying the convolution mask matrix with the input feature matrix after sliding cropping to obtain a masked input feature matrix; The third computing submodule is used to perform convolution calculation by multiplying the masked input feature matrix with the weight matrix using matrix multiplication, and obtain the masked convolution calculation result after adding the bias.

8. The system according to claim 7, characterized in that The first calculation submodule is specifically used for: Perform sliding cropping function operation on the input convolution feature map and random grouping matrix; The random grouping matrix after sliding cropping is compared with the original random grouping matrix to see if the grouping index is the same to generate the convolution mask matrix.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, a convolutional network mask method as described in any one of claims 1 to 4 is implemented.

10. A readable storage medium, characterized in that: An execution program is stored thereon, and when the execution program is executed, a convolutional network mask method as described in any one of claims 1 to 4 is implemented.

Images