An SVT image pre-training method based on variance entropy equivalent complexity ranking

Abstract

The present application relates to the technical field of image pre-training, and particularly relates to an SVT image pre-training method based on variance entropy equivalent complexity sorting. The steps are as follows: S1: obtaining a training image set, and performing normalization processing on label images in the training image set; S2: performing equivalent complexity sorting on the label images based on variance entropy coordinate pair image classification; S3: dividing a large number of ordered label images into subblocks of a fixed size, linearly embedding each subblock, adding a position embedding mark, and feeding a generated vector sequence to an encoder in a SpanVision Transformer for feature extraction to obtain subblock sample features corresponding to a sample subblock set. The present application provides an SVT image pre-training method based on variance entropy equivalent complexity sorting. The SpanVision Transformer method after image complexity sorting can reduce the time of image pre-training and improve the performance of image tasks.

Description

Technical Field

[0001] This invention relates to the field of image pre-training technology, and in particular to an SVT image pre-training method based on variance entropy equivalent complexity ranking. Background Technology

[0002] With the development of artificial intelligence, a large number of pre-trained models have emerged in the field of image processing, such as VisionTransformer. Image pre-training requires a large amount of image resources and computing resources. How to reduce the training time of pre-trained models while maintaining their performance is a problem that must be addressed. To solve this problem, we propose the following solutions.

[0003] The Span Vision Transformer (SVT) is a self-attention-based visual Transformer model used for image classification tasks. SVT divides the input image into a series of image spans. Each span is flattened into a vector, which is then fed into the Transformer's encoder for feature extraction. The difference between the Span Vision Transformer and the Vision Transformer lies in its masking strategy: the Span Vision Transformer uses a fixed-length sequence of consecutive sub-blocks for random masking.

[0004] However, in the Vision Transformer, the number of masks for sub-blocks is configured as a percentage, and the masking attributes of each sub-block are configured discretely and independently, generally preventing the formation of continuous sub-block mask sequences. Experimental results show that the Span Vision Transformer can improve model performance by using a fixed-length continuous sub-block sequence random masking method.

[0005] The main purpose of the equivalent complexity ranking method for image classification based on variance entropy coordinates is to rank the images that are about to enter the neural network training by complexity. The purpose of ranking is that an ordered set of images can reduce the training time of the model and improve the accuracy of the model.

[0006] To address this issue, a pre-training method for SVT images based on variance entropy equivalent complexity ranking is designed to provide an alternative technical solution. Summary of the Invention

[0007] Therefore, it is necessary to provide an SVT image pre-training method based on variance entropy equivalent complexity ranking to address the aforementioned technical problems.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A pre-training method for SVT images based on variance entropy equivalent complexity ranking, comprising the following steps:

[0010] S1: Obtain the training image set and perform normalization processing on the label images in the training image set;

[0011] S2: Sort the labeled images by equivalent complexity based on variance entropy coordinates for image classification;

[0012] S3: Divide a large number of ordered label images into fixed-size sub-blocks, linearly embed each sub-block, add position information, and feed the generated vector sequence into the encoder in Span Vision Transformer for feature extraction to obtain the sub-block sample features corresponding to the sample sub-block set.

[0013] As a preferred embodiment of the SVT image pre-training method based on variance entropy equivalent complexity ranking provided by the present invention, the original label image set in the training image set P is defined as I. label ={B1,...,B n ,...,B N}, that is, the original set of labeled images I label The label corresponding to the nth image in the dataset is B. n For the original set of labeled images I label Image straightening processing:

[0014] Using the averaging method, the original set of labeled images I label Each image in the image is compressed and converted into a single-channel h·w image, i.e., B n Convert to B' n And set the new set consisting of all single-channel images as the label image set. Tag Image Collection Single-channel image B' n The matrix is ​​represented as follows:

[0015]

[0016] Where h and w are positive integers.

[0017] As a preferred embodiment of the SVT image pre-training method based on variance entropy equivalent complexity ranking provided by the present invention, the steps of ranking the labeled images based on the equivalent complexity of image classification using variance entropy coordinates are as follows:

[0018] S21: Set of labeled images For each class, n images are extracted, denoted as image sorting set I. sort Now sort the image set I sort Perform image variance calculation; sum and average the values ​​of each pixel in each image. In obtaining Then, for each pixel a in each image i,j Calculate the standard deviation to obtain the corresponding image set I. sort Each image S n ;

[0019]

[0020] Among them, S n Represents the variance of the image;

[0021] S22: For image set I sort Calculate the two-dimensional entropy for each image, based on the formula for calculating the two-dimensional entropy of a grayscale image.

[0022] Where i represents the gray value of a pixel (0≤i≤L-1), j represents the average gray value of the neighborhood (0≤j≤L-1), p(i,j) represents the probability that gray value i and neighborhood gray value j occur simultaneously; log2 is the logarithm to the base 2.

[0023] S23: In the image set I sort The two-dimensional entropy H of each image n S and variance n Normalize; for image set I sort The variance S of each image n Perform Z-score standardization to obtain the standardized variance for each image.

[0024]

[0025] Subsequently, the image set I was examined. sort The two-dimensional entropy E of each image n Perform min-max normalization to obtain the standardized two-dimensional entropy of each image.

[0026]

[0027] in, and These are image sets I sort The minimum and maximum values ​​of the two-dimensional entropy;

[0028] S24: Based on image set I sort The standardized variance and standardized two-dimensional entropy of each image are used to construct a pseudo-score G for image ranking.sort G sort Image set I sort Each image in the dataset is represented by its two-dimensional entropy and variance as coordinates. Representing the coordinates of each image, image set I sort Coordinate pairs are constructed, and the K-Means clustering algorithm is used to divide the coordinate pairs into K classes. The classes farthest from the origin are ranked in ascending order of their scores. Represents category scores, where K is a positive integer;

[0029] S25: Once the pseudo-scores corresponding to the images are obtained, a backpropagation neural network based on variance entropy is used to calculate the hidden layers of the labeled images during the forward propagation of the neural network.

[0030] Where H is the output of the hidden layer, f is the activation function, b is the bias vector of the hidden layer, and w1 and w2 are respectively... and Weight parameters;

[0031] It is the output layer of the neural network, where It is the predicted score, W o H represents the weights of the output layer, H represents the output of the hidden layer, and b represents the weights of the output layer. o It is the bias of the output layer;

[0032] Using mean squared error as the loss function Where n is the number of images. G is the prediction score for the i-th image. i These are the actual scores; during backpropagation, gradient descent is used to update the network weights.

[0033] As a preferred embodiment of the SVT image pre-training method based on variance entropy equivalent complexity ranking provided by the present invention, the labeled image is divided into multiple image blocks, then position encoding is added to the image block sequence, and the image block sequence is masked with an image block span of length d, so that the Span Vision Transformer has better context understanding ability, improves the model's generalization ability, and strengthens the learning of important features.

[0034] The specific method of step S3 is as follows:

[0035] S31: Transfer image Remodeled into a series of image patches (h,w) where h is the resolution of the original grayscale image, 1 is the number of channels, (P,P) is the resolution of each image patch, and N = h·w / P 2This is the number of patches obtained. If the division is not exact, and a patch block is encountered at the boundary that is incomplete, then fill it with 0.

[0036] Image I label ={X1,..,X n ,.,X N}, where X n Represents the nth patch; randomly selects a span of length d for patch masking; sets the masking span ratio to p; the masked I... label use express,

[0037] S32: Perform a linear transformation on each image patch, E i =W e ·flatten(X n )+b e W e and b e These are the embedding layer and the bias, respectively, flatten(X) n This represents flattening the image patch into a one-dimensional vector; a positional encoding is added to each embedding vector: E′ i =E i +P i , where P i It is the position code of the i-th image block;

[0038] S33: Embedded vector sequence {E′1,E′2,...,E′} N Input Transformer encoder: H = TransformerEncoder(E′), where H represents the encoder output. The masking strategy in S31 is applied to H to obtain Ho. mask ;

[0039] S34: For a masked image patch, predict its original embedding. Where H mask It is the output of the masked encoder. It is a predicted embedding; the loss function is calculated based on the predictions of the masking blocks. Where ||·|| represents the Euclidean norm.

[0040] As a preferred embodiment of the SVT image pre-training method based on variance entropy equivalent complexity ranking provided by the present invention, in step S31, additional learnable classification labels are added to the sequence for image classification, and the sequence is flattened and mapped to the D dimension through a trainable linear projection patch.

[0041] As a preferred embodiment of the SVT image pre-training method based on variance entropy equivalent complexity sorting provided by the present invention, in step S32, a span mask of length d is performed, and the output of the Transformer encoder representing each label in the sequence is used.

[0042] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0043] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0044] This invention provides an SVT image pre-training method based on variance entropy equivalent complexity ranking. It measures the complexity of an image by variance entropy and uses an equivalent complexity ranking strategy for classifying images using deep neural networks. This allows the training data of the SpanVision Transformer to be quickly ranked based on complexity, prioritizing images with rich information and complex structures. This helps the SpanVision Transformer learn useful feature representations faster, thereby reducing the training time of large deep neural network structures and improving the image classification ability of large deep neural networks. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the process of the present invention;

[0047] Figure 2 This is a schematic diagram of image dimensionality reduction (conversion to grayscale image) according to the present invention;

[0048] Figure 3 This is a schematic diagram of the image patching method of the present invention;

[0049] Figure 4 This is a schematic diagram of the equivalent complexity ranking method for image classification using variance entropy coordinates according to the present invention.

[0050] Figure 5 This is a schematic diagram of the SpanVision Transformer of the present invention;

[0051] Figure 6 This is a schematic diagram illustrating an implementation example of the animal image classification method of the present invention.

[0052] Figure 7 This is a schematic diagram of the network parameters of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0055] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0057] Reference Figures 1-7 A pre-training method for SVT images based on variance entropy equivalent complexity ranking, the steps of which are as follows:

[0058] S1: Obtain the training image set and perform dimensionality reduction on the label images in the training image set;

[0059] The specific steps of step S1 are as follows:

[0060] The original set of labeled images in the training image set is defined as I. label ={B1,...,B n ,...,B N}, that is, the original set of labeled images I label The label corresponding to the nth image in the dataset is B. N For the original set of labeled images I label Dimensionality reduction of the image:

[0061] Using the averaging method, the original set of labeled images I label Each image in the image is compressed and converted into a single-channel h·w image, i.e., B... N Convert to B' N And set the new set consisting of all single-channel images as the label image set. Tag Image Collection This sets a single-channel image B'. N The matrix is ​​represented as follows:

[0062]

[0063] Where h and w are positive integers.

[0064] S2: Sort the labeled images by equivalent complexity based on variance entropy coordinates for image classification;

[0065] Equivalent complexity ranking for image classification based on variance entropy coordinates refers to a method of ranking the image complexity of the labeled images of the neural network model to be entered. Image complexity refers to the amount of information contained in an image.

[0066] The steps for S2 are as follows:

[0067] S21: Set of labeled images For each class, n images are extracted, denoted as image sorting set I. sort Now sort the image set I sort Perform image variance calculation. Sum and average the values ​​for each pixel in each image. In obtaining Then, for each pixel a in each image i,j Calculate the standard deviation to obtain the corresponding image set I. sort Each image S n .

[0068]

[0069] Among them, S n This represents the variance of the image.

[0070] S22: For image set I sort Calculate the two-dimensional entropy for each image, based on the formula for calculating the two-dimensional entropy of a grayscale image. Where i represents the gray value of a pixel (0≤i≤L-1), j represents the average gray value of the neighborhood (0≤j≤L-1), p(i,j) represents the probability that gray value i and neighborhood gray value j occur simultaneously; log2 is the logarithm to the base 2, which conforms to the conventional definition of entropy in information theory.

[0071] S23: In the image set I sort The two-dimensional entropy E of each image n S and variance n Perform normalization. For image set I sort The variance S of each image n Perform Z-score standardization to obtain the standardized variance for each image.

[0072]

[0073] Subsequently, the image set I was examined. sort The two-dimensional entropy E of each image nPerform min-max normalization to obtain the standardized two-dimensional entropy of each image.

[0074]

[0075] in, and These are image sets I sort The minimum and maximum values ​​of the two-dimensional entropy.

[0076] S24: Based on image set I sort The standardized variance and standardized two-dimensional entropy of each image are used to construct a pseudo-score G for image ranking. sort G sort Image set I sort Each image in the image is represented by its standardized two-dimensional entropy value and standardized variance as coordinates; here, we use... The coordinates of each image are represented, therefore image set I sort Coordinate pairs are constructed, and then the K-Means clustering algorithm is used to divide the coordinate pairs into K classes. The class farthest from the origin is ranked in ascending order of its score. Represents category scores, where K is a positive integer.

[0077] S25: Once the pseudo-scores corresponding to the image are obtained, the hidden layers of the labeled image in the forward propagation of the neural network are calculated based on the variance entropy coordinates. Where H is the output of the hidden layer, f is the activation function, b is the bias vector of the hidden layer, and w1 and w2 are respectively... and The weight parameters. It is the output layer of the neural network, where It is the predicted score, W o H represents the weights of the output layer, H represents the output of the hidden layer, and b represents the weights of the output layer. o This refers to the bias of the output layer. Mean squared error is used as the loss function. Where n is the number of images. G is the prediction score for the i-th image. i This is the actual score. During backpropagation, gradient descent is used to update the network weights.

[0078] S26: Once a well-trained image complexity ranking algorithm based on variance entropy is available, the image dataset I... label The variance and two-dimensional entropy of each image are input into the neural network to obtain the image complexity C for each image. n C n Sort the images from smallest to largest to obtain a new sequence of labeled images.

[0079] S3: Divide a large number of ordered labeled images into fixed-size sub-blocks, linearly embed each sub-block, add positional embedding, and feed the generated vector sequence into the encoder in Span Vision Transformer for feature extraction to obtain the sub-block sample features corresponding to the sample sub-block set. Add additional learnable classification labels to the sequence for image classification.

[0080] The steps for S3 are as follows:

[0081] S31: Image patch segmentation and masking. (This involves segmenting and masking the image...) Remodeled into a series of image patches (h,w) where h is the resolution of the original grayscale image, 1 is the number of channels, (P,P) is the resolution of each image patch, and N = h·w / P 2 This is the number of patches obtained. If the division is not exact, and there are incomplete patch blocks encountered at the boundary, it is filled with 0. Therefore, image I label ={X1,..,X n ,.,X N}, where X n This represents the nth patch. A span of length d is randomly selected for patch masking. The proportion of the masking span is set to p. The resulting I after masking... label use express,

[0082] S32: Image patch embedding and positional encoding. A linear transformation is performed on each image patch, E i =W e ·flatten(X n )+b e W e and b e These are the embedding layer and the bias, respectively, flatten(X) n This represents flattening an image patch into a one-dimensional vector. A positional encoding is added to each embedding vector: E′ i =E i +P i , where P i It is the positional encoding of the i-th image block.

[0083] S33: Transformer encoder processing. Embedded vector sequence {E′1, E′2, ..., E′} N Input Transformer encoder: H = TransformerEncoder(E′), where H represents the encoder output. The masking strategy in S31 is applied to H to obtain Ho. mask .

[0084] S34: Predict the masked patch and calculate the loss. For the masked image patch, predict its original embedding. Where H mask It is the output of the masked encoder. It is a predicted embedding. The loss function is calculated based on the predictions of the masking blocks. Where ||·|| represents the Euclidean norm.

[0085] Span masking of length d using Span Vision Transformer class ,X1,..,X n-d ,X mask ,...,X mask ,X n ,...X N And for each patch image x i The loss is summed by span masking to complete the pre-training of the image.

[0086] The following steps are performed to categorize animal images: Figure 6 Categorize animal images.

[0087] 1. First, compress and grayscale all animal images. For example, convert a 3-channel 1920×1080 image to a 540×540 grayscale image. That is, convert all images, regardless of their size or color channels, to 540×540 grayscale images (e.g., Figure 2 (The colored images on the left are converted to grayscale images on the right). Use 70% of these animal images as the training set.

[0088] 2. From all the animal images, randomly select 10% of the images from each category as training backpropagation neural networks with variance entropy coordinates as the equivalent complexity of image classification. This will enable the training of the neural network for sorting the animal images.

[0089] 3. Input the animal images from the training set into the SpanVision Transformer network. That is, input one image at a time, divide that image into 9 patches, and input them into the SpanVision Transformer in sequence. This process is used to train the network parameters of the SpanVision Transformer. Figure 7 .

[0090] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A pre-training method for SVT images based on variance entropy equivalent complexity ranking, comprising the following steps: S1: Obtain the training image set and perform normalization processing on the label images in the training image set; S2: Sort the labeled images by equivalent complexity based on variance entropy coordinates for image classification; S3: Divide a large number of ordered label images into fixed-size sub-blocks, linearly embed each sub-block, add position information, and feed the generated vector sequence into the encoder in Span Vision Transformer for feature extraction to obtain the sub-block sample features corresponding to the sample sub-block set; The difference between Span Vision Transformer and Vision Transformer is that Span Vision Transformer uses a fixed-length continuous sub-block sequence for random positioning masking. The training image set The original label image set definition in That is, the original set of labeled images The label corresponding to the nth image in the dataset is: For the original set of labeled images Image straightening processing: Using the averaging method, the original set of labeled images Each image in the process is compressed and converted to a single channel. The image, soon Convert to And set the new set consisting of all single-channel images as the label image set. , tag image collection Single-channel image The matrix is ​​represented as follows: in It is a positive integer.

2. The SVT image pre-training method based on variance entropy equivalent complexity ranking according to claim 1, characterized in that, The steps for ranking the label images based on the equivalent complexity of image classification using variance entropy coordinates are as follows: S21: Set of labeled images Extraction of each category The set of images is denoted as the image sorting set. Now sorting the image set Perform image variance calculation; sum and average the values ​​of each pixel in each image. After obtaining Then, each pixel of each image Calculate the standard deviation to obtain the corresponding image set. Each image ; in, Represents the variance of the image; S22: For image sets Calculate the two-dimensional entropy for each image, based on the formula for calculating the two-dimensional entropy of a grayscale image. ; in, Represents the grayscale value of a pixel ( ), Represents the average gray level of the neighborhood ( ), Represents grayscale value and domain grayscale value The probability of them occurring simultaneously; It is a logarithm to the base 2; S23: In the image set Two-dimensional entropy of each image and variance Normalize; for the image set Variance of each image Perform Z-score standardization to obtain the standardized variance for each image. , Subsequently, the image set Two-dimensional entropy of each image Perform min-max normalization to obtain the standardized two-dimensional entropy of each image. , ， in, and These are image sets The minimum and maximum values ​​of the two-dimensional entropy; S24: Based on image set The standardized variance and standardized two-dimensional entropy of each image are used to construct a pseudo-score for image ranking. ; It is an image set Each image in the dataset is represented by its two-dimensional entropy and variance as coordinates. The coordinates of each image represent the coordinates of the image set. The coordinate pairs are formed, and the K-Means clustering algorithm is used to divide the coordinate pairs into... Classes, the classes furthest from the origin, are sorted by their scores in ascending order. Represents category scores, where , It is a positive integer; S25: Once the pseudo-scores corresponding to the images are obtained, a backpropagation neural network based on variance entropy is used to calculate the hidden layers of the labeled images during the forward propagation of the neural network. ; in, It is the output of the hidden layer. It is an activation function. It is the bias vector of the hidden layer. and They are and Weight parameters; It is the output layer of the neural network, where It is a predicted score. These are the weights of the output layer. It is the output of the hidden layer. It is the bias of the output layer; Using mean squared error as the loss function ,in It is the number of images. It is the first The predicted score of the image. These are the actual scores; during backpropagation, gradient descent is used to update the network weights.

3. The SVT image pre-training method based on variance entropy equivalent complexity ranking according to claim 2, characterized in that, By dividing a labeled image into multiple image blocks, then adding positional encoding to the image block sequence, and finally dividing the image block sequence into blocks of length [missing information], the process is as follows: Image patch span masking enables the Span Vision Transformer to have better contextual understanding, improve model generalization ability, and enhance the learning of important features; The specific method of step S3 is as follows: S31: Transfer image Remodeled into a series of image patches , Where is the resolution of the original grayscale image, and 1 is the number of channels. It is the resolution of each image patch. This is the number of patches obtained. If the division is not exact, and a patch block is encountered at the boundary that is incomplete, then fill it with 0. image ,in Representing the Zhang Bu Block; Randomly select a length of... The span is used for patching and masking; Set the mask span ratio to ; behind the cover use express, ; S32: Perform a linear transformation on each image patch. ,in and These are the embedding layer and the bias layer, respectively. This means flattening the image patch into a one-dimensional vector; adding position encoding to each embedding vector: ,in It is the first Position encoding of each image block; S33: Embedded Vector Sequence Input Transformer encoder: ,in This indicates the encoder output, for Executing the masking strategy in S31 yields ; S34: For a masked image patch, predict its original embedding. ,in It is the output of the masked encoder. It is a predicted embedding; the loss function is calculated based on the predictions of the masking blocks. ,in This represents the Euclidean norm.

4. The SVT image pre-training method based on variance entropy equivalent complexity ranking according to claim 3, characterized in that, In step S31, additional learnable classification labels are added to the sequence for image classification, and these labels are flattened and mapped using a trainable linear projection patch. Dimension.

5. The SVT image pre-training method based on variance entropy equivalent complexity ranking according to claim 3, characterized in that, In step S32, a span mask of length d is performed, and the output of the Transformer encoder representing each marker in the sequence is used.

Images