An unsupervised pre-training method for a neural network for image processing

Abstract

The present application relates to neural network unsupervised learning technical field, specifically to a kind of neural network unsupervised pre-training method for image processing, comprising the following steps: first, image is divided into image block, then mask operation is carried out, then perception loss is calculated, contrast loss and reconstruction loss are calculated, finally, training is carried out using loss.After training, the input image is processed using the trained model, and the category feature vector and the reconstructed image vector are obtained.The present application can measure the influence of mask operation on neural network by using perception loss, and make the feature more obvious by using contrast loss, and finally learn how to abstract the image into feature by reconstruction loss, while reducing the loss of information in the abstraction process, improve the feature extraction capability of neural network for image.

Description

Technical Field

[0001] This invention relates to the field of unsupervised learning technology for neural networks, and more specifically to an unsupervised pre-training method for neural networks used in image processing. Background Technology

[0002] With the development of neural networks, machine learning's thirst for data is growing daily. However, generating dataset labels is a time-consuming, laborious, and thankless task. Especially now that the scale of data is in the hundreds of millions, manually adding labels to it is almost a pipe dream. Therefore, to alleviate data hunger, unsupervised learning methods can be used.

[0003] Unsupervised learning algorithms commonly fall into three categories: clustering, dimensionality reduction, and self-supervised learning. Clustering, one of the earliest unsupervised algorithms, partitions elements by minimizing intra-class distances and maximizing inter-class distances. This is an NP-hard problem. Existing clustering methods achieve good convergence speeds and results when the number of elements is small and the dimensionality is low, but they incur significant costs when dealing with high-dimensional features. Dimensionality reduction methods map high-dimensional data to a low-dimensional space using some mapping technique, preserving the original distance relationships between data points. However, dimensionality reduction methods cannot capture the abstract connections between data points.

[0004] Self-supervised learning utilizes auxiliary tasks to extract supervisory information from the data itself, and uses this constructed supervisory information to train neural networks to extract features needed for downstream tasks. It can be divided into two main directions: generative and discriminative. One of the earliest architectures proposed in the generative direction is the autoencoder. First, data is input into an encoder, allowing the neural network to learn its features; this is called encoding. Then, the learned features are used to reconstruct the original input data using a decoder; this is called decoding. The goal of the autoencoder is to minimize the difference between the reconstructed data and the input data. The encoder is the desired feature extractor. Next, denoising autoencoders proposed obtaining more generalized feature extraction capabilities by "zeroing" certain input items. Mask autoencoders were inspired by denoising autoencoders and BERT in the field of natural language processing. It found that constructing noise with a large proportion (approximately 75%) of a 16*16 mask forces the network to learn some higher-order semantic information. Another major direction is the discriminative direction. CPC uses InfoNCE loss to build an autoregressive model to predict features in the latent space through a contrastive approach. This inspired a direction that obtains supervisory information by comparing differences between samples. SimCLR uses a Siamese network to generate two different augmented images of the same image. These two images are used as positive samples, while augmented images of other images in the batch are used as negative samples to obtain contrast information. Existing self-supervised learning methods still suffer from insufficient generalization ability, especially in large-scale image processing where they lack sufficient image feature extraction capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an unsupervised pre-training method for neural networks used in image processing. By using perceptual loss, contrast loss, and reconstruction loss, the method reduces the loss of image features during the abstraction process, thereby solving the technical problem of insufficient image feature extraction capability in existing unsupervised pre-training methods.

[0006] To achieve the above objectives, the present invention provides an unsupervised pre-training method for neural networks used in image processing, comprising the following steps:

[0007] Step 1: Input a dataset containing sample images of multiple classes;

[0008] Step 2: Perform a masking operation on the image input of the incoming dataset to obtain the original dataset and the masked dataset of the image respectively;

[0009] Step 3: Divide the neural network into multiple stages. Each stage uses multiple visual transformers as the backbone network. Then, input the original dataset and the masked dataset respectively, check the difference between the two outputs of each stage, and record it as the perceptual loss.

[0010] Step 4: In the last layer of the neural network, obtain the output of the visual transformer and divide it into classification units and image units; for the classification units, calculate the difference between the mask input of the image and its original input and the neural network output of other original image inputs, and record it as the contrast loss; for the image units, calculate the difference between the output of the neural network of the image mask input and the pixel values ​​of the original image, and record it as the reconstruction loss.

[0011] Step 5: Train the neural network using the perceptual loss, the contrast loss, and the reconstruction loss together as the total loss function;

[0012] Step 6: After training is complete, the model takes an image as input and outputs a category feature vector and a reconstructed image vector.

[0013] Optionally, the process of performing a masking operation on the image input of the incoming dataset includes the following steps:

[0014] Define the image as B refers to the number of input data in each batch, and H, W, and C refer to the width, height, and channel dimensions of the image, respectively. First, the image is divided into multiple image patches, with the patch size set to P×P, and the patch set defined. in The data is fed into a linear layer of a neural network to obtain its vector-based encoded embedding, and randomly initialized class units are added. Image patch feature set obtained by stitching D refers to the feature dimension, which can be expressed by the formula:

[0015] T = Concat(Patches, V) CLS )

[0016] Set a mask rate m r Construct a mask M ∈ [0,1] for each ∈ [0,1]. B×N Let it satisfy (Σ) M[i]＝1 1) / N≈m r Construct a randomly initialized mask vector. Therefore, a mask input is constructed based on the mask, and the operation can be described by the following formula:

[0017]

[0018] Where M[i] refers to the mask label corresponding to the i-th image block feature of mask M, and 1 indicates that this image block feature needs to be masked.

[0019] Optionally, the neural network can be divided into multiple stages, and the difference between the outputs of each stage can be checked and recorded as a perceptual loss process, including the following steps:

[0020] Step 3.1: Divide the neural network f into n stages, where stage i is denoted as Stage. i (X):

[0021] f j (X) = Stage j Stage j-1 ⊙…⊙Stage1(X)

[0022] Where ⊙ represents the function composition operation;

[0023] Step 3.2: Each Stage i (X) contains λ i A vision transformer, the process of which is described by the following formula:

[0024] X' (l) =X (l) +MSA(LN(X (l) ))

[0025] X (l+1) =X' (l) +FFN(LN(X' (l) ))

[0026] Where l represents the l-th layer of the neural network, LN refers to layer normalization, MSA refers to multi-head attention mechanism, and FFN refers to feedforward neural network.

[0027] The MSA process can be described by the following formula:

[0028]

[0029] Where Concat refers to the concatenation operation, N h This refers to the attention head, i.e., N. h The attention mechanism outputs, and the attention of the h-th head is defined as:

[0030] SelfAttention (h) (X):=[φ (h) (X)]V (h)

[0031] in It is a function that provides spatial attention based on the content of the input data, and its role is to aggregate V (h) Defined as:

[0032]

[0033] in It is a linear projection matrix, and τ is a temperature parameter;

[0034] The FFN process description is as follows:

[0035] FFN(X)=σ(XW1)W2

[0036] Where W1 and W2 are linear projection matrices, and σ is the activation function;

[0037] Step 3.3: Calculate the perceptual loss, only calculating the perceptual loss of the masked region.

[0038]

[0039] Where ξ j ξ is the hyperparameter coefficient that measures the weight of the perceived loss at each stage, and ξ j <ξ j+1 T[b,i] represents the feature of the i-th image patch in the b-th image.

[0040] Optionally, the output of the network inputting the original image can be re-divided into image units. With category unit Similarly, the output of the network that takes the masked image as input is divided into... and

[0041] f n (T)→[T IMG ;T CLS ],

[0042] The similarity function sim(·) is constructed to measure the similarity between classification units. The formula is as follows:

[0043] sim(a,b)=a T b

[0044] Use the cross-entropy function as the contrastive loss.

[0045]

[0046] Where T CLS [b], Refers to category unit T CLS , The b-th image data corresponds to the b-th image of the input data. τ is a temperature parameter used to control the inter-class distance.

[0047] The L1 loss is used to calculate the reconstruction loss of the masked region.

[0048]

[0049] Where Patches[b,i], This refers to the feature of the i-th image patch in the b-th image of the original image and the output image.

[0050] Optionally, the total loss function is:

[0051]

[0052] Where ξ, β, and γ are hyperparameter coefficients, with ξ representing the hyperparameter coefficient ξ that measures the weight of the perceptual loss at each stage. j A set of.

[0053] This invention provides an unsupervised pre-training method for neural networks used in image processing, comprising the following steps: first, dividing the image into image patches; then, performing a masking operation; next, calculating perceptual loss, contrastive loss, and reconstruction loss; and finally, training using these losses. After training, the trained model is used to process the input image to obtain category feature vectors and reconstructed image vectors. This invention uses perceptual loss to measure the impact of masking operations on the neural network, while using contrastive loss to make its features more apparent. Finally, the reconstruction loss teaches the network how to abstract images into features while reducing information loss during the abstraction process, thereby improving the neural network's feature extraction capability for images. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating an unsupervised pre-training method for neural networks used in image processing according to the present invention.

[0056] Figure 2 This is the original image of the pre-trained input image data in a specific embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram illustrating the masking operation effect of a specific embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram illustrating the training effect of the training network in a specific embodiment of the present invention.

[0059] Figure 5 This is a schematic diagram of the network output without masking operation in a specific embodiment of the present invention. Detailed Implementation

[0060] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0061] This invention provides an embodiment of an unsupervised pre-training method for neural networks used in image classification, comprising the following steps:

[0062] S1: Input a dataset containing sample images of multiple classes;

[0063] S2: Perform a masking operation on the image input of the incoming dataset to obtain the original dataset and the masked dataset of the image respectively;

[0064] S3: Divide the neural network into multiple stages. Each stage uses multiple visual transformers as the backbone network. Then, input the original dataset and the masked dataset respectively, check the difference between the two outputs of each stage, and record it as the perceptual loss.

[0065] S4: In the last layer of the neural network, the output of the visual transformer is obtained and divided into classification units and image units; for the classification unit, the difference between the mask input of the image and its original input and the neural network output of other original image inputs is calculated and recorded as the contrast loss; for the image unit, the difference between the output of the neural network of the image mask input and the pixel value of the original image is calculated and recorded as the reconstruction loss.

[0066] S5: The neural network is trained by using the perceptual loss, the contrast loss, and the reconstruction loss together as the total loss function;

[0067] S6: After training, the model takes an image as input and outputs a category feature vector and a reconstructed image vector.

[0068] Detailed step-by-step flowchart is shown below Figure 1 As shown.

[0069] Furthermore, the steps of the present invention will be further explained below with reference to specific embodiments:

[0070] In step S1, the image dataset used is ImageNet-1K, which includes more than 1.4 million images and 1,000 image categories.

[0071] The steps for performing the masking operation on the image in step S2 are as follows:

[0072] 2.1 Define the image as B refers to the number of input image data in each batch, and H, W, and C refer to the width, height, and channel dimensions of the image, respectively. First, the image is divided into multiple image patches, with the patch size set to P×P = 16×16. A set of segmented image patches is then defined. in The data is fed into a linear layer of a neural network to obtain its vector-based encoded embedding, and then randomly initialized class unit vectors are added. Image patch feature set obtained by stitching D refers to the feature dimension. It can be described by the formula...

[0073] T = Concat(Patches, V) CLS (1)

[0074] 2.2 Set a mask rate m r =0.75. Construct a mask M∈{0,1} B×N Make it satisfy m r ≈(∑ M[i]＝ 11) / N. Construct a randomly initialized mask vector. Therefore, a mask input is constructed based on the mask. This operation can be described by the following formula:

[0075]

[0076] Where M[i] refers to the mask label of the i-th image patch feature corresponding to mask M. 1 indicates that this image patch feature needs to be masked. In this operation, the mask vector covers the original feature vector.

[0077] The step S3, which involves checking the output differences at each stage and recording them as perceptual loss, is as follows:

[0078] 3.1 Divide the neural network f into n stages, where stage i is denoted as S. i (X)

[0079] f j (X)=S j (S j-1 (…S1(X) (3)

[0080] 3.2 Each stage S i (X) contains λ i A visual transformer. It is set as Λ = {λ} i}={2,2,6,2}, the visual transformer process is described by the formula:

[0081]

[0082] Where l represents the l-th layer of the neural network. LN refers to layer normalization. MSA refers to multi-head attention mechanism. FFN refers to feedforward neural network.

[0083] The MSA process can be described by the following formula:

[0084]

[0085] Where Concat refers to the concatenation operation, N h This refers to the attention head, i.e., N. h The attention mechanism outputs. The attention of the h-th head is defined as...

[0086] SelfAttention (h) (X):=[φ (h) (X)]V (h) (6)

[0087] in It is a function that provides spatial attention based on the content of the input data. Its role is to aggregate V (h) It is defined as:

[0088]

[0089] in It is a linear projection matrix. τ φ It is a temperature parameter, and its value is...

[0090] The FFN process description is as follows:

[0091] FFN(X)=σ(XW1)W2 (8)

[0092] Where W1 and W2 are linear projection matrices. σ is the activation function GeLU.

[0093] 3.3 Calculate the sensing loss. Only calculate the sensing loss of the masked region.

[0094]

[0095] Where ξ j ξ is a hyperparameter that measures the weight of perceived loss at each stage, and ξ j <ξ j+1 T[b,i] represents the feature of the i-th image patch in the b-th image.

[0096] The steps in step S4 for calculating the contrast loss and reconstruction loss are as follows:

[0097] 4.1 The output of the network that inputs the original image is re-divided into image units. With category unit Similarly, the output of the network that takes the masked image as input is divided into... and

[0098]

[0099] 4.2 Constructing a similarity function sim(·) to measure the similarity between classification units

[0100]

[0101] Use the cross-entropy function as the contrastive loss.

[0102]

[0103] Where T CLS [b], Refers to category unit T CLS , The b-th image data corresponds to the b-th image of the input data. τ is a temperature parameter used to control the inter-class distance.

[0104] 4.3 Use L1 loss to calculate the reconstruction loss of the masked region.

[0105]

[0106] Where Patches[b,i], This refers to the feature of the i-th image patch in the b-th image of the original image and the output image.

[0107] The total loss in step S5 is

[0108] The total loss function is calculated as follows:

[0109]

[0110] Where ξ, β, and γ are all hyperparameters. ξ refers to the coefficients appearing in formula (9). j set

[0111] Train the neural network using existing neural network training tools until the appropriate number of rounds are reached.

[0112] The downstream task fine-tuning process after training in step S6 is as follows:

[0113] 6.1 Input a batch of image data into the neural network f n Obtain the final visual transformer output of the model and divide it into category units and image units. Then, adjust the category units to a one-hot encoding using a linear layer and activation function.

[0114]

[0115] 6.2 Using the cross-entropy function as the classification loss to train the entire network

[0116]

[0117] Train the neural network using existing neural network training tools until the appropriate number of rounds are reached.

[0118] The final process of using the network for classification is as follows:

[0119] Calculate the category to which the image belongs. The process is described as follows:

[0120]

[0121] Class is a series of positive integers that indicate the most likely class number that the neural network considers for each image in the batch. It means The i-th vector represents the one-hot prediction probability that the image belongs to the i-th class.

[0122] Furthermore, the present invention provides a specific embodiment for further explanation, showing the execution effect of pre-training as follows: Figures 2 to 5 As shown. Specifically, Figure 2 The input image data is displayed. Figure 3 This demonstrates the network's masking operation on images. Figure 4 This demonstrates the training effect of the pre-trained network. Figure 5 This demonstrates the network output without masking. From... Figure 4 As can be seen above, this invention enables the network to extract higher-level semantics, allowing the network to possess some reasoning capabilities. From... Figure 5 As can be seen, this invention allows the neural network to retain a lot of structural and color information from the original image, which is helpful for training downstream tasks.

[0123] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. An unsupervised pre-training method of a neural network for image processing, characterized in that, The method comprises the following steps: Step 1: input a data set with multiple types of sample images; Step 2: perform a mask operation on the image input of the input data set to obtain an original data set and a masked data set of the image respectively; Step 3: divide the neural network into multiple stages, use multiple visual transformers as the backbone network in each stage, input the original data set and the masked data set respectively, check the difference between the outputs of the two in each stage, and record the difference as a perception loss; The process of dividing the neural network into multiple stages, checking the difference between the outputs of the two in each stage, and recording the difference as a perception loss comprises the following steps: Step 3.1: Divide the neural network f into n stages, stage i is denoted as Stage i (X): f j (X) = Stage j ⊙Stage j-1 ⊙…⊙Stage1(X) Where represents the function composition operation; Step 3.2: Each stage Stage i (X) comprises a number of visual transducers, the flow of which is described by the equation: i (X) comprises a number of visual transducers, the flow of which is described by the equation: X' (l) = X (l) + MSA(LN(X (l) )) X (l+1) = X' (l) + FFN(LN(X' (l) )) Where l represents the lth layer of the neural network, LN represents layer normalization, MSA represents a multi-head attention mechanism, and FFN represents a feedforward neural network, The process of MSA is described by the formula: where Concat denotes a concatenation operation, N h denotes the attention head, i.e., N h attention mechanism outputs, the attention of the h-th head is defined as: SelfAttenion (h) (X):=[φ (h) (X)]V (h) wherein is a function that provides spatial attention based on the input data content, and serves to aggregate V (h) , defined as: wherein is a linear projection matrix, τ φ is a temperature parameter; The process of FFN is described by the formula: FFN(X)=σ(XW1)W2 Where W1 and W2 are linear projection matrices, and σ is an activation function; Step 3.3: Compute perceptual loss, only compute perceptual loss for the masked region where ξ j is a hyper-parameter coefficient measuring the perceptual loss weight of each stage, and ξ j <ξ j+1 , T[b, i] represents the i-th image block feature of the b-th image. Step 4: obtain the output of the visual transformer at the last layer of the neural network, and divide the output into a classification unit and an image unit; for the classification unit, calculate the difference between the mask input of the image and the neural network output of the original input of the image and the original input of other images, and record the difference as a contrast loss; for the image unit, calculate the difference between the neural network output of the mask input of the image and the pixel value of the original image, and record the difference as a reconstruction loss; Step 5: use the perception loss, the contrast loss, and the reconstruction loss as a total loss function to train the neural network; The total loss function is: where ξ, β, γ are hyperparameter coefficients, and ξ denotes the hyperparameter coefficient in step 3.3 that measures the weight of the perception loss of each stage j a set of values of β and γ; Step 6: after the training is completed, input an image into the model to output a category feature vector and a reconstructed image vector.

2. The unsupervised pre-training method of the neural network for image processing according to claim 1, wherein the process of performing a mask operation on the image input of the input data set comprises the following steps: Where M[i] represents the mask label of the mask M corresponding to the ith image block feature, and 1 represents that the image block feature needs to be masked. The image is defined as B refers to the number of input data per batch, H, W, and C refer to the width, height, and channel dimension of the image, respectively; first, the image is divided into multiple image blocks, the size of the divided image block P x P is set, and the set of divided image blocks is defined as wherein It is input into a linear layer of the neural network to obtain a vector form encoding embedding, and a randomly initialized class unit is added The image block feature set is obtained by splicing D refers to the feature dimension, which is expressed by the formula: T = Concat(Patches, V CLS ) Setting a mask rate m r ∈ [0, 1], construct a mask M ∈ {0, 1} B×N , so that it satisfies (Σ M[i]＝1 1) / N ≈ m r ; construct a randomly initialized mask vector So as to construct a mask input according to the mask, and the operation is described by formula:

3. The unsupervised pre-training method of the neural network for image processing according to claim 2, wherein the specific execution process of step 4 comprises the following steps: A similarity function sim(·) is constructed to measure the similarity between the classification units, and the formula is: A cross-entropy function is used as the contrast loss, repartitioning the output of the network for the input original image into image cells with the category cells for the output of the network for the input masked image, the same applies and : An L1 loss is used to calculate the reconstruction loss of the masked region, ​ where T CLS [b], pointing category unit T CLS , the bth image data, corresponding to the bth image of the input data, τ is a temperature parameter for controlling the inter-class distance; ​ where Paches[b, i], denotes the i-th patch feature of the b-th image in the original and output.

Images