A single image snow removal method based on GNN-transformer architecture

Abstract

The application discloses a kind of based on GNN-transformer architecture's single image snow removal method, belong to computer image processing technical field, first the image to be snow removal is input to GNN-transformer framework, and residual network trained using contrast learning is realized based on image graph construction;Second, the trained graph convolution network model is used to extract features to the snow image, and the attention mechanism is introduced to locate the snowflake position, aggregate feature information;Finally, the snowflake feature extracted generates snowflake attention map, and it is input into generative adversarial network to carry out snow removal operation, and finally output snow removal image.The example of the application is a kind of based on GNN-transformer architecture's single image snow removal method, can efficiently identify the position of snow through the snowflake structure and texture features in image, and remove it;While learning image detail features, accurately complete image snow removal task, restore the image covered by snow, so as to improve image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image processing, and particularly relates to a single image snow removal method based on a GNN-transformer architecture. BACKGROUND

[0002] With the progress and development of science and technology, visual information is an important source for human beings to understand the world. Images, as the most important visual information transmission mode, especially outdoor acquired video images, play an important role in maintaining public safety. Due to the influence of rain and snow weather, snowflakes cover the target objects, resulting in loss of key information of images.

[0003] In recent years, the application of deep learning to image processing tasks has attracted more and more attention. The research on image clarification processing under bad weather is an important direction in the field of image processing, and the purpose is to remove rain and snow in the image, so as to restore the covered image. At present, the research on image snow removal task mainly focuses on the method based on deep learning, which contributes to the research on image clarification processing under bad weather and provides a variety of efficient solutions.

[0004] Most of the traditional image snow removal methods are evolved from rain removal algorithms. Most of the traditional modeling snowflake principles are to remove the snowflakes in the image by using the characteristics of the image and monitoring and removing the snowflake components in the image according to the physical characteristics of the snowflakes. The high-dimensional characteristics of the snowflakes in the snow-removed image are relatively ignored. Although the image snow removal method based on convolutional neural network (CNN) has a strong feature extraction ability and a high-dimensional data representation ability, the framework improves the efficiency of the snow removal task. However, this method only stays at the pixel level of the picture in essence, and the image data is usually represented by a regular grid of pixels in Euclidean space, and the physical meaning is not clear. The relationship between the structure and attributes of the background and elements in the picture is ignored, and the feature learning ability for irregular and complex objects is insufficient.

[0005] Therefore, the application proposes a single image snow removal method based on a GNN-transformer architecture. Considering that in actual severe snowy weather, the snowflake structure in the image that needs to perform the snow removal task is complex and small in volume, the method represents the snow image with a graph structure, uses the GNN-transformer framework to extract features of the snow image, and references the attention mechanism to locate the snow position, finally maps the extracted snowflake features back to the image space to form a snowflake attention map, and inputs it into the generative adversarial network (GNN) for snow removal operation, and finally outputs the snow-removed image. The snow removal operation in the GNN model is realized, and the snow removal task in image processing is realized. SUMMARY

[0006] In order to solve the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a single image snow removal method based on a GNN-transformer architecture, which interprets the pixels in the image that needs to perform snow removal as nodes with mutual correlation in a graph structure, which is more flexible in processing images, builds the relationship between the local structure and the global of the sample, realizes the real modeling of the snow image data, and in the research of image clarification in actual severe weather, since the shape of the processing object is usually complex and small in volume, the GCN model is used in the method to extract features of the snow image, the attention mechanism is introduced to locate the position of the snowflake node in the graph, which can more effectively learn the complex snowflake structure features in the graph, reduce the parameter amount, improve the training speed, and greatly improve the training efficiency of the model; the generative adversarial network is used to perform snow removal operation on the snow image, which improves the snow removal ability of the image and greatly improves the precision and efficiency of the image snow removal task.

[0007] The technical scheme adopted by the application to solve the technical problems is:

[0008] A single image snow removal method based on a GNN-transformer architecture is provided, comprising the following steps:

[0009] S1: first input the image that needs to perform the snow removal task into the GNN-transformer architecture, and divide it into patches of the same size, each patch represents a node in the graph structure, and use the residual network (ResNet) trained by contrast learning to obtain the embedding vector of each patch, thereby realizing graph construction based on the image;

[0010] S2: Feature extraction is performed on the snowflake image using a trained Graph Convolutional Network (GCN) model, and an attention mechanism is introduced to select the snowflake nodes in the graph and aggregate information through the mechanism to obtain a feature map of the snowflake image.

[0011] S3: The extracted snowflake feature map is converted back to the image space to form a snowflake attention map, which is input into a Generative Adversarial Networks (GAN) for snow removal, and finally an image without snow is output.

[0012] Further, in step S1, the image pixels that need to perform the snow removal task are WxHx3.

[0013] Further, in step S1, a residual network is trained using contrastive learning, which maximizes the consistency between two different augmented views of the same image patch through a contrastive loss in the latent space to generate an embedding representation for the graph construction of the image to be removed, thereby calculating the feature matrix of the nodes in the graph construction structure, and realizing the construction of the image-based graph G=(V, E).

[0014] Further, for the feature matrix of the nodes, the feature vectors contained therein are used to define the information contained in the patch, and each patch is a node in the image. The dimension of the generated node feature vector is reduced from Wp x Hp x Cp to D x 1, where Wp, Hp and Cp are the width, height and channel of the image patch, and D x 1 is the dimension of the extracted feature vector.

[0015] Further, in step S2, the test image is input into the trained graph convolutional network model, and the trained graph convolutional network model is used to extract features from the snow-removed image through convolution, propagate and aggregate node information, and introduce an attention mechanism (transformer) to locate the position of the snowflake node in the graph, and finally obtain a snowflake feature map.

[0016] Further, the feature extraction process for the snow-removed image includes: in each hidden layer of the graph convolutional network model, the graph with features is convolved with a trainable matrix, and then the result is sent to the next layer. After passing through two hidden layers, the loss function of the training weight matrix w 0 and w 1 is calculated through the backpropagation algorithm. If the loss does not decrease within a few epochs, the training process will stop, and the final model learns the node features of the image to be removed.

[0017] Further, the cross-entropy function is used as the loss function in the GCN structure.

[0018] Further, in the introduction of attention mechanism to locate the position of snowflakes, the aggregation feature information, the introduction of transformer layer finally learns the mapping process from the graph space to the transformer space, which includes: taking the feature nodes as the tokens in the sequence, calculating the q vector, the k vector and the v vector, using the graph convolution layer in the form of the adjacent matrix of the position embedding to locate the position of the snowflakes, and generating the snowflake attention graph.

[0019] Further, the snowflake attention graph is used as a reference graph to guide the pixel repair of the snowflake position by the snow-free image generation module in the generative adversarial network, and finally the snow removal task is completed.

[0020] The snowflake attention graph is formed, and the process includes: mapping the extracted snowflake features, and transforming the extracted feature vectors from the graph space back to the image space by using the reconstruction network R to form the snowflake attention graph.

[0021] Further, the snowflake attention graph is input into the generative adversarial network for snow removal operation, and finally the snow-removed image is input, and the process includes: inputting the snowflake attention graph as a guide graph and an artificially synthesized graph into the generator module in the generative adversarial network, repairing the pixels covered by the snowflakes through the encoder-decoder, generating a snow-free image, and completing the single-image snow removal operation.

[0022] Specifically, during the training phase of the data, the GCN network training process is carried out, and the parameters of the network are constantly corrected until the GCN network learns to correctly extract the snowflake node features in the snow-containing image, and the parameters obtained by training are saved.

[0023] During the network model training phase, the iteration batch size is set to 32, and the total iteration number is 600 times. The initial learning rate is set to 0.001, the small batch gradient descent method is adopted, and the Adam optimizer is used to calculate the adaptive learning rate of each parameter.

[0024] Compared with the prior art, the beneficial effects of the present application are that:

[0025] 1. The single image snow removal method based on the GNN-transformer architecture provided in the application can remove snow from a single snow-containing image in a rainy and snowy weather background, and the snow-containing image to be removed is input into a GNN-transformer framework, and a ResNet trained by contrast learning is used to realize image-based graph construction. Secondly, the image to be tested is input into the GNN-transformer framework, a trained GCN model is used to extract features of the snow-containing image, and an attention mechanism is introduced to locate the snowflake position, aggregate feature information, and obtain snowflake feature mapping. Finally, the extracted snowflake feature mapping is converted back to the image space to form a snowflake attention map, which is input into a GAN for snow removal operation, and finally an image removed of snow is output, thereby improving the snow removal capability of the image and greatly improving the precision and efficiency of the image snow removal task.

[0026] 2. The single image snow removal method based on the GNN-transformer architecture provided in the application can interpret the pixels in the image to be removed as nodes having mutual correlations in a graph structure, and the image processing method is more flexible, the relationship between the local structure and the global structure of the sample is constructed, the real modeling of the overall structure data of the snow-containing image is realized, and the control of the layout of the snowflakes in the image to be removed and the attribute relationship thereof is improved.

[0027] 3. The single image snow removal method based on the GNN-transformer architecture provided in the application uses a GCN model to extract features of the snow-containing image, and in the research direction of image clarification processing in actual severe weather, the features of irregular and complex objects such as rain and snow in the image can be more effectively learned, the parameter amount is reduced, the training speed is improved, and the training efficiency of the model is greatly improved.

[0028] 4. The single image snow removal method based on the GNN-transformer architecture provided in the application introduces an attention mechanism to locate the position of the snowflake nodes in the image, can more effectively learn the complex snowflake structure features in the image, and greatly improves the precision and efficiency of the single image snow removal task.

[0029] 5. The single image snow removal method based on the GNN-transformer architecture provided in the application adopts a cross-entropy function as a loss function for target frame regression, and since CIOU Loss measures from three aspects of overlapping area, center point distance and aspect ratio, the effect of prediction frame regression is better. BRIEF DESCRIPTION OF DRAWINGS

[0030] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0031] Figure 1A single image snow removal method based on a GNN-transformer architecture is provided.

[0032] Figure 2 A GNN-transformer architecture is provided.

[0033] Figure 3 An embedding vector generation process is provided.

[0034] Figure 4 A generative adversarial network structure is provided. DETAILED DESCRIPTION

[0035] The application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended for the purpose of interpretation of the related application, and not for the limitation of the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0036] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0037] The present embodiment provides a single image snow removal method based on a GNN-transformer architecture, comprising the following steps:

[0038] S1: First, input the image that needs to perform the snow removal task into the GNN-transformer architecture, and divide it into patches of the same size, each patch represents a node in the graph structure, use the residual network trained by contrast learning to obtain the embedding vector of each patch, so as to realize the graph construction based on the image;

[0039] S2: Use the trained GCN model to extract features from the snow-containing image, and introduce an attention mechanism to select the nodes containing snowflakes in the graph, and aggregate information through the mechanism to obtain the feature mapping of the snowflake graph;

[0040] S3: Convert the extracted snowflake feature mapping back to the image space to form a snowflake attention map, and input it into the GAN for snow removal operation, and finally output the snow-removed image.

[0041] In the present embodiment, please refer to Table 1 for data set category distribution. It should be noted that the Snow100K data set is used, which contains different snowflake scenes of different sizes and different scenes. 8000 snowflake masks and 10000 clean backgrounds are used to combine 28620 snow-containing images. The data set is divided into training set and test set in the ratio of 8:2. The data set category distribution is shown in Table 1.

[0042]

[0043] Table 1. Dataset class distribution

[0044] In this embodiment, an image of size WxHx3 to be de-snowed is divided into N patches of the same size, each patch representing a node in the graph. The constructed graph G = (V, E) has V representing the set of image patches and E representing the set of edges between nodes in V, indicating whether two patches are adjacent.

[0045] A ResNet is trained using contrastive learning, which maximizes the consistency between two different augmented views of the same image patch through a contrastive loss in the latent space, resulting in embedding representations that are used to construct the graph structure, see Figure 3 Embedding vector generation process.

[0046] In generating embedding representations, first the de-snowed image is divided into N patches of the same size, and a small batch of K patches is randomly sampled. For each patch (p), two different data augmentations are applied, resulting in two augmented patches: p i and p j . Two augmented patches from the same patch are represented as a positive pair, and for a small batch of K patches, there are a total of 2K augmented patches. Given a positive pair, the other 1-2K augmented patches are considered as negative samples.

[0047] Subsequently, a representative embedding vector (f i , f j ) is extracted from each augmented patch (p i , p j ) using a ResNet. The embedding vectors are mapped to a latent space (z i , z j ) by a projection head, where a contrastive learning loss is applied. The contrastive learning loss function for a positive pair of augmented patches (p i , p j ) is defined as follows:

[0048]

[0049] where is an indicator function that is 1 when k≠i and τ represents a temperature parameter. In addition, sim(u, v) = u T v / ||u|||v|| represents the dot product (i.e., cosine similarity) between L2-normalized u and v.

[0050] In the ResNet model training, the patches are densely cropped without overlap and processed as individual images. The final loss is calculated through all the positive pairs, including (i, j) and (j, i) in a mini-batch. After the model converges, we keep the feature extractor and use it to compute the feature vectors from the patches of the snow-removed image.

[0051] In the graph construction phase, these computed feature vectors are used as node features. Specifically, we get a node-specific feature matrix F, F ∈ IR N×D Correspondingly, the information in each node is converted into a feature vector f i , F = {f1, f2, f3, ··· f N ,} and D is the feature dimension.

[0052] where f i is the D-dimensional embedding vector obtained by training the Resnet using contrastive learning, and N is the number of patches of the snow-removed image. We define an edge between a pair of nodes according to the spatial position of the corresponding patches on the snow-removed image. If patch i is a neighbor of patch j on the WSI, an edge is created between node i and node i, and set A ij = 1 and A ji = 1, otherwise A ij = 0 and A ji = 0. The present application uses the feature node matrix F and the adjacency matrix A to construct a graph to represent each snow-removed image.

[0053] In this embodiment, the snow-removed image is input into the GNN-transformer framework, and after the graph structure is constructed, the trained GCN model is used to extract features from the snow-removed image.

[0054] In training the GCN model, the node feature information in the graph is aggregated and updated using convolution. In the process of training the GCN model, the input graph is convolved with a trainable matrix A, and the loss function of the training weight matrix w 0 and w 1 is calculated by the backpropagation algorithm. If the loss does not decrease for several epochs, the training process will stop, and the final model learns the node features of the microphone normal sample graph.

[0055] The GCN performs information propagation and aggregation operations in the graph, defined as a trainable matrix A, which performs node information propagation and aggregation operations in the formula

[0056] where H m ​is the input of the m-th graph convolution layer, and H1 is initialized with the node feature matrix F.

[0057] A represents the adjacency matrix of the graph G = (V, E), is the degree matrix of A.

[0058] where, I N is an N x N identity matrix, and ReLU is an activation function.

[0059] In this embodiment, a transformer layer is introduced to interpret the data of the graph structure. It selects the most important nodes in the graph and aggregates information through the attention mechanism. For the graph, the feature nodes are taken as tokens in the sequence, and the position information of the nodes is represented by the adjacency matrix A.

[0060] In the transformer layer, suppose x ∈ IR N×D is the sequence of patches (feature nodes) in the graph, where N is the number of patches, and D is the embedding dimension of each patch. We calculate the q vector, k vector and v vector. The attention weight A ij is based on the pairwise similarity between the two patches in the sequence and the respective query q vector and k vector in the following equation:

[0061] [q,k,v]=xU qkv ,

[0062] where D h is usually set to D / k for easy calculation and to keep the number of parameters unchanged when k changes.

[0063]

[0064] where,

[0065] SA(x)=Av

[0066] MSA(x)=x)=[SA1(x);SA2(x);...SA k (x)]×U msa

[0067] where MSA is a multi-head self-attention mechanism that involves combining the knowledge explored by k self-attention operations called "heads". It projects the concatenated output of the self-attention as follows:

[0068] SA(x)=Av

[0069] ​The goal of the transformer layer is to learn a mapping: H→T, H is the graph space, and T is the transformer space. We define the mapping of H→T as:

[0070] t0= [x class ; h (1) ; h (2) ;... ; h (N) ], h (i) ∈ H

[0071] t′ l = MSA(LN(t l-1 ))+t l-1 , l = 1...L

[0072] t l = MSA(LN(t1))+t l , l = 1...L

[0073] where LN is the layer norm, L is the number of MSA blocks, and MSA is the multi-head self-attention mechanism.

[0074] In this embodiment, the extracted feature mapping f i is converted back to the image space by the reconstruction network R to form a snowflake attention map. In this part, we first upsample f i to the original resolution using a 3x3 deconvolution layer. The filter number is 64, and the stride is set to 2. Then, experiments are performed on 4 residual blocks with 2 3x3 convolution layers and PReLU activation layers. The formula is as follows:

[0075] f out = R(f i )

[0076] In this embodiment, we use a GAN model to perform snow removal on the image to be snow removed. For the specific structure, please refer to Figure 4 , the structure of the generative adversarial network. The GAN model includes two modules: a snow-free image generation module and a discriminator module.

[0077] In the snow-free image generation module, the obtained snowflake feature map is input into an encoder-decoder based on a U-shaped network structure together with an artificially synthesized snow-containing image as a guide image to repair the pixels covered by the snowflakes.

[0078] When training the U-shaped network, the network loss function is as follows:

[0079]

[0080] where R i represents the output image of the decoder, T i represents the corresponding snow-free background, and β iDifferent loss weights are represented at different scales.

[0081] The total loss function of the generator module is represented as follows:

[0082] l G =l AML ({A},M)+l({R},{T})+l PL ({O},{T})+l GAN (O)

[0083] wherein, l GAN (O)=log(1-D(O))

[0084] wherein, O represents the final output image of the generator.

[0085] In the discriminator module, the input image is discriminated to be a real image or a snow-removed image generated by the generator module. The loss function of the discriminator is defined as follows:

[0086] l D =-log(D(T))-log(1-D(G(I))) wherein, T is a real snow-free image, and I is an input image containing snow.

[0087] In the GAN training phase, the algorithm model mainly processes the following steps:

[0088] Step 1: input the image and the snow attention map into the snow-free image generation module, generate a snow-free image after processing by the module, and update the parameters in the network through the loss function;

[0089] Step 2: the training of the classification discriminator is completed by the snow-removed image generated by the generator module and the input real background image, and the network parameters of the module are updated through the loss function;

[0090] Through the above two-step cyclic training, the purpose of network training is finally achieved, and the snow-removal ability of the model is continuously improved to achieve the final single image snow-removal processing.

[0091] Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A method for single image snow removal based on GNN-transformer architecture, characterized in that, The method comprises the following steps: S1: first input the image needing to perform the snow removal task into the GNN-transformer architecture, the pixel of the image needing to perform the snow removal task is WxHx3, and the image is divided into patches of the same size, each patch represents a node in the graph structure, and an embedding vector of each patch is obtained by using a residual network trained by contrastive learning, so as to realize the graph construction based on the image; S2: using the trained graph convolution network model to extract the features of the snow image, and introducing an attention mechanism to select the snowflake nodes in the graph, and aggregating information through the mechanism to obtain the feature mapping of the snowflake graph; S3: converting the extracted snowflake feature mapping back to the image space to form a snowflake attention graph, and inputting the snowflake attention graph into a generative adversarial network for snow removal operation, and finally outputting the snow-removed image; In step S1, a residual network is trained by using contrastive learning, the network maximizes the consistency between two different augmented views of the same image patch through the contrastive loss in the latent space, thereby generating an embedding representation for the graph construction of the image to be snow-removed, so as to calculate the feature matrix of the nodes in the graph construction structure, and realize the graph G=(V, E) construction based on the image; for the feature matrix of the nodes, the feature vectors contained therein are used to define the information contained in the patch, each patch is a node in the image, and the dimension of the generated node feature vector is reduced from Wp x Hp x Cp to D x 1, wherein Wp, Hp and Cp are the width, height and channel of the image patch, and D x 1 is the dimension of the extracted feature vector.

2. The method of claim 1, wherein the GNN-transformer architecture is based on a GNN- transformer architecture. In step S2, the to-be-tested graph is input into the trained graph convolution network model, the trained graph convolution network model is used to extract the features of the snow-removed image by convolution, the node information is propagated and aggregated, and the attention mechanism is introduced to locate the position of the snowflake node in the graph, and finally the snowflake feature mapping is obtained.

3. The method of claim 2, wherein the GNN-transformer architecture is a GNN- transformer architecture based on a GNN and a transformer. The feature extraction is performed on the snow-removed graph, and the process includes: in each hidden layer of the graph convolution network model, the graph with features is convolved with a trainable matrix, and then the result is sent to the next layer; after passing through two hidden layers, the loss function of the training weight matrix w 0 and w 1 is calculated by the back propagation algorithm, and if the loss does not decrease in several epochs, the training process will be stopped, and finally the model learns the node features of the snow-removed graph.

4. The single image snow removal method based on GNN-transformer architecture according to claim 3, characterized in that, In the GCN structure, the cross-entropy function is used as the loss function.

5. The single image snow removal method based on GNN-transformer architecture according to claim 2, characterized in that, The attention mechanism is introduced to locate the snowflake position and aggregate the feature information. The transformer layer is introduced to finally learn the mapping process from the graph space to the transformer space, which includes: taking the feature nodes as tokens in the sequence, calculating the q vector, the k vector and the v vector, using the graph convolution layer in the form of the position embedding adjacency matrix to locate the snowflake position, and generating the snowflake attention graph.

6. The single image snow removal method based on GNN-transformer architecture according to claim 1, wherein, The snowflake attention graph is used as a reference graph to guide the snowflake position pixel repair of the snow-removed image generation module in the generative adversarial network, and finally the snow removal task is completed; The process of forming the snowflake attention graph includes: transforming the extracted snowflake feature mapping from the graph space back to the image space by using the reconstruction network R to form the snowflake attention graph.

7. The single image snow removal method based on GNN-transformer architecture according to claim 6, wherein, The snowflake attention map is input into the generative adversarial network for snow removal, and finally the image after snow removal is input, and the process includes: inputting the snowflake attention map as a guide image and an artificially synthesized image into a generator module in the generative adversarial network, repairing the pixels covered by the snowflake through an encoder-decoder, generating a snow-free image, and completing the single-image snow removal operation.

Images