A one-pass model method for image restoration based on pyramid transformer and information loss regularization

Abstract

The present invention relates to the field of computer vision technology, and in particular to a one-way model method for image restoration based on a pyramid converter and information loss regularization, comprising: firstly extracting multi-scale underlying features of images of various types of regression, then constructing a channel grouping multi-head attention mechanism and a spatial segmentation multi-head attention mechanism; using a grouping multi-head attention and a spatial segmentation multi-head attention component visual converter module, and constructing a multi-scale encoder and a multi-scale decoder based on the module; next, connecting the same layers of the multi-scale encoder and the multi-scale decoder to obtain a pyramid converter; finally, calculating an information loss regularization loss function to optimize the parameters in the pyramid converter. The present invention can make full use of the common advantages of specificity and versatility between various image restoration tasks, thereby solving the seesaw phenomenon problem that occurs in the integration process of various image restoration tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a one-way model method for image restoration based on a pyramid converter and information loss regularization. Background Art

[0002] Due to the physical limitations of camera sensors or unfavorable imaging conditions, image quality degradation is inevitable during the process of capturing, storing, transmitting, and rendering. Accordingly, the image restoration task attempts to restore the original real image from the degraded image. According to the type of image degradation factors processed, the image restoration task is specifically divided into image defogging, image deraining, image desnowing, image deblurring, image denoising, and low-light image enhancement. Since the image restoration task is crucial to high-level computer vision tasks such as image segmentation and object detection, this task has attracted widespread attention from many scholars in the field of computer vision.

[0003] The image restoration task is a highly ill-posed problem due to the infinite number of feasible solutions to invert the image degradation process. Therefore, existing works are devoted to designing various priors to constrain the solution space. Early works mainly relied on statistical observations to form hand-crafted priors, but this method has poor generality and is time-consuming to optimize. In contrast, the current popular methods use powerful deep learning techniques to learn more general implicit priors, achieving excellent image restoration results. In addition, various general methods have recently emerged that solve a variety of image restoration tasks well, indicating that deep neural networks are capable of solving different forms of inverse problems under a general framework. Therefore, this new research provides strong technical support for the integrated research of image restoration. The integrated research of image restoration aims to establish a unified model that accommodates multiple image restoration tasks, so as to bring image restoration research closer to real-world applications.

[0004] Obviously, image restoration integration falls within the scope of multi-task learning, which aims to approach the common optimal state of all specific tasks in the entire solution space. To solve this challenging problem, the initial methods focused on resolving feature entanglement to maintain the specificity between different tasks. These methods are called specificity-based methods. In these methods, due to the competition between different tasks, the final solution point assigns a greater weight to the strong task and may deviate from the optimal point. Differently, recent methods have begun to model the commonality of different tasks by mapping the various distributions of multiple image restoration tasks to a shared distribution. These methods are called commonality-based methods. In these methods, due to the pursuit of common promotion, the solution point may assign a greater weight to the weak task. Therefore, both specificity-based methods and commonality-based methods may encounter the seesaw phenomenon. In order to avoid the seesaw phenomenon, this application takes advantage of the common advantages of specificity and versatility to push the solution point in the direction of the optimal point. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings existing in the prior art, and to propose a one-pass model method for image restoration based on pyramid transformer and information loss regularization, which can make full use of the common advantages of specificity and versatility among various image restoration tasks to obtain an excellent one-pass integrated model for image restoration.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A one-pass model method for image restoration based on pyramid transformer and information loss regularization, comprising the following steps:

[0008] Step 1: Multi-scale underlying feature extraction: Input a batch of images of various return types. For any input image, use convolution operation to convert it into multi-scale tensor features;

[0009] Step 2: Construct a channel grouping multi-head attention mechanism: For any tensor feature, convert it into a matrix form, group its channels, and calculate the multi-head attention in each group;

[0010] Step 3: Construct a spatial segmentation multi-head attention mechanism: For any tensor feature, divide its space into multiple windows, convert it into a matrix form, and calculate the multi-head attention in each window;

[0011] Step 4: Construct a multi-scale encoder and a multi-scale decoder: Use the grouped multi-head attention and spatial segmentation multi-head attention component visual converter modules, and construct a multi-scale encoder and a multi-scale decoder based on the module;

[0012] Step 5: Construct a pyramid converter: connect the same layers of the multi-scale encoder and the multi-scale decoder to obtain a pyramid converter;

[0013] Step 6. Calculate the information loss regularization loss function: Input any input image into the pyramid converter, and calculate the information loss regularization loss function for the restored image and its corresponding true image. After optimizing the parameters in the pyramid converter based on the loss function, use the pyramid converter to complete the one-way image restoration task for various types of regression images.

[0014] Preferably, in step 1, the specific steps are as follows:

[0015] S11: Given any input degraded image I∈R H×W×3 , a 3×3 convolutional layer is used to extract multi-scale underlying features. At the 0th scale, the underlying features are represented as:

[0016] F0=Conv(I)

[0017] where \(F_0\in\mathbb{R}\) H×W×C , \(H\times W\) represents the spatial size of the tensor feature \(F_0\), \(C\) represents the number of channels of the tensor feature \(F_0\), and \(\text{Conv}(\cdot)\) represents the convolution operation;

[0018] S12: At the \(l\)-th scale, where \(0 < l < L\) and \(L\) is the total number of scales, the low-level feature is represented as:

[0019]

[0020] where is the representation of the degraded image \(I\) at the \((l - 1)\)-th scale, \(\text{DS}(\cdot)\) represents the downsampling operation, and \(\text{Conv}(\cdot)\) represents the convolution operation.

[0021] Preferably, in step 2, the specific steps are as follows:

[0022] S21: For any input tensor feature transform it into the form of a matrix, that is:

[0023] \(F'=\text{Reshape}(F)\)

[0024] where \(F'\in\mathbb{R}\) T×C , and \(\text{Reshape}(\cdot)\) represents the matrix transformation function;

[0025] S22: Divide the channels of \(F'\in\mathbb{R}\) T×C into \(G\) groups, and each group is represented as Assume the number of self-attention mechanisms is \(H\), then the calculation formula for the \(h\in H\) self-attention mechanism is:

[0026]

[0027] where is the \(h\in H\) attention feature of \(F_g\) 的 , represents the projection matrix, is the dimension of each attention feature; the above attention operation is performed \(H\) times, and then the \(H\) attention features are combined to obtain the attention feature of the channel-grouped multi-head attention mechanism of the tensor feature \(F\).

[0028] Preferably, in step 3, the specific steps are as follows:

[0029] S31: For any input tensor feature split it spatially into several small windows, assume the size of each window is \(M\times M\), and the tensor feature within any window is represented as \(F\) w \(\in\mathbb{R}\) M×M×C, transform it into a matrix form, that is:

[0030] F′ W =Reshape(F w )

[0031] in, Reshape(·) represents the matrix transformation function;

[0032] S32: Assuming that the number of self-attention mechanisms is H, the calculation formula for the h-th self-attention mechanism is:

[0033]

[0034] in, F w The h-th attention feature of represents the projection matrix, is the dimension of each attention feature; the above attention operation is performed H times, and then the H attention features are merged to obtain the attention feature of the tensor feature F based on the spatial segmentation multi-head attention mechanism.

[0035] Preferably, in step 4, the specific steps are as follows:

[0036] S41: Assume that the feature of the (l-1)th stage of the multi-scale encoder is represented as Input it into the downsampling layer to get:

[0037]

[0038] Where DS(·) represents the downsampling operation;

[0039] S42: Calculate the query, key value and value matrix, namely:

[0040]

[0041] in, is a linear projection matrix, the fusion of adjacent size features is expressed as:

[0042]

[0043] Where d represents and The number of channels in ;

[0044] S43: Fusion features generated by step S42 Perform the following encoding operations:

[0045]

[0046] in, is the feature output of the lth stage of the multi-scale encoder, LN(·) represents the regularization operation, FFN(·) represents the fully connected operation of the neural network, GCA(·) represents the channel grouping multi-head attention mechanism in step 2, and WSA(·) represents the spatial segmentation multi-head attention mechanism in step 3;

[0047] S44: Assume that the feature of the (l-1)th stage of the multi-scale decoder is represented as Input it into the downsampling layer to get:

[0048]

[0049] Among them, US(·) represents the upsampling operation;

[0050] S45: Features obtained from step S44 Perform the following decoding operations:

[0051]

[0052] in, is the feature output of the l-th stage of the multi-scale decoder, LN(·) represents the regularization operation, FFN(·) represents the fully connected operation of the neural network, GCA(·) represents the channel grouping multi-head attention mechanism in step 2, and WSA(·) represents the spatial segmentation multi-head attention mechanism in step 3.

[0053] Preferably, in step 5, the specific steps are as follows:

[0054] S51: Assuming that the multi-scale encoding process in S41, S42, and S43 is integrated into E(·), and assuming that the multi-scale decoding process in S44 and S45 is integrated into D(·), the backbone network is expressed as:

[0055]

[0056] in, Indicates a cascade operation;

[0057] S52: Randomly extract a batch of images of multiple regression types, the number of images in each batch is N, given any input degraded image I i ∈R H×W×3 , 1<i<N, the features after the backbone network are expressed as:

[0058] F i =B(I i )

[0059] S53: For feature F i, assuming that the post-processing operation is represented by Post(·), its specific operation is as follows:

[0060]

[0061] S54: Cascading the backbone network B(·) and the post-processing Post(·), we get a pyramid converter, namely:

[0062]

[0063] in, Indicates a cascade operation.

[0064] Preferably, in step 6, the specific steps are as follows:

[0065] S61: Randomly extract a batch of images of multiple regression types, the number of images in each batch is N, given any input degraded image I i ∈R H×W×3 , which is fed into the pyramid transformer to obtain the following restored image:

[0066]

[0067] S62: Assume that the degraded image I i The true value image is denoted as y i ∈R H×W×3 ,but With y i The reconstruction loss calculation formula between is:

[0068]

[0069] Among them, |·|1 is the F-norm;

[0070] S63: Calculate the degraded image I i The distance between the image and the true value is used to obtain the initial weight coefficient, which is expressed as:

[0071]

[0072] The weight coefficient W is adjusted using power series conversion, and its expression is:

[0073]

[0074] Among them, ε is a very small constant, α and β are parameters to be adjusted;

[0075] S64: The expression of the information loss regularization loss function for optimizing the pyramid transformer is:

[0076] L=L R ×W

[0077] After completing the training of the pyramid converter using the gradient descent algorithm according to the above formula, the pyramid converter is used to complete the one-way image restoration task for various regression type images.

[0078] Compared with the prior art, the present invention has the following beneficial effects:

[0079] The present invention can fully utilize the common advantages of specificity and versatility among various image restoration tasks to obtain an excellent one-way model for image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0081] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the advantages and features of the present invention, thereby making a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0082] The present invention is illustrated by way of example. Figure 1 Schematic diagram of a one-way model of image restoration based on a pyramid converter and information loss regularization according to the present invention. Figure 1 As shown in FIG. 1 , a one-way integrated model method for image restoration based on pyramid transformer and information loss regularization is presented. The specific steps are as follows:

[0083] Step 1: Multi-scale underlying feature extraction, given any input degraded image I∈R H×W×3 , use 3×3 convolutional layers to extract multi-scale underlying features, and obtain the underlying features F0 at the 0th scale and the underlying features at the l-1th scale

[0084] Step 2: Construct a channel grouping multi-head attention mechanism for any input tensor feature

[0085] Transform it into a matrix form F′∈R T×C , divide the channels of F′ into G groups, each group is represented by Perform H self-attention calculations on it, and then merge the H attention features to obtain the attention features of the tensor feature F based on the channel grouping multi-head attention mechanism.

[0086] Step 3: Construct a spatial segmentation multi-head attention mechanism for any input tensor feature It is divided into several small windows in space. Assuming that the size of each window is M×M, the tensor feature in any window is represented as F w ∈R M×M×C . Transform it into the form of matrix F′ W , perform H self-attention calculations on it, and then merge the H attention features to obtain the attention features of the tensor feature F based on the spatial segmentation multi-head attention mechanism.

[0087] Step 4: Construct a multi-scale encoder and a multi-scale decoder. Assume that the feature of the (l-1)th stage of the multi-scale encoder is represented as Input it to the downsampling layer to get The cross attention mechanism is used to fuse the adjacent size features to obtain Then it is encoded. Assume that the feature of the (l-1)th stage of the multi-scale decoder is represented as Input it to the downsampling layer to get

[0088] Then perform a decoding operation on it.

[0089] Step 5: Construct a pyramid converter, connect the equivalent layers of the multi-scale encoder and the multi-scale decoder to obtain the backbone network B, and post-process the features processed by the backbone network to obtain the pyramid converter T.

[0090] Step 6: Calculate the information loss regularization loss function, randomly select a batch of images of various regression types, the number of images in each batch is N, given any input degraded image I i ∈R H×W×3 , which is input into the pyramid converter to obtain the following restored image Assume that the degraded image I i The true value image is denoted as y i ∈R H×W×3 ,but With y i The reconstruction loss L between R , calculate the degraded image I i The distance between the real image and the real image is used to obtain the weight coefficient W, based on L R The information loss regularization loss function is obtained by using W. After the pyramid converter is trained using the gradient descent algorithm, the golden converter is used to complete the one-way image restoration task for various types of regression images.

[0091] Table 1 shows the performance of the present invention on the image denoising subtask.

[0092]

[0093] Table 2 Performance of the present invention on image defogging, image deraining and image desnowing subtasks

[0094]

[0095] In summary, the one-way image restoration model developed in the present invention can fully utilize the common advantages of specificity and versatility among various image restoration tasks, thereby solving the seesaw phenomenon problem that occurs in the integration process of various image restoration tasks.

[0096] The description and practice disclosed in the present invention are easy to think and understand for ordinary technicians in the technical field, and several improvements and modifications can be made without departing from the principles of the present invention. Therefore, modifications or improvements made without departing from the spirit of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A one-pass model method for image restoration based on pyramid transformer and information loss regularization, characterized in that: It includes the following steps: Step 1, multi-scale low-level feature extraction: Input a batch of images of multiple return types. For any input image, use convolution operations to convert it into multi-scale tensor features; Step 2, construct a channel-grouped multi-head attention mechanism: For any tensor feature, convert it into a matrix form, group its channels, and calculate multi-head attention within each group; In Step 2, the specific steps are as follows: S21: Tensor features for any input Transform it into matrix form, namely: F′ = Reshape(F) Among them, F′∈R T×C , Reshape(·) represents the matrix transformation function; S22: F′∈R T×C The channels are divided into G groups, each group is represented by Assuming that the number of self-attention mechanisms is H, the calculation formula for the h∈H self-attention mechanism is: in, F g The h∈Hth attention feature of represents the projection matrix, is the dimension of each attention feature; the above attention operation is performed H times, and then the H attention features are merged to obtain the attention feature of the tensor feature F based on the channel grouping multi-head attention mechanism; Step 3, construct a space-partitioned multi-head attention mechanism: For any tensor feature, partition its space into multiple windows, convert it into a matrix form, and calculate multi-head attention within each window; Step 4, construct a multi-scale encoder and a multi-scale decoder: Use the grouped multi-head attention and space-partitioned multi-head attention components in the vision transformer module, and construct a multi-scale encoder and a multi-scale decoder based on this module; Step 5, construct a pyramid transformer: Connect the equivalent layers of the multi-scale encoder and the multi-scale decoder to obtain a pyramid transformer; Step 6, calculate the information loss regularization loss function: Input any input image into the pyramid transformer to obtain the restored image, and calculate the information loss regularization loss function with its corresponding ground truth image. After optimizing the parameters in the pyramid transformer based on this loss function, use the pyramid transformer to complete the one-way image restoration task for images of multiple return types.

2. The one-pass model method for image restoration based on pyramid transformer and information loss regularization according to claim 1, characterized in that: In Step 1, the specific steps are as follows: s 11 : Given any input degraded image I∈R H×W×3 , a 3×3 convolutional layer is used to extract multi-scale underlying features. At the 0th scale, the underlying features are represented as: F0 = Conv(I) Among them, F0∈R H×W×C , H×W represents the spatial size of the tensor feature F0, C represents the number of channels of the tensor feature F0, and Conv(·) represents the convolution operation; S12: At the l-th scale, where 0 < l < L and L is the total number of scales, the low-level features are represented as: in, is the representation of the degraded image I at the l-1th scale, DS(·) represents the downsampling operation, and Conv(·) represents the convolution operation.

3. The one-pass model method for image restoration based on pyramid transformer and information loss regularization according to claim 1, characterized in that: In Step 3, the specific steps are as follows: S31: Tensor features for any input It is divided into several small windows in space. Assuming that the size of each window is M×M, the tensor feature in any window is represented as F w ∈R M×M×C , transform it into a matrix form, that is: F′ W =Reshape(F w ) in, Reshape(·) represents the matrix transformation function; S32: Assume the number of self-attention mechanisms is H, then the calculation formula for the h-th self-attention mechanism is: in, F w The h-th attention feature of represents the projection matrix, is the dimension of each attention feature; the above attention operation is performed H times, and then the H attention features are merged to obtain the attention feature of the tensor feature F based on the spatial segmentation multi-head attention mechanism.

4. The one-pass model method for image restoration based on pyramid transformer and information loss regularization according to claim 1, characterized in that: In Step 4, the specific steps are as follows: S41: Assume that the feature of the (l-1)th stage of the multi-scale encoder is represented as Input it into the downsampling layer to get: where DS(·) represents the downsampling operation; S42: Calculate the query, key, and value matrices, that is: in, is a linear projection matrix, the fusion of adjacent size features is expressed as: Where d represents and The number of channels in ; S43: Fusion features generated by step S42 Perform the following encoding operations: in, is the feature output of the lth stage of the multi-scale encoder, LN(·) represents the regularization operation, FFN(·) represents the fully connected operation of the neural network, GCA(·) represents the channel grouping multi-head attention mechanism in step 2, and WSA(·) represents the spatial segmentation multi-head attention mechanism in step 3; S44: Assume that the feature of the (l-1)th stage of the multi-scale decoder is represented as Input it into the downsampling layer to get: where US(·) represents the upsampling operation; S45: Features obtained from step S44 Perform the following decoding operations: in, is the feature output of the l-th stage of the multi-scale decoder, LN(·) represents the regularization operation, FFN(·) represents the fully connected operation of the neural network, GCA(·) represents the channel grouping multi-head attention mechanism in step 2, and WSA(·) represents the spatial segmentation multi-head attention mechanism in step 3.

5. The one-pass model method for image restoration based on pyramid transformer and information loss regularization according to claim 4, characterized in that: In Step 5, the specific steps are as follows: S51: Assume that the multi-scale encoding processes in S41, S42, and S43 are integrated into E(·), and assume that the multi-scale decoding processes in S44 and S45 are integrated into D(·), then the backbone network is represented as: in, Indicates a cascade operation; S52: Randomly extract a batch of images of multiple regression types, the number of images in each batch is N, given any input degraded image I i ∈R H×W×3 , 1<i<N, the features after the backbone network are expressed as: F i =B(I i ) S53: For feature F i , assuming that the post-processing operation is represented by Post(·), its specific operation is as follows: S54: Concatenate the backbone network B(·) and the post-processing Post(·), then obtain the pyramid transformer, that is: in, Indicates a cascade operation.

6. The one-pass model method for image restoration based on pyramid transformer and information loss regularization according to claim 1, characterized in that: In Step 6, the specific steps are as follows: S61: Randomly extract a batch of images of multiple regression types, the number of images in each batch is N, given any input degraded image I i ∈R H×W×3 , which is fed into the pyramid transformer to obtain the following restored image: S62: Assume that the degraded image I i The true value image is denoted as y i ∈R H×W×3 ,but With y i The reconstruction loss calculation formula between is: where |·|1 is the F-norm; S63: Calculate the degraded image I i The distance between the image and the true value is used to obtain the initial weight coefficient, which is expressed as: Adjust the weight coefficient W using power series transformation, and its expression is: where ε is a very small constant, and α and β are parameters to be determined; S64: The expression of the information loss regularization loss function for optimizing the pyramid transformer is: L=L R ×W After training the pyramid transformer using the gradient descent algorithm according to the above formula, use the pyramid transformer to complete the one-way image restoration task for images of multiple return types.

Images