Image restoration method based on cascade deformable convolution
Through the combination of cascaded deformable convolution and adjacent layer conduction attention module, the problems of semantic ambiguity and information gap in image repair are solved, and high-quality image repair effect is achieved, especially in the processing of large hole damaged images. .
Patent Information
- Application Number
- CN202410928739.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-15
AI Technical Summary
The existing image repair technology fails to effectively consider semantic ambiguity and information gap when processing large holes and damaged images, resulting in semantic errors and information distortion, which makes the repair effect poor.
Using the image repair method based on cascaded deformable convolution, the cascaded deformable convolution encoder and decoder combine with adjacent layers to conduct attention modules to achieve efficient fusion and residual reconstruction of depth features, and design cascaded losses Functions strengthen the constraints of the intermediate features of the decoder and gradually improve the repair effect.
High-quality image repair is achieved, reducing semantic errors and improving repair effects, especially when dealing with large hole-damaged images, which can reconstruct coherent texture content more robustly.
Smart Images

Figure CN118967519A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to an image restoration method based on cascaded deformable convolution. Background Art
[0002] Image restoration is an important research topic in the field of computer vision and image processing. With the excellent performance of deep learning in the field of vision, the frontier of the field of vision is basically occupied by deep learning technology. The intersection of graphics and vision is a hot topic of discussion and research, especially image editing and image generation. Image restoration is also a problem between image editing and image generation. Image restoration can be simply summarized as a problem of estimating the value of a missing part. How to efficiently repair large and irregular images is still a problem that needs to be improved and solved.
[0003] In recent years, many scholars have also contributed many algorithms in the field of deep learning image restoration, which can be summarized into three categories: 1) special convolutional network image restoration methods; 2) clue prior progressive restoration algorithms; 3) restoration algorithms based on attention mechanisms. For example, the partial convolutional network proposed by Guilin Liu et al. and the subsequent deformable convolutional network can repair irregular damaged images and borrow feature blocks from distant areas, but these methods do not take into account the problem of semantic ambiguity. As long as the damaged image has large holes, the correlation between the damaged center and the known area will be weakened, resulting in semantic errors. Jie Wu et al. used the intermediate prior to first repair the potential contour of the image, and then continued to use the two-stage self-attention to fill in the features to enrich the texture details. However, due to the high computational cost of self-attention, and some methods are prone to information distortion during the decoding process due to the large number of iterations. Later, many researchers focused on how to narrow the huge restoration information gap, and successively proposed more powerful attention mechanisms to capture potential correlations or obtain additional information from auxiliary conditions such as the frequency domain, but still did not consider the lack of reference information for the restoration task itself, and the improvement obtained by these improvements was still insufficient. Summary of the invention
[0004] The purpose of the present invention is to provide an image restoration method based on cascaded deformable convolutions. On the basis of achieving high-performance restoration of images, it alleviates the current restoration information gap with a progressive strategy and introduces a cascaded deformable convolution architecture, which can robustly reconstruct missing structures and fill in coherent texture content, ultimately achieving high-quality image restoration.
[0005] To achieve the above object, the technical solution adopted by the present invention is:
[0006] The present invention provides an image restoration method based on cascaded deformable convolution, and the image restoration method based on cascaded deformable convolution comprises:
[0007] Step S1: Obtain the damaged sample image F to be repaired in ;
[0008] Step S2: The damaged sample image F to be repaired in Input a cascaded deformable convolution encoder to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7, wherein the cascaded deformable convolution encoder includes five convolution downsampling layers with a step size of 2 and two layers of deformable convolution blocks;
[0009] Step S3: Input the deep feature maps X1, X2, X3, X4, X5, X6 and X7 into the cascaded deformable convolution decoder to obtain multi-size intermediate feature maps. k∈{1,2,3,4,5,6}, the cascaded deformable convolutional decoder includes six adjacent layer conduction attention modules;
[0010] Step S4: the damaged sample image F to be repaired in The downsampling operation is performed to obtain the size of the intermediate feature map Corresponding feature map of the sample to be repaired The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k , and calculate the residual reconstruction features CL k The error between the real image and the damaged sample image F is used to obtain the cascade loss. The cascade deformable convolution encoder and the cascade deformable convolution decoder are trained and updated based on the cascade loss. The real image is the damaged sample image to be repaired. in The corresponding undamaged image;
[0011] Step S5, in the inference phase, the damaged image to be repaired is processed based on the cascaded deformable convolution encoder and cascaded deformable convolution decoder that have been updated through training to obtain an intermediate feature map For the intermediate feature map A double upsampling operation and a convolution operation with an output channel of 3 and a convolution kernel of 3 are performed to obtain the final repaired image.
[0012] As a preference, the damaged sample image F to be repaired in Input the cascaded deformable convolution encoder to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7, including:
[0013] First, the damaged sample image to be repaired F in Perform 5 convolution downsampling operations with a step size of 2 and a convolution kernel of 3 to obtain downsampled feature maps X1, X2, X3, X4 and X5 of various sizes as depth feature maps;
[0014] Then the downsampled feature map X5 is input into the deformable convolution block, where the feature X is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor , and then connect the feature X through residual connection conv , X5 and X defor Splicing is performed in the channel dimension, and finally a convolution with a convolution kernel of 3 is used to obtain the offset result feature map X ret , for the offset result feature map X ret Continue to use the convolution operation with a step size of 2 to obtain the downsampled feature map X6 as the depth feature map;
[0015] Finally, the downsampled feature map X6 is input into the deformable convolution block, where the feature X is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv1 , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor1 , and then connect the feature X through residual connection conv1 , X6 and X defor1 Splicing is performed in the channel dimension, and finally a convolution with a convolution kernel of 3 is used to obtain the offset result feature map X ret1 , for the offset result feature map X ret1 Continue to use the convolution operation with a step size of 2 to obtain the downsampled feature map X7 as the depth feature map.
[0016] Preferably, the depth feature maps X1, X2, X3, X4, X5, X6 and X7 are input into the cascaded deformable convolution decoder to obtain intermediate feature maps of multiple sizes. include:
[0017] The deep feature map X7 is used as the low-resolution input F high , taking the deep feature map X6 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the first adjacent layer conduction attention module to obtain the intermediate feature map output by the first adjacent layer conduction attention module
[0018] The intermediate feature map As low-resolution input F high , the deep feature map X5 is used as the high-resolution input F low , input low resolution into F high and high resolution input F lowSend it to the second adjacent layer conduction attention module to obtain the intermediate feature map output by the second adjacent layer conduction attention module
[0019] The intermediate feature map As low-resolution input F high , taking the deep feature map X4 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the third adjacent layer conduction attention module to obtain the intermediate feature map output by the third adjacent layer conduction attention module
[0020] The intermediate feature map As low-resolution input F high , taking the deep feature map X3 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fourth adjacent layer conduction attention module to obtain the intermediate feature map output by the fourth adjacent layer conduction attention module
[0021] The intermediate feature map As low-resolution input F high , taking the deep feature map X2 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fifth adjacent layer conduction attention module to obtain the intermediate feature map output by the fifth adjacent layer conduction attention module
[0022] The intermediate feature map As low-resolution input F high , taking the deep feature map X1 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the sixth adjacent layer conduction attention module to obtain the intermediate feature map output by the sixth adjacent layer conduction attention module
[0023] Preferably, the adjacent layer conduction attention module operates as follows:
[0024] Low resolution input F high First, the size is enlarged to the same size as the high-resolution input F through interpolation. low Align and get the feature map F r ; Then the high resolution is input to F lowAnd the feature map F r Concatenate in the channel dimension and perform convolution operation to achieve feature fusion to obtain feature map F c , and then the feature map F c and low resolution input F high Send to the multi-scale context fusion block output feature I out ; Finally, the generated feature I out and feature map F c Perform channel splicing and convolution operations to achieve feature fusion and obtain intermediate feature maps
[0025] As a preference, the feature map F c and low resolution input F high Send to the multi-scale context fusion block output feature I out ,include:
[0026] Enter low resolution into F high Block segmentation is performed to obtain a 3×3 feature block sequence. The formula for calculating pixel attention based on the feature block is:
[0027]
[0028] in, represents the pixel-level similarity between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, p i represents the i-th feature block extracted from the complete region, p j represents the jth feature block extracted from the damaged area;
[0029] The pixel-level similarity obtained Apply the softmax operation to calculate the attention score of each feature block, which is defined as follows:
[0030]
[0031] in, represents the attention score between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, and N represents the total number of feature blocks extracted from the damaged area;
[0032] Using the deconvolution operation, the attention scores of the high-level feature maps are used to fill the damaged areas of the low-level feature maps. The formula is as follows:
[0033]
[0034] in, The feature block As the deconvolution kernel in the attention score matrix The deconvolution operation performed on From the feature map F c The 3×3 feature block extracted from the feature map F c The total number of feature blocks extracted from For all attention scores The attention score matrix is composed of is the learned weighted affinity matrix, i.e., the complete feature map I rec ;
[0035] Using four sets of dilated convolutional layers with different dilation rates, the complete feature map I rec Further transformed into a multi-scale semantic feature map I ret , the specific process is as follows:
[0036]
[0037] Among them, g r=k (·)(k∈{1,2,4,8}) represents four dilated convolutions with dilation rates of 1, 2, 4, and 8, respectively. ⊕ represents the corresponding element addition operation.
[0038] Finally, the feature map F c and multi-scale semantic feature maps I ret Combined to generate output feature I out :
[0039] I out =μ(g(Concat(I ret ,F c )))
[0040] Among them, μ(·) is the activation function, g(·) is the mapping function implemented by the convolutional layer, and Concat(·) is the channel concatenation operation.
[0041] Preferably, the residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired. k ,include:
[0042]
[0043] Among them, CL k represents the residual reconstruction feature of the kth size, Represents the feature map of the sample to be repaired of the kth size, represents the intermediate feature map of the kth size, ⊙ represents the element-wise multiplication operation, are the convolution function and activation function, where the convolution operation output channel of the convolution function is fixed to 3, and v(·) is a convolution operation containing three layers with kernel sizes of 1, 3, and 1 respectively.
[0044] Preferably, the calculation of each residual reconstruction feature CL k The error between the real image and the real image is obtained by cascading the loss, including:
[0045]
[0046] Among them, L cl is the cascade loss, F gt For real images.
[0047] The present invention also provides an image restoration method based on cascaded deformable convolution, the image restoration method based on cascaded deformable convolution comprising:
[0048] Step S1: Obtain the damaged sample image F to be repaired in ;
[0049] Step S2: The damaged sample image F to be repaired in Input a cascaded deformable convolution encoder to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7, wherein the cascaded deformable convolution encoder includes five convolution downsampling layers with a step size of 2 and two layers of deformable convolution blocks;
[0050] Step S3: Input the deep feature maps X1, X2, X3, X4, X5, X6 and X7 into the cascaded deformable convolution decoder to obtain multi-size intermediate feature maps. k∈{1,2,3,4,5,6}, and the intermediate feature map Input output module, the output module outputs the repaired image F out , the cascaded deformable convolutional decoder includes six adjacent layer conduction attention modules;
[0051] Step S4: the damaged sample image F to be repaired in The downsampling operation is performed to obtain the size of the intermediate feature map Corresponding feature map of the sample to be repaired The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k , and calculate the residual reconstruction features CL k The error between the real image and the original image is used to obtain the cascade loss. out Calculate the conventional loss, integrate the cascade loss and the conventional loss, train and update the cascade deformable convolution encoder, the cascade deformable convolution decoder and the output module, and the real image is the damaged sample image to be repaired F in The corresponding undamaged image;
[0052] Step S5, in the inference stage, the damaged image to be repaired is processed based on the cascaded deformable convolution encoder, the cascaded deformable convolution decoder and the output module that have been updated based on the training to obtain a repaired image, and the output module includes a double upsampling operation and a convolution operation with an output channel of 3 and a convolution kernel of 3.
[0053] Preferably, the conventional loss includes perceptual loss, style loss and absolute value loss.
[0054] The present invention provides an image restoration method based on cascaded deformable convolution, which ensures that the features of adjacent network layers can be efficiently fused and have good reconstruction and restoration performance. Then, the corresponding cascade loss is designed to strengthen the constraints on the intermediate features of the decoder, strengthen the guiding role of the restoration task, and achieve better restoration effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a framework diagram of the image restoration method based on cascaded deformable convolution of the present invention;
[0056] Figure 2 It is a flowchart of the multi-scale context fusion block of the present invention;
[0057] Figure 3 It is a flow chart of cascade loss of the present invention. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0060] Example 1
[0061] The present invention provides an image restoration method based on cascaded deformable convolutions, which fully utilizes the advantages of deformable convolutions and embeds deformable convolution blocks in the cascade structure to achieve efficient fusion of features of adjacent network layers and thus complete a restoration method with high-quality pixel filling.
[0062] Specifically, as attached Figure 1 As shown in Figure 2, the complete image restoration process is shown, including the following steps:
[0063] Step S1: Obtain the damaged sample image F to be repaired in .
[0064] In this embodiment, the damaged sample image to be repaired can be directly obtained for training, or the damaged sample image to be repaired can be simulated based on the intact image. When performing simulation acquisition, the intact image Y and the mask image are first obtained, and the corresponding elements are multiplied to simulate the real damaged sample image F to be repaired. in ,like Figure 1 The left input in F is visible in , from a visual observation, there are some irregular missing blocks in the image; the image Y comes from a high-quality face image dataset; the mask is a randomly lost binary image, in which the white area is defined as the damaged area of the image, and the loss rate of the damaged area in the entire image is controlled between 30% and 60%.
[0065] Step S2: Use a cascaded deformable convolutional encoder to perform a reconstruction on the damaged sample image F in Processing is performed to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7. The cascaded deformable convolution encoder structure includes: 5 layers of convolution downsampling layers with a step size of 2, 2 layers of deformable convolution layers, and the specific cascaded deformable convolution encoder network structure is shown in the attached figure. Figure 1 as shown in the lower left part of .
[0066] Step S21: The damaged sample image to be repaired with a channel number of 3 in The cascaded deformable convolution encoder is first used to repair the damaged sample image F in Five convolution operations with a stride of 2 and a convolution kernel of 3 are performed to obtain downsampled feature maps X1, X2, X3, X4 and X5 of various sizes as deep feature maps, with sizes of H / 2×W / 2, H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, respectively.
[0067] Step S22: Using ordinary convolution kernels to process the feature maps with damaged areas will obviously cause certain errors. Therefore, the method of the present invention uses a deformable convolution operation to extract features in the deep layer of the encoder. This operation supports the convolution kernel to adaptively explore suitable pixels outward when extracting local features. Figure 1 As shown in the upper left corner, the deep feature map X5 is input into the deformable convolution block, and the feature X is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor, since the deformable convolution operation can generate an offset in the feature mapping, the network can learn useful feature pixels through training; then X conv , X5 and X defor The concatenation is performed on the channel dimension, and finally a convolution with a convolution kernel of 3 is used to implement feature screening and dimension mapping to obtain the offset result feature map X ret .
[0068] Step S23: offset result feature map X ret Continue to use the convolution operation with a stride of 2 to obtain a downsampled feature map X6 of size H / 64×W / 64 as the depth feature map.
[0069] Step S24: The depth feature map X6 is input into the deformable convolution block. The feature X6 is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv1 , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor1 , since the deformable convolution operation can generate an offset in the feature mapping, the network can learn useful feature pixels through training; then X conv1 , X6 and X defor1 The concatenation is performed on the channel dimension, and finally a convolution with a convolution kernel of 3 is used to implement feature screening and dimension mapping to obtain the offset result feature map X ret1 .
[0070] Step S25: offset result feature map X ret1 Continue to use the convolution operation with a stride of 2 to obtain the downsampled feature map X7 of size H / 128×W / 128 as the depth feature map.
[0071] Step S3, the method of the present invention divides the multi-level feature map into high-resolution features with texture information and low-resolution features with global structural information. How to reasonably design a decoding strategy to efficiently fuse the texture and structural features between adjacent resolution features has become a key challenge for the image restoration task; the method uses the adjacent layer conduction attention module embedded in the cascaded deformable convolution decoder to process the feature maps X7 and X6 generated in step S2 to obtain the reconstructed feature map after restoration. The adjacent layer conduction attention module can integrate texture features and structural features to obtain a resolution-size upsampled feature map. The specific adjacent layer conduction attention module structure is shown in the attached Figure 1 .
[0072] Step S31, the deep feature map X7 is fed into a cascaded deformable convolution decoder, where a single-layer decoder is embedded with an adjacent-layer conduction attention module; first, the deep feature map X7 is used as the deep low-resolution input F of the adjacent-layer conduction attention module. high , and the higher shallow feature map X6 of the encoding layer is used as the high-resolution input F of the first adjacent layer conduction attention module low .
[0073] In the adjacent layer conduction attention module, F high First, the size is doubled to F by interpolation. low Align and get the feature map F r ; Then F low and F r Concatenate in the channel dimension and perform convolution operation to achieve feature fusion to obtain F c , and then add it to F high The function used in the calculation process of the multi-scale context fusion block is defined as follows:
[0074] F c =μ(d(g(Concat(ε(F high ),F low ))))
[0075] I out =MCB(F high ,F c )
[0076] Among them, I out Concat(·) is the output feature of the multi-scale context fusion block, the channel concatenation operation, ε(·) is the interpolation function, g(·) is the mapping function implemented by the convolution layer, d(·) is the deformable convolution operation, μ(·) is the activation function, and MCB(·) is the multi-scale context fusion block.
[0077] The multi-scale context fusion block is shown in the attached Figure 2 As shown, F high The feature map is divided into blocks to obtain a 3×3 block sequence p0...p r , at this time F high It is a reconstructed structural feature. The pixel attention calculation performed on it can make it pay more attention to the potential structure during subsequent texture fusion. The formula for calculating pixel attention is:
[0078]
[0079] in, represents the pixel-level similarity between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, p irepresents the i-th feature block extracted from the complete region, p j represents the jth feature block extracted from the damaged area.
[0080] The pixel-level similarity obtained Apply the softmax operation to calculate the attention score of each feature block, which is defined as follows:
[0081]
[0082] in, represents the attention score between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, and N represents the total number of feature blocks extracted from the damaged area.
[0083] Using the deconvolution operation, the attention scores of the high-level feature maps are used to fill the damaged areas of the low-level feature maps. The process can be described as follows:
[0084]
[0085] in, The feature block As the deconvolution kernel in the attention score matrix The deconvolution operation performed on From the feature map F c The 3×3 feature block extracted from the feature map F c The total number of feature blocks extracted from For all attention scores The attention score matrix is composed of is the learned weighted affinity matrix, i.e., the complete feature map I rec .
[0086] In order to obtain multi-scale semantic features, four groups of dilated convolutional layers with different dilation rates are used to convert the complete feature map I rec Further transformed into a multi-scale semantic feature map I ret , the specific process is as follows:
[0087]
[0088] Among them, g r=k (·)(k∈{1,2,4,8}) represents four dilated convolutions with dilation rates of 1, 2, 4, and 8 respectively, and ⊕ represents the corresponding element addition operation.
[0089] Finally, the feature map F c and multi-scale semantic feature maps I ret Combined to generate output feature I out :
[0090] I out =μ(g(Concat(I ret ,F c )))
[0091] Among them, μ(·) is the activation function, g(·) is the mapping function implemented by the convolutional layer, and Concat(·) is the channel concatenation operation.
[0092] Finally, the generated feature I out and residual connection F c Perform channel concatenation and convolution operations to achieve feature fusion The specific operation process is as follows:
[0093]
[0094] According to the above operation process, the multi-level cascade deformable convolution decoder can decode the deep features of the decoder And the same-layer encoder features X1, X2, X3, X4, X5 obtained by skip connection are used as input to generate reconstructed repair features k∈{1,2,3,4,5,6}, the specific steps are as follows:
[0095] Step S32: The intermediate feature map As low-resolution input F high , the deep feature map X5 is used as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the second adjacent layer conduction attention module to obtain the intermediate feature map output by the second adjacent layer conduction attention module
[0096] Step S33: The intermediate feature map As low-resolution input F high , taking the deep feature map X4 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the third adjacent layer conduction attention module to obtain the intermediate feature map output by the third adjacent layer conduction attention module
[0097] Step S34: The intermediate feature map As low-resolution input F high , taking the deep feature map X3 as the high-resolution input F low , input low resolution into F high and high resolution input F lowSend it to the fourth adjacent layer conduction attention module to obtain the intermediate feature map output by the fourth adjacent layer conduction attention module
[0098] Step S35: The intermediate feature map As low-resolution input F high , the deep feature map X2 is used as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fifth adjacent layer conduction attention module to obtain the intermediate feature map output by the fifth adjacent layer conduction attention module
[0099] Step S36: The intermediate feature map As low-resolution input F high , taking the deep feature map X1 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the sixth adjacent layer conduction attention module to obtain the intermediate feature map output by the sixth adjacent layer conduction attention module
[0100] Step S4: In the process of training the network model, it is necessary to design a loss function corresponding to image restoration to guide the overall network to perform reverse conduction, and then learn the corresponding optimization parameters; the classic loss functions in the current image restoration field are all for network output results. Considering the specific network structure of the method of the present invention, and the fact that this type of U-shaped network decoding process is prone to feature distortion and difficulty in convergence, this method designs a cascade loss function to constrain the decoder intermediate features. k∈{1,2,3,4,5,6}, thereby achieving better network parameter optimization guidance.
[0101] The present invention performs a repair on the damaged sample image F in The downsampling operation is performed to obtain the size of the intermediate feature map Corresponding feature map of the sample to be repaired The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k , and calculate the residual reconstruction features CL k The error between the real image and the damaged sample image to be repaired is used to obtain the cascade loss. The cascaded deformable convolution encoder and the cascaded deformable convolution decoder are updated based on the cascade loss. The real image is the damaged sample image to be repaired F in The corresponding undamaged image.
[0102] Step S41, cascade loss function as shown in the attached Figure 3 As shown, the output of any layer of the cascade decoder For example, As an input, the input sample feature map to be repaired As another input, is the damaged sample image to be repaired F in The size is changed, and the two data are combined to generate a residual reconstruction feature CL k To achieve the subsequent loss constraint, the formula is as follows:
[0103]
[0104] Among them, ⊙ represents the element-wise multiplication operation. By using the interpolation operation, we can ensure The size and Same; Function It is specifically used to convert feature maps with different channels into feature maps with fixed 3 channels, that is, it is a convolution function plus an activation function. It can be understood as a specific convolution layer for feature mapping, but the output channel of the convolution operation is fixed to 3. The function v(·) is a convolution operation with three layers and kernel sizes of 1, 3, and 1 respectively.
[0105] Step S42: Reconstruct the residual feature CL k , the method of the present invention requires that it and the real image F gt The error loss is calculated between to constrain the reconstruction process in the decoder. Since k∈{1,2,3,4,5,6}, this operation can be performed simultaneously at multiple levels of the decoder to achieve better constraints. The cascade loss L cl The specific formula is as follows:
[0106]
[0107] This embodiment uses a cascaded deformable convolution framework to ensure that the features of adjacent network layers can be efficiently fused and have good reconstruction and restoration performance. Then the corresponding cascade loss is designed to strengthen the constraints on the intermediate features of the decoder, strengthen the guiding role of the restoration task, and achieve better restoration results.
[0108] Step S5, in the inference phase, the damaged image to be repaired is processed based on the cascaded deformable convolution encoder and cascaded deformable convolution decoder that have been updated through training to obtain an intermediate feature map For the intermediate feature map The final restored image is obtained by performing a two-fold upsampling operation and a convolution operation with an output channel of 3 and a convolution kernel of 3. The parameters of the upsampling operation and the convolution operation can be obtained through pre-training.
[0109] It should be noted that the training phase and the inference phase are regular phases of the neural network module, but this does not mean that the two phases must be executed at the same time, that is, the process of this application may be to execute only the training phase, or only the inference phase (usually the inference phase is executed separately after the training phase).
[0110] The present invention utilizes cascaded deformable convolutional coding to realize the offset extraction of deep features; performs adjacent layer conduction attention calculation to integrate structural texture details for multi-level features; cascaded deformable convolutional decoding gradually expands the feature resolution to the size of the original image to obtain a complete repaired image; cascaded loss performs deep supervision on the intermediate features of all decoder levels to guide the rapid convergence of repair and reconstruction.
[0111] Example 2
[0112] The present invention provides an image restoration method based on cascaded deformable convolutions, which fully utilizes the advantages of deformable convolutions, embeds deformable convolution blocks in the cascade structure, and realizes efficient fusion of features of adjacent network layers to complete a restoration method for high-quality pixel filling. The main difference between this embodiment and embodiment 1 is that, in order to further improve the quality of the restored image, this embodiment, in addition to the cascade loss that can effectively improve the restoration effect, also introduces conventional losses in the field of image restoration.
[0113] Specifically, as attached Figure 1 As shown in Figure 2, the complete image restoration process is shown, including the following steps:
[0114] Step S1: Obtain the damaged sample image F to be repaired in .
[0115] In this embodiment, the damaged sample image to be repaired can be directly obtained for training, or the damaged sample image to be repaired can be simulated based on the intact image. When performing simulation acquisition, the intact image Y and the mask image are first obtained, and the corresponding elements are multiplied to simulate the real damaged sample image F to be repaired. in ,like Figure 1 The left input in F is visible in , from a visual observation, there are some irregular missing blocks in the image; the image Y comes from a high-quality face image dataset; the mask is a randomly lost binary image, in which the white area is defined as the damaged area of the image, and the loss rate of the damaged area in the entire image is controlled between 30% and 60%.
[0116] Step S2: Use a cascaded deformable convolutional encoder to perform a reconstruction on the damaged sample image F inProcessing is performed to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7. The cascaded deformable convolution encoder structure includes: 5 layers of convolution downsampling layers with a step size of 2, 2 layers of deformable convolution layers, and the specific cascaded deformable convolution encoder network structure is shown in the attached figure. Figure 1 as shown in the lower left part of .
[0117] Step S21: The damaged sample image to be repaired with a channel number of 3 in The cascaded deformable convolution encoder is first used to repair the damaged sample image F in Five convolution operations with a stride of 2 and a convolution kernel of 3 are performed to obtain downsampled feature maps X1, X2, X3, X4 and X5 of various sizes as deep feature maps, with sizes of H / 2×W / 2, H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, respectively.
[0118] Step S22: Using ordinary convolution kernels to process the feature maps with damaged areas will obviously cause certain errors. Therefore, the method of the present invention uses a deformable convolution operation to extract features in the deep layer of the encoder. This operation supports the convolution kernel to adaptively explore suitable pixels outward when extracting local features. Figure 1 As shown in the upper left corner, the deep feature map X5 is input into the deformable convolution block, and the feature X is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor , since the deformable convolution operation can generate an offset in the feature mapping, the network can learn useful feature pixels through training; then X conv , X5 and X defor The concatenation is performed on the channel dimension, and finally a convolution with a convolution kernel of 3 is used to implement feature screening and dimension mapping to obtain the offset result feature map X ret .
[0119] Step S23: offset result feature map X ret Continue to use the convolution operation with a stride of 2 to obtain a downsampled feature map X6 of size H / 64×W / 64 as the depth feature map.
[0120] Step S24: The depth feature map X6 is input into the deformable convolution block. The feature X6 is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv1 , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor1, since the deformable convolution operation can generate an offset in the feature mapping, the network can learn useful feature pixels through training; then X conv1 , X6 and X defor1 The concatenation is performed on the channel dimension, and finally a convolution with a convolution kernel of 3 is used to implement feature screening and dimension mapping to obtain the offset result feature map X ret1 .
[0121] Step S25: offset result feature map X ret1 Continue to use the convolution operation with a stride of 2 to obtain the downsampled feature map X7 of size H / 128×W / 128 as the depth feature map.
[0122] Step S3, the method of the present invention divides the multi-level feature map into high-resolution features with texture information and low-resolution features with global structural information. How to reasonably design a decoding strategy to efficiently fuse the texture and structural features between adjacent resolution features has become a key challenge for the image restoration task; the method uses the adjacent layer conduction attention module embedded in the cascaded deformable convolution decoder to process the feature maps X7 and X6 generated in step S2 to obtain the reconstructed feature map after restoration. The adjacent layer conduction attention module can integrate texture features and structural features to obtain a resolution-size upsampled feature map. The specific adjacent layer conduction attention module structure is shown in the attached Figure 1 .
[0123] Step S31, the deep feature map X7 is fed into a cascaded deformable convolution decoder, where a single-layer decoder is embedded with an adjacent-layer conduction attention module; first, the deep feature map X7 is used as the deep low-resolution input F of the adjacent-layer conduction attention module. high , and the higher shallow feature map X6 of the encoding layer is used as the high-resolution input F of the first adjacent layer conduction attention module low .
[0124] In the adjacent layer conduction attention module, F high First, the size is doubled to F by interpolation. low Align and get the feature map F r ; Then F low and F r Concatenate in the channel dimension and perform convolution operation to achieve feature fusion to obtain F c , and then add it to F high The function used in the calculation process of the multi-scale context fusion block is defined as follows:
[0125] F c =μ(d(g(Concat(ε(Fhigh ),F low ))))
[0126] I out =MCB(F high ,F c )
[0127] Among them, I out Concat(·) is the output feature of the multi-scale context fusion block, the channel concatenation operation, ε(·) is the interpolation function, g(·) is the mapping function implemented by the convolution layer, d(·) is the deformable convolution operation, μ(·) is the activation function, and MCB(·) is the multi-scale context fusion block.
[0128] The multi-scale context fusion block is shown in the attached Figure 2 As shown, F high The feature map is divided into blocks to obtain a 3×3 block sequence p0...p r , at this time F high It is a reconstructed structural feature. The pixel attention calculation performed on it can make it pay more attention to the potential structure during subsequent texture fusion. The formula for calculating pixel attention is:
[0129]
[0130] in, represents the pixel-level similarity between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, p i represents the i-th feature block extracted from the complete region, p j represents the jth feature block extracted from the damaged area.
[0131] The pixel-level similarity obtained Apply the softmax operation to calculate the attention score of each feature block, which is defined as follows:
[0132]
[0133] in, represents the attention score between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, and N represents the total number of feature blocks extracted from the damaged area.
[0134] Using the deconvolution operation, the attention scores of the high-level feature maps are used to fill the damaged areas of the low-level feature maps. The process can be described as follows:
[0135]
[0136] in, The feature block As the deconvolution kernel in the attention score matrix The deconvolution operation performed on From the feature map F c The 3×3 feature block extracted from the feature map F c The total number of feature blocks extracted from For all attention scores The attention score matrix is composed of is the learned weighted affinity matrix, i.e., the complete feature map I rec .
[0137] In order to obtain multi-scale semantic features, four groups of dilated convolutional layers with different dilation rates are used to convert the complete feature map I rec Further transformed into a multi-scale semantic feature map I ret , the specific process is as follows:
[0138]
[0139] Among them, g r=k (·)(k∈{1,2,4,8}) represents four dilated convolutions with dilation rates of 1, 2, 4, and 8 respectively, and ⊕ represents the corresponding element addition operation.
[0140] Finally, the feature map F c and multi-scale semantic feature maps I ret Combined to generate output feature I out :
[0141] I out =μ(g(Concat(I ret ,F c )))
[0142] Among them, μ(·) is the activation function, g(·) is the mapping function implemented by the convolutional layer, and Concat(·) is the channel concatenation operation.
[0143] Finally, the generated feature I out and residual connection F c Perform channel concatenation and convolution operations to achieve feature fusion The specific operation process is as follows:
[0144]
[0145] According to the above operation process, the multi-level cascade deformable convolution decoder can decode the deep features of the decoder And the same-layer encoder features X1, X2, X3, X4, X5 obtained by skip connection are used as input to generate reconstructed repair features k∈{1,2,3,4,5,6}, the specific steps are as follows:
[0146] Step S32: The intermediate feature map As low-resolution input F high , the deep feature map X5 is used as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the second adjacent layer conduction attention module to obtain the intermediate feature map output by the second adjacent layer conduction attention module
[0147] Step S33: The intermediate feature map As low-resolution input F high , taking the deep feature map X4 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the third adjacent layer conduction attention module to obtain the intermediate feature map output by the third adjacent layer conduction attention module
[0148] Step S34: The intermediate feature map As low-resolution input F high , taking the deep feature map X3 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fourth adjacent layer conduction attention module to obtain the intermediate feature map output by the fourth adjacent layer conduction attention module
[0149] Step S35: The intermediate feature map As low-resolution input F high , the deep feature map X2 is used as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fifth adjacent layer conduction attention module to obtain the intermediate feature map output by the fifth adjacent layer conduction attention module
[0150] Step S36: The intermediate feature map As low-resolution input F high , taking the deep feature map X1 as the high-resolution input F low , input low resolution into F high and high resolution input F lowSend it to the sixth adjacent layer conduction attention module to obtain the intermediate feature map output by the sixth adjacent layer conduction attention module
[0151] After getting the complete intermediate feature map After that, the intermediate feature map Input output module, the output module outputs the repaired image F out , that is, the complete RGB image is finally restored. Among them, the output module performs the intermediate feature map Perform a double upsampling operation and a convolution operation with 3 output channels and 3 convolution kernels.
[0152] Step S4: In the process of training the network model, it is necessary to design a loss function corresponding to image restoration to guide the overall network to perform reverse conduction, and then learn the corresponding optimization parameters; the classic loss functions in the current image restoration field are all for network output results. Considering the specific network structure of the method of the present invention, and the fact that this type of U-shaped network decoding process is prone to feature distortion and difficulty in convergence, this method also designs a cascade loss function to constrain the decoder intermediate features. k∈{1,2,3,4,5,6}, thereby achieving better network parameter optimization guidance.
[0153] The present invention performs a repair on the damaged sample image F in The downsampling operation is performed to obtain the size of the intermediate feature map Corresponding feature map of the sample to be repaired The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k , and calculate the residual reconstruction features CL k The error between the real image and the damaged sample image F is used to obtain the cascade loss. The cascaded deformable convolution encoder and the cascaded deformable convolution decoder are updated based on the cascade loss. The real image is the damaged sample image to be repaired. in The corresponding undamaged image.
[0154] Step S41, cascade loss function as shown in the attached Figure 3 As shown, the output of any layer of the cascade decoder For example, As an input, the input sample feature map to be repaired As another input, is the damaged sample image to be repaired F in The size is changed, and the two data are combined to generate a residual reconstruction feature CL k To achieve the subsequent loss constraint, the formula is as follows:
[0155]
[0156] Among them, ⊙ represents the element-wise multiplication operation. By using the interpolation operation, we can ensure The size and Same; Function It is specifically used to convert feature maps with different channels into feature maps with fixed 3 channels, that is, it is a convolution function plus an activation function. It can be understood as a specific convolution layer for feature mapping, but the output channel of the convolution operation is fixed to 3. The function v(·) is a convolution operation with three layers and kernel sizes of 1, 3, and 1 respectively.
[0157] Step S42: Reconstruct the residual feature CL k , the method of the present invention requires that it and the real image F gt The error loss is calculated between to constrain the reconstruction process in the decoder. Since k∈{1,2,3,4,5,6}, this operation can be performed simultaneously at multiple levels of the decoder to achieve better constraints. The cascade loss L cl The specific formula is as follows:
[0158]
[0159] This embodiment uses a cascaded deformable convolution framework to ensure that the features of adjacent network layers can be efficiently fused and have good reconstruction and restoration performance. Then the corresponding cascade loss is designed to strengthen the constraints on the intermediate features of the decoder, strengthen the guiding role of the restoration task, and achieve better restoration results.
[0160] The conventional losses used in this embodiment include the classic perceptual loss (Perceptual Loss), style loss (Style Loss) and absolute value loss (Absolute Loss or L1 Loss). Perceptual loss calculates the difference between the input image and the target image in the feature space, focusing on the perceptual quality of the image, so that the trained network can generate visually more natural and realistic restoration results. Style loss is calculated by comparing the statistical information (such as Gram matrix) of the input image and the target image on the feature map, thereby ensuring that the restored image is consistent with the target image in style. Absolute value loss evaluates the quality of the restoration result by calculating the sum of the absolute differences in pixel values between the input image and the target image, ensuring that the restored image is as close as possible to the target image at the pixel level.
[0161] This embodiment introduces cascade loss to ensure the integrity of image restoration, and also aims at the restored image F outPerceptual loss, style loss and absolute value loss are imposed as combined constraints, and the hybrid loss is obtained by integrating the cascade loss and the conventional loss. Based on the hybrid loss, the cascaded deformable convolution encoder, cascaded deformable convolution decoder and output module are trained and updated, thereby achieving complete and realistic restoration of the image.
[0162] It should be noted that perceptual loss, style loss and absolute value loss are conventional losses in image restoration. This embodiment does not describe the calculation process in detail, and in specific implementation, the loss type of conventional loss can also be replaced according to the required target effect.
[0163] Step S5, in the inference phase, the damaged image to be repaired is processed based on the cascaded deformable convolution encoder, cascaded deformable convolution decoder and output module that have been updated through training to obtain a repaired image.
[0164] It should be noted that the training phase and the inference phase are regular phases of the neural network module, but this does not mean that the two phases must be executed at the same time, that is, the process of this application may be to execute only the training phase, or only the inference phase (usually the inference phase is executed separately after the training phase).
[0165] The present invention utilizes cascaded deformable convolutional coding to realize the offset extraction of deep features; performs adjacent layer conduction attention calculation to integrate structural texture details for multi-level features; cascaded deformable convolutional decoding gradually expands the feature resolution to the size of the original image to obtain a complete repaired image; cascaded loss performs deep supervision on the intermediate features of all decoder levels to guide the rapid convergence of repair and reconstruction.
[0166] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0167] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.
Claims
1. An image restoration method based on cascaded deformable convolution, characterized in that: The image restoration method based on cascaded deformable convolution comprises: Step S1: Obtain the damaged sample image F to be repaired in ; Step S2: The damaged sample image F to be repaired in Input a cascaded deformable convolution encoder to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7, wherein the cascaded deformable convolution encoder includes five convolution downsampling layers with a step size of 2 and two layers of deformable convolution blocks; Step S3: Input the deep feature maps X1, X2, X3, X4, X5, X6 and X7 into the cascaded deformable convolution decoder to obtain multi-size intermediate feature maps. k∈{1,2,3,4,5,6}, the cascaded deformable convolutional decoder includes six adjacent layer conduction attention modules; Step S4: the damaged sample image F to be repaired in The downsampling operation is performed to obtain the size of the intermediate feature map Corresponding feature map of the sample to be repaired The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k , and calculate the residual reconstruction features CL k The error between the real image and the damaged sample image F is used to obtain the cascade loss. The cascade deformable convolution encoder and the cascade deformable convolution decoder are trained and updated based on the cascade loss. The real image is the damaged sample image to be repaired. in The corresponding undamaged image; Step S5, in the inference phase, the damaged image to be repaired is processed based on the cascaded deformable convolution encoder and cascaded deformable convolution decoder that have been updated through training to obtain an intermediate feature map For the intermediate feature map A double upsampling operation and a convolution operation with an output channel of 3 and a convolution kernel of 3 are performed to obtain the final repaired image.
2. The image restoration method based on cascaded deformable convolution according to claim 1, characterized in that: The damaged sample image to be repaired F in Input the cascaded deformable convolution encoder to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7, including: First, the damaged sample image to be repaired F in Perform 5 convolution downsampling operations with a step size of 2 and a convolution kernel of 3 to obtain downsampled feature maps X1, X2, X3, X4 and X5 of various sizes as depth feature maps; Then the downsampled feature map X5 is input into the deformable convolution block, where the feature X is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor , and then connect the feature X through residual connection conv , X5 and X defor Splicing is performed in the channel dimension, and finally a convolution with a convolution kernel of 3 is used to obtain the offset result feature map X ret , for the offset result feature map X ret Continue to use the convolution operation with a step size of 2 to obtain the downsampled feature map X6 as the depth feature map; Finally, the downsampled feature map X6 is input into the deformable convolution block, where the feature X is first obtained by a convolution operation with a convolution kernel size of 3 and a step size of 1. conv1 , and then apply a deformable convolution operation with a convolution kernel size of 3 to obtain the offset feature X defor1 , and then connect the feature X through residual connection conv1 , X6 and X defor1 Splicing is performed in the channel dimension, and finally a convolution with a convolution kernel of 3 is used to obtain the offset result feature map X ret1 , for the offset result feature map X ret1 Continue to use the convolution operation with a step size of 2 to obtain the downsampled feature map X7 as the depth feature map.
3. The image restoration method based on cascaded deformable convolution according to claim 1, characterized in that: The depth feature maps X1, X2, X3, X4, X5, X6 and X7 are input into the cascaded deformable convolution decoder to obtain intermediate feature maps of multiple sizes. include: The deep feature map X7 is used as the low-resolution input F high , taking the deep feature map X6 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the first adjacent layer conduction attention module to obtain the intermediate feature map output by the first adjacent layer conduction attention module The intermediate feature map As low-resolution input F high , the deep feature map X5 is used as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the second adjacent layer conduction attention module to obtain the intermediate feature map output by the second adjacent layer conduction attention module The intermediate feature map As low-resolution input F high , taking the deep feature map X4 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the third adjacent layer conduction attention module to obtain the intermediate feature map output by the third adjacent layer conduction attention module The intermediate feature map As low-resolution input F high , taking the deep feature map X3 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fourth adjacent layer conduction attention module to obtain the intermediate feature map output by the fourth adjacent layer conduction attention module The intermediate feature map As low-resolution input F high , the deep feature map X2 is used as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the fifth adjacent layer conduction attention module to obtain the intermediate feature map output by the fifth adjacent layer conduction attention module The intermediate feature map As low-resolution input F high , taking the deep feature map X1 as the high-resolution input F low , input low resolution into F high and high resolution input F low Send it to the sixth adjacent layer conduction attention module to obtain the intermediate feature map output by the sixth adjacent layer conduction attention module 4. The image restoration method based on cascaded deformable convolution according to claim 3, characterized in that: The operation of the adjacent layer conduction attention module is as follows: Low resolution input F high First, the size is enlarged to the same size as the high-resolution input F through interpolation. low Align and get the feature map F r ; Then the high resolution is input to F low And the feature map F r Concatenate in the channel dimension and perform convolution operation to achieve feature fusion to obtain feature map F c , and then the feature map F c and low resolution input F high Send to the multi-scale context fusion block output feature I out ;Finally, the generated feature I out and feature map F c Perform channel splicing and convolution operations to achieve feature fusion and obtain intermediate feature maps 5. The image restoration method based on cascaded deformable convolution according to claim 4, characterized in that: The feature map F c and low resolution input F high Send to the multi-scale context fusion block output feature I out ,include: Enter low resolution into F high Block segmentation is performed to obtain a 3×3 feature block sequence. The formula for calculating pixel attention based on the feature block is: in, represents the pixel-level similarity between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, p i represents the i-th feature block extracted from the complete region, p j represents the jth feature block extracted from the damaged area; The pixel-level similarity obtained Apply the softmax operation to calculate the attention score of each feature block, which is defined as follows: in, represents the attention score between the i-th feature block extracted from the intact area and the j-th feature block extracted from the damaged area, and N represents the total number of feature blocks extracted from the damaged area; Using the deconvolution operation, the attention scores of the high-level feature maps are used to fill the damaged areas of the low-level feature maps. The formula is as follows: in, The feature block As the deconvolution kernel in the attention score matrix The deconvolution operation performed on From the feature map F c The 3×3 feature block extracted from the feature map F c The total number of feature blocks extracted from For all attention scores The attention score matrix is composed of is the learned weighted affinity matrix, i.e., the complete feature map I rec ; Using four sets of dilated convolutional layers with different dilation rates, the complete feature map I rec Further transformed into a multi-scale semantic feature map I ret , the specific process is as follows: Among them, g r=k (·)(k∈{1,2,4,8}) represents four dilated convolutions with dilation rates of 1, 2, 4, and 8 respectively. Indicates the addition operation of corresponding elements; Finally, the feature map F c and multi-scale semantic feature mapping I ret Combined to generate output feature I out : I out =μ(g(Concat(I ret ,F c ))) Among them, μ(·) is the activation function, g(·) is the mapping function implemented by the convolutional layer, and Concat(·) is the channel concatenation operation.
6. The image restoration method based on cascaded deformable convolution according to claim 1, characterized in that: The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k ,include: Among them, CL k represents the residual reconstruction feature of the kth size, Represents the feature map of the sample to be repaired of the kth size, represents the intermediate feature map of the kth size, ⊙ represents the element-wise multiplication operation, are the convolution function and activation function, where the convolution operation output channel of the convolution function is fixed to 3, and v(·) is a convolution operation containing three layers with kernel sizes of 1, 3, and 1 respectively.
7. The image restoration method based on cascaded deformable convolution according to claim 6, characterized in that: The calculation of each residual reconstruction feature CL k The error between the real image and the real image is obtained by cascading the loss, including: Among them, L cl is the cascade loss, F gt For real images.
8. An image restoration method based on cascaded deformable convolution, characterized in that: The image restoration method based on cascaded deformable convolution comprises: Step S1: Obtain the damaged sample image F to be repaired in ; Step S2: The damaged sample image F to be repaired in Input a cascaded deformable convolution encoder to obtain multi-level deep feature maps X1, X2, X3, X4, X5, X6 and X7, wherein the cascaded deformable convolution encoder includes five convolution downsampling layers with a step size of 2 and two layers of deformable convolution blocks; Step S3: Input the deep feature maps X1, X2, X3, X4, X5, X6 and X7 into the cascaded deformable convolution decoder to obtain multi-size intermediate feature maps. k∈{1,2,3,4,5,6}, and the intermediate feature map Input output module, the output module outputs the repaired image F out , the cascaded deformable convolutional decoder includes six adjacent layer conduction attention modules; Step S4: the damaged sample image F to be repaired in The downsampling operation is performed to obtain the size of the intermediate feature map Corresponding feature map of the sample to be repaired The residual reconstruction feature CL is calculated based on the intermediate feature map of the same size and the feature map of the sample to be repaired k , and calculate the residual reconstruction features CL k The error between the real image and the original image is used to obtain the cascade loss. out Calculate the conventional loss, integrate the cascade loss and the conventional loss, train and update the cascade deformable convolution encoder, the cascade deformable convolution decoder and the output module, and the real image is the damaged sample image to be repaired F in The corresponding undamaged image; Step S5, in the inference stage, the damaged image to be repaired is processed based on the cascaded deformable convolution encoder, the cascaded deformable convolution decoder and the output module that have been updated based on the training to obtain a repaired image, and the output module includes a double upsampling operation and a convolution operation with an output channel of 3 and a convolution kernel of 3.
9. The image restoration method based on cascaded deformable convolution according to claim 8, characterized in that: The conventional losses include perceptual loss, style loss and absolute value loss.
Citation Information
Cited By
Fast image restoration method based on stream matching
CN121563840A
A Fast Image Restoration Method Based on Stream Matching
CN121563840B
