An image restoration method based on dynamic matching of multiple receptive fields and damaged features

Abstract

The invention discloses an image restoration method based on dynamic matching of multiple receptive fields and damaged features. The invention comprises the following steps: using a damaged image as an input, constructing a unified scale multi-receptive field feature extraction module, and extracting a unified scale feature group with different receptive fields; using a damaged image mask as a constraint, establishing a linear mapping relationship through a damaged feature mapping module, and extracting a one-dimensional weight feature corresponding to the current damaged feature; using the weight feature to weight the feature group, and obtaining a weighted unified scale multi-receptive field feature group; using a feature screening module, establishing a dynamic matching relationship between damaged features and features, and obtaining a re-screening feature with a high matching degree; using the re-screening feature as an input, decoding is performed through a decoder group based on fast Fourier convolution, and obtaining a restored image; through the setting, the constraints of the restoration network are strengthened, the artifacts appearing in the repaired image are effectively removed, and the semantic consistency of the restored image is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image restoration, and in particular to an image restoration method based on dynamic matching of multiple receptive fields and damaged features. Background Art

[0002] Image restoration is one of the key basic tasks in the field of computer vision. It is a technology that uses the information of the intact areas in the damaged image to infer and supplement the content of the missing areas. In recent years, with the development of deep learning technology, image restoration technology based on deep learning has been widely used in the fields of old photo restoration, image editing, and cultural relic image restoration.

[0003] At present, image restoration methods are mainly divided into partial differential equation-based methods, block matching-based methods and deep learning-based image restoration methods. The partial differential equation-based image restoration propagates the gradient information of the undamaged area to the area to be repaired in a direction orthogonal to the gradient by diffusion. The block matching-based method searches for blocks similar to the damaged area in the undamaged area, and then copies them to the damaged area for filling. These methods have achieved good restoration effects when repairing images with large similarities between the damaged area and the undamaged area (such as white walls with high consistency). However, due to the lack of flexibility of these methods, they cannot achieve good restoration effects when facing large continuous areas of damage or when there is less information available for reference in the undamaged area. The deep learning-based image restoration method uses artificial intelligence algorithms to learn diverse patterns from a large amount of data to repair images. Although it has achieved good restoration effects, since the adaptability of multiple receptive fields and image damage features is not considered in the network design, there are still problems of artifacts and confusion of semantic information during restoration. Summary of the invention

[0004] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides an image restoration method based on dynamic matching of multiple receptive fields and damaged features, which aims to establish a dynamic matching relationship between the receptive field and the damaged features, and solve the problems of artifacts and semantic information confusion caused by the lack of adaptation of the receptive field and the damaged features of the image during the restoration process.

[0005] To achieve the above object, the technical solution adopted by the present invention is: to provide an image restoration method based on dynamic matching of multiple receptive fields and damaged features, the steps of which are:

[0006] Step 1, taking the damaged image as input and constructing a unified scale multi-receptive field feature extraction module based on the damaged image, and extracting a unified scale feature group with different receptive fields;

[0007] Step 2, using the damaged image mask as a constraint, a linear mapping relationship is established through a damaged feature mapping module composed of a convolutional layer and a pixel-level feature map, and a one-dimensional weight feature corresponding to the damaged feature of the damaged area of ​​the current damaged image mask is extracted;

[0008] Step 3, using the weight feature of step 2 to weight the feature group obtained in step 1, to obtain a weighted unified scale multi-receptive field feature group;

[0009] Step 4: Taking the one-dimensional weighted features obtained in step 2 and the uniform-scale multi-receptive field feature group obtained in step 3 as input, a feature screening module consisting of feature point multiplication and channel convolution is used to establish a dynamic matching relationship between the damaged features and the features, and obtain a re-screening feature with a high matching degree;

[0010] Step 5: Using the re-screened features as input, the decoder group based on fast Fourier convolution is used to decode and obtain the repaired image.

[0011] As a preferred implementation of the present invention: the step 1 constructs a sub-coding feature extraction network composed of multiple filter groups, and the unified scale multi-receptive field feature group extracted from the damaged image through the network is specifically:

[0012] ;

[0013] in, is the input damaged image, is a unified scale multi-receptive field feature group, i is the i-th filter group and its extracted features, Filter banks with different receptive field feature extraction capabilities.

[0014] As a preferred implementation of the present invention: the step 2 establishes a learnable one-dimensional linear score mapping relationship of the damage feature through multi-channel convolution and pixel-level feature mapping, specifically:

[0015] ;

[0016] in, is the damaged image mask, is a learnable one-dimensional weight feature, PLM is a pixel-level feature map, Encoder module for corrupted image masks.

[0017] As a preferred implementation of the present invention: Step 3 performs pixel-level channel-by-channel dot multiplication of the one-dimensional weighted features extracted in step 2 and the feature group extracted in step 1 to obtain a weighted multi-receptive field feature group, specifically:

[0018] ;

[0019] in, is the one-dimensional weight feature extracted in step 2, is a multi-scale receptive field feature group, To obtain the weighted multi-receptive field feature group, Represents a pixel-by-pixel dot multiplication operation.

[0020] As a preferred embodiment of the present invention: Step 4 selects high-frequency effective features from the weighted multi-receptive field feature group to obtain a mixed receptive field feature group suitable for the current damage feature after screening. , specifically:

[0021] ;

[0022] in, represents the normalization operation, represents a leaky random linear rectified unit, i is the i-th filter bank and its extracted features.

[0023] As a preferred embodiment of the present invention: the step 5 is composed of three groups of filters with gradually decreasing channel numbers and three groups of Fourier convolution modules in series, and jump connections are added between the filters, wherein the encoder output As the input features of the decoder, the decoder consists of three upsampling groups and an output head, where the output of a single upsampling group It can be expressed as:

[0024] ;

[0025] In the above formula, is the activation function, represents the deconvolution operation, For normalization operation, represents fast Fourier convolution;

[0026] by is the input of the next upsampling group. After three upsampling group operations, the final As input, the final repaired image is obtained by reorganization through the output head ,

[0027] ;

[0028] in, is the nonlinear hyperbolic tangent activation function.

[0029] As a preferred embodiment of the present invention:

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention takes the dynamic matching relationship between multiple receptive fields and image damage features as the starting point, uses the mask image as the constraint condition, and utilizes the learnable latent feature mapping of the mask and the extraction and re-screening mechanism of the multi-receptive field feature group to establish a dynamic link relationship between the receptive field and the image damage features, thereby strengthening the constraints of the repair network. Compared with the current excellent methods, the present invention effectively removes the artifacts that appear in the repaired image, while enhancing the semantic consistency of the repaired image. Moreover, on three public data sets, the average objective index of the present invention has reached the best. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of an image restoration method based on dynamic matching of multiple receptive fields and damaged features of the present invention;

[0033] Figure 2 This is a comparison of the results of this method and different methods on the Places2 dataset;

[0034] Figure 3 This is a comparison of the results of this method and different methods on the Paris StreetView dataset. DETAILED DESCRIPTION

[0035] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0036] The present invention aims at repairing damaged images and proposes an image repair method based on dynamic matching of multiple receptive fields and damaged features; the proposed image repair method has better artifact removal ability and semantic consistency. An image repair method based on dynamic matching of multiple receptive fields and damaged features comprises the following steps:

[0037] Step 1 uses the input damaged image to construct a unified scale multi-receptive field feature extraction module, and extracts a unified scale feature group with different receptive fields;

[0038] Step 2: Using the damaged image mask as a constraint, a linear mapping relationship is established through the damaged feature mapping module to extract the one-dimensional weighted feature corresponding to the current damaged feature;

[0039] Step 3 uses the weight features of step 2 to weight the feature group obtained in step 1 to obtain a weighted unified scale multi-receptive field feature group;

[0040] Step 4: Through the feature screening module, a dynamic matching relationship between the damage feature and the features is established to obtain a re-screening feature with a high matching degree;

[0041] In step 5, the re-screened features are used as input and decoded by a decoder group based on fast Fourier convolution to obtain a repaired image.

[0042] Example 1

[0043] like Figure 1 As shown, an image restoration method based on dynamic matching of multiple receptive fields and damaged features is provided, comprising the following steps:

[0044] Step 1: Take the damaged image as input and construct a unified scale multi-receptive field feature extraction module based on the damaged image, and extract a unified scale feature group with different receptive fields. Specifically, a sub-coding feature extraction network composed of multiple filter groups is constructed, and a unified scale multi-receptive field feature group extracted from the damaged image is defined by the network. For the damaged image input, filter groups with different receptive field feature extraction capabilities are used for feature extraction. The formula is as follows:

[0045] (1)

[0046] in the formula The input damaged image and mask are extracted by different sub-encoders with a unified scale feature set with different receptive fields. Filter banks with different receptive field feature extraction capabilities.

[0047] Step 2: Taking the damaged image mask as a constraint, a linear mapping relationship is established through a damaged feature mapping module composed of a convolutional layer and a pixel-level feature map to extract the one-dimensional weighted features corresponding to the damaged features of the damaged area of ​​the current damaged image mask. Pixel level Mapping (PLM) will damage the image mask Mapped into learnable one-dimensional weight features , the specific formula is as follows:

[0048] (2)

[0049] Encoder module for damaged image mask;

[0050] Step 3: Group the features of multiple receptive fields of uniform scale and the damage mask map weight feature At the same time as input, through the weight feature For multi-scale receptive field feature group The features in are weighted to obtain the weighted multi-receptive field feature set The formula is as follows:

[0051] (3)

[0052] in, is the one-dimensional weight feature extracted in step 2, is a multi-scale receptive field feature group, To obtain the weighted multi-receptive field feature group, the formula Represents a pixel-by-pixel dot multiplication operation.

[0053] Step 4: Taking the one-dimensional weighted features obtained in step 2 and the uniform-scale multi-receptive field feature group obtained in step 3 as input, a feature screening module consisting of feature point multiplication and channel convolution is used to establish a dynamic matching relationship between the damaged features and features, and obtain a re-screening feature with a high matching degree. Specifically, Operation pair Screen and select the high-frequency effective features, and obtain the mixed receptive field feature group suitable for the current damaged features after screening:

[0054] (4)

[0055] in the formula represents the normalization operation, represents a leaky random linear rectified unit, i is the i-th filter bank and its extracted features.

[0056] Step 5: Using the re-screened features as input, the decoder group based on fast Fourier convolution is used to decode and obtain the repaired image; specifically, it is composed of three groups of filters with gradually decreasing channel numbers and three groups of Fourier convolution modules connected in series, and jump connections are added between the filters, wherein the encoder output As the input features of the decoder, the decoder consists of three upsampling groups and an output head, where the output of a single upsampling group It can be expressed as:

[0057] (5)

[0058] In the above formula, is the activation function, represents the deconvolution operation, For normalization operation, Represents fast Fourier convolution.

[0059] Next, is the input of the next upsampling group. After three upsampling group operations, the final As input, the final repaired image is obtained by reorganization through the output head .

[0060] (6)

[0061] in the formula is the nonlinear hyperbolic tangent activation function.

[0062] As shown in Table 1, this is the experimental results of the present invention on the current general image restoration datasets Paris StreetView, CelebA-HQ and Places2. The method proposed in the present invention has significant advantages over the current mainstream image restoration methods CA, EC, MEDFE, MADF, TFill and W-Net.

[0063] Table 1 Experimental results of Paris StreetView dataset

[0064]

[0065] PSNR stands for Peak Signal-to-Noise Ratio, and the larger the value, the better; SSIM stands for Structural Similarity, and the larger the value, the better. Both of the above indicators are image restoration evaluation indicators. In the table, - means that the method does not provide a pre-trained model for testing under the corresponding data set.

[0066] Furthermore, the restoration results of the proposed method are compared with those of the current mainstream image restoration methods CA, EC, MEDFE, MADF, TFill and W-Net. Figure 2-Figure 3 As shown in the figure, it can be seen that the method proposed in the invention has better artifact removal ability and better semantic consistency than CA, MADF, TFill and W-Net.

[0067] This embodiment extracts uniform-scale multi-receptive field features and uses damage masks to enhance network constraints, thereby establishing a dynamic matching relationship between damage features and receptive field features, and further uses a decoder with fast Fourier convolution for decoding to obtain a repaired image, which can effectively remove artifacts in the repaired image and enhance the semantic consistency of the repaired image.

[0068] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image restoration method based on dynamic matching of multiple receptive fields and damaged features, characterized in that: The steps are: Step 1, taking the damaged image as input and constructing a unified scale multi-receptive field feature extraction module based on the damaged image, and extracting a unified scale feature group with different receptive fields; Step 2, using the damaged image mask as a constraint, a linear mapping relationship is established through a damaged feature mapping module composed of a convolutional layer and a pixel-level feature map, and a one-dimensional weight feature corresponding to the damaged feature of the damaged area of ​​the current damaged image mask is extracted; Step 3, using the weight feature of step 2 to weight the feature group obtained in step 1, to obtain a weighted unified scale multi-receptive field feature group; Step 4: Taking the one-dimensional weighted features obtained in step 2 and the uniform-scale multi-receptive field feature group obtained in step 3 as input, a feature screening module consisting of feature point multiplication and channel convolution is used to establish a dynamic matching relationship between the damaged features and the features, and obtain a re-screening feature with a high matching degree; Step 5: Using the re-screened features as input, the decoder group based on fast Fourier convolution is used to decode and obtain the repaired image.

2. The image restoration method based on dynamic matching of multiple receptive fields and damaged features according to claim 1 is characterized in that: The step 1 constructs a sub-coding feature extraction network composed of multiple filter groups, through which the uniform scale multi-receptive field feature group extracted from the damaged image is extracted. , specifically: ； in, is the input damaged image, is a unified scale multi-receptive field feature group, i is the i-th filter group and its extracted features, Filter banks with different receptive field feature extraction capabilities.

3. The image restoration method based on dynamic matching of multiple receptive fields and damaged features according to claim 2 is characterized in that: The step 2 establishes a learnable one-dimensional linear score mapping relationship of the damage feature through multi-channel convolution and pixel-level feature mapping, specifically: ； in, is the damaged image mask, is a learnable one-dimensional weight feature, is the pixel-level feature map, Encoder module for corrupted image masks.

4. The image restoration method based on dynamic matching of multiple receptive fields and damaged features according to claim 3 is characterized in that: The step 3 performs pixel-level channel-by-channel dot multiplication on the one-dimensional weighted features extracted in step 2 and the feature group extracted in step 1 to obtain a weighted multi-receptive field feature group, specifically: ； in, is the one-dimensional weight feature extracted in step 2, Indicates that the features are concatenated according to the channel dimension. is a multi-scale receptive field feature group, To obtain the weighted multi-receptive field feature group, Represents a pixel-by-pixel dot multiplication operation.

5. The image restoration method based on dynamic matching of multiple receptive fields and damaged features according to claim 4 is characterized in that: The step 4 selects high-frequency effective features from the weighted multi-receptive field feature group to obtain a mixed receptive field feature group suitable for the current damage feature after screening, specifically: ； in, represents the normalization operation, represents a leaky random linear rectifier unit, It indicates that a one-dimensional filter is used to perform filtering operation in the channel dimension, and i is the i-th filter group and its extracted features.

6. The image restoration method based on dynamic matching of multiple receptive fields and damaged features according to claim 5 is characterized in that: The step 5 is composed of three groups of filters with gradually decreasing channel numbers and three groups of Fourier convolution modules connected in series, and jump connections are added between the filters, wherein the encoder output As the input features of the decoder, the decoder consists of three upsampling groups and an output head, where the output of a single upsampling group It can be expressed as: ; In the above formula, is the activation function, represents the deconvolution operation, For normalization operation, represents fast Fourier convolution; by is the input of the next upsampling group. After three upsampling group operations, the final As input, the final repaired image is obtained by reorganization through the output head , ； in, is the nonlinear hyperbolic tangent activation function.

Images