Image restoration method, system, device and storage medium

Abstract

The present invention discloses an image restoration method, system, device and storage medium, comprising: inputting an image to be restored into an encoder for feature conversion to obtain a coded feature image, extracting spatial information features of the coded feature image according to a spatial path to obtain a first feature image; being able to obtain more effective feature information, extracting context information features of the coded feature image according to a context path to obtain a second feature image; being able to provide a maximum receptive field with global context information, fusing features of the first feature image and the second feature image to obtain a feature fused image; extracting features of the coded feature image according to a number of residual blocks to obtain a third feature image; connecting the feature fused image with the third feature image to obtain a connected fused feature image; inputting the connected fused feature image into a decoder for decoding to obtain a restored image, thereby improving the quality of the restored image.

Description

Technical Field

[0001] The present invention relates to the technical field related to image restoration, and in particular to an image restoration method, system, device and storage medium. Background Art

[0002] Digital image restoration technology is a technology that uses known information in an image to fill in the missing areas in the image. It belongs to the image restoration problem in the field of computer vision. In the field of physical engineering, experts need to analyze the images produced by experiments, but due to the influence of noise from optical devices and other noise, the objects that should be continuous and uniform in the image are broken and weakened. How to restore the lost information in the image as much as possible is a problem that needs to be solved urgently.

[0003] Traditional image restoration methods are mainly divided into two categories: pixel-based image restoration and block-based image restoration. The basic idea of ​​pixel-based image restoration technology is to start from the boundary of the area to be inpainted, and gradually fill all the pixels in the area to be inpainted from the boundary to the center. The pixels to be filled are obtained by the weighted sum of all known pixels in its neighborhood; the representative algorithm of this type of restoration method is the inpainting algorithm based on the fast marching method; when the inpainting area is small, the restoration effect is better, but when the inpainting area is large, a blurring effect will occur. The basic idea of ​​block-based image restoration technology is to complete the image by searching for a block similar to a small block in the inpainting area to be filled. Its essence is to use the information of the known part of the image to complete the unknown part; the representative algorithm of this type of method is the example-based inpainting algorithm. When the image color is monotonous, the restoration effect is better, and when the color difference in the image is large, the restoration effect is very poor.

[0004] Extracting effective features of background area images is a key step in generating high-quality inpainted images. In image inpainting tasks, currently, some methods try to maintain the resolution of the input image and use dilated convolution to encode sufficient spatial information, while other methods try to capture sufficient receptive fields through pyramid modules or larger convolution kernels. However, both sufficient spatial information and a larger receptive field are crucial to achieving high-precision inpainted images. However, current methods find it difficult to simultaneously meet the requirements of sufficient spatial information and a larger receptive field. Summary of the invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention proposes an image restoration method, system, device and storage medium, which can provide a maximum receptive field with global context information, obtain more effective feature information, and obtain a higher quality restoration image.

[0006] A first aspect of the present invention provides an image restoration method, comprising the following steps:

[0007] Obtain the image to be repaired;

[0008] Inputting the image to be repaired into an encoder for feature conversion to obtain a coded feature image;

[0009] Extracting spatial information features from the coded feature image according to the spatial path to obtain a first feature image;

[0010] Extracting context information features from the encoded feature image according to the context path to obtain a second feature image;

[0011] Performing feature fusion on the first feature image and the second feature image to obtain a feature fusion image;

[0012] Extracting features from the encoded feature image according to a plurality of residual blocks to obtain a third feature image;

[0013] Connecting the feature fusion image with the third feature image to obtain a connected fusion feature image;

[0014] The connected fused feature image is input into a decoder for decoding to obtain a restored image.

[0015] The control method according to the embodiment of the present invention has at least the following beneficial effects:

[0016] The method obtains an image to be repaired, inputs the image to be repaired into an encoder for feature conversion to obtain a coded feature image, extracts spatial information features of the coded feature image according to a spatial path to obtain a first feature image, extracts context information features of the coded feature image according to a context path to obtain a second feature image, performs feature fusion of the first feature image and the second feature image to obtain a feature fusion image; extracts features of the coded feature image according to a plurality of residual blocks to obtain a third feature image, connects the feature fusion image with the third feature image to obtain a connected fused feature image; inputs the connected fused feature image into a decoder for decoding to obtain a repaired image, more effective feature information can be obtained by extracting spatial information features of the coded feature image according to the spatial path, a maximum receptive field with global context information can be provided by extracting context information features of the coded feature image according to the context path, and the quality of the repaired image is improved by connecting the fusion features with the features of a plurality of residual blocks.

[0017] According to some embodiments of the present invention, the spatial path includes three convolutional layers with a step size of 2, and the step of extracting spatial information features from the encoded feature image according to the spatial path to obtain a first feature image includes:

[0018] Performing spatial information feature extraction on the coded feature image according to a first convolutional layer with a step size of 2 to obtain a feature image that is half of the coded feature image;

[0019] Performing spatial information feature extraction on the half of the feature image according to a second convolutional layer with a step size of 2 to obtain a feature image of one quarter of the encoded feature image;

[0020] The spatial information features of the one-quarter feature images are extracted according to the third convolutional layer with a step size of 2 to obtain the first feature image.

[0021] According to some embodiments of the present invention, the context path includes 4 global average poolings, and the extracting context information features from the encoded feature image according to the context path to obtain a second feature image includes:

[0022] Performing context information feature extraction on the encoded feature image according to a first global average pooling to obtain a feature image that is four times the size of the encoded feature image;

[0023] Performing context information feature extraction on the 4-fold feature image according to the second global average pooling to obtain a feature image 8 times the size of the encoded feature image;

[0024] Performing context information feature extraction on the 8-fold feature image according to a third global average pooling to obtain a 16-fold feature image of the encoded feature image;

[0025] Performing context information feature extraction on the 16-fold feature image according to a fourth global average pooling to obtain a feature image 32 times the size of the encoded feature image;

[0026] Performing an attention operation on the 16-fold feature image to obtain a 16-fold feature map after the attention operation; performing an attention operation on the 32-fold feature image to obtain a 32-fold feature map after the attention operation;

[0027] The feature map that is 16 times larger than the attention operation is connected with the feature map that is 32 times larger than the attention operation to obtain the second feature image.

[0028] According to some embodiments of the present invention, performing an attention operation on the 16-fold feature image to obtain a 16-fold feature image after the attention operation includes:

[0029] The 16-fold feature image is subjected to an attention operation through a channel information attention mechanism and a spatial information attention mechanism to obtain a 16-fold feature map after the attention operation; wherein, the calculation formula for obtaining the 16-fold feature map after the attention operation based on the 16-fold feature image being subjected to an attention operation through a channel information attention mechanism and a spatial information attention mechanism is:

[0030]

[0031]

[0032] F∈R C×H×W

[0033] M c ∈R C×1×1

[0034] M s ∈R 1×H×W

[0035] Among them, F is the 16-fold feature image, M c is a one-dimensional channel attention feature map, M s is a two-dimensional spatial attention feature map, F′ is a feature map 16 times larger than the channel attention operation, and F″ is a feature map 16 times larger than the attention operation.

[0036] According to some embodiments of the present invention, performing feature fusion on the first feature image and the second feature image to obtain a feature fused image includes:

[0037] Connecting the first feature image and the second feature image to obtain a connected feature image;

[0038] Calculating feature weights of the connected feature images;

[0039] The connected feature images are subjected to feature fusion according to the feature weights to obtain the feature fusion image.

[0040] According to some embodiments of the present invention, the calculating the feature weight of the connected feature image includes:

[0041] Performing scale balancing on the connected feature image through a batch normalization operation to obtain a balanced connected feature map;

[0042] Performing a pooling operation on the balanced connected feature map to obtain a pooled connected feature map;

[0043] Performing a convolution operation on the pooled connected feature map to obtain a convolutional connected feature map;

[0044] Performing a pooling operation on the convolutional connected feature map to obtain a pooled connected feature map;

[0045] An activation function operation is performed on the balanced connected feature map to obtain a feature weight of the connected feature image.

[0046] According to some embodiments of the present invention, the number of the residual blocks is 8, and the step of connecting the feature fusion image with the third feature image to obtain a connected fusion feature image includes:

[0047] Adding a skip connection between the context path and the spatial path to the eight residual blocks;

[0048] The feature fusion image is connected to the third feature image according to the jump connection to obtain a connected fusion feature image.

[0049] A second aspect of the present invention provides an image restoration system, the image restoration system comprising:

[0050] A data acquisition module, used for acquiring the image to be restored;

[0051] An encoding module, used for inputting the image to be repaired into an encoder for feature conversion to obtain an encoded feature image;

[0052] A first feature image extraction module, used for extracting spatial information features from the coded feature image according to the spatial path to obtain a first feature image;

[0053] A second feature image extraction module, used for extracting context information features from the encoded feature image according to the context path to obtain a second feature image;

[0054] A feature fusion module, used for performing feature fusion on the first feature image and the second feature image to obtain a feature fusion image;

[0055] A third feature image extraction module, used for performing feature extraction on the coded feature image according to a plurality of residual blocks to obtain a third feature image;

[0056] A feature connection module, used for connecting the feature fusion image with the third feature image to obtain a connected fusion feature image;

[0057] The decoding module is used to input the connected fused feature image into a decoder for decoding to obtain a restored image.

[0058] The system obtains an image to be repaired, inputs the image to be repaired into an encoder for feature conversion to obtain a coded feature image, extracts spatial information features of the coded feature image according to a spatial path to obtain a first feature image, extracts contextual information features of the coded feature image according to a context path to obtain a second feature image, fuses the first feature image with the second feature image to obtain a feature fused image; extracts features of the coded feature image according to a number of residual blocks to obtain a third feature image, connects the feature fused image with the third feature image to obtain a connected fused feature image; inputs the connected fused feature image into a decoder for decoding to obtain a repaired image, more effective feature information can be obtained by extracting spatial information features of the coded feature image according to the spatial path, and a maximum receptive field with global context information can be provided by extracting contextual information features of the coded feature image according to the context path, and the quality of the repaired image is improved by connecting the fused features with the features of a number of residual blocks.

[0059] According to a third aspect of the present invention, there is provided an image restoration electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can perform the above-mentioned image restoration method.

[0060] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned image restoration method.

[0061] It should be noted that the beneficial effects between the second to fourth aspects of the present invention and the prior art are the same as the beneficial effects between the above-mentioned image restoration system and the prior art, and will not be described in detail here.

[0062] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0064] Figure 1 is a flow chart of an image restoration method according to an embodiment of the present invention;

[0065] Figure 2 It is a network overall framework diagram of an image restoration method provided by an embodiment of the present invention and other methods;

[0066] Figure 3 It is a detailed information diagram of the spatial path and context path of an image restoration method provided by an embodiment of the present invention and other methods;

[0067] Figure 4 It is a feature fusion module diagram of an image restoration method provided by an embodiment of the present invention and other methods;

[0068] Figure 5 It is a channel- and position-based attention module diagram of an image restoration method provided by an embodiment of the present invention and other methods;

[0069] Figure 6 is a diagram comparing the restoration results of an image restoration method provided by an embodiment of the present invention and other methods;

[0070] Figure 7 It is a structural schematic diagram of an image restoration system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0072] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0073] In the description of the present invention, it should be understood that descriptions involving orientation, such as orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0074] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0075] Traditional image restoration methods are mainly divided into two categories: pixel-based image restoration and block-based image restoration. The basic idea of ​​pixel-based image restoration technology is to start from the boundary of the area to be inpainted, and gradually fill all the pixels in the area to be inpainted from the boundary to the center. The pixels to be filled are obtained by the weighted sum of all known pixels in its neighborhood; the representative algorithm of this type of restoration method is the inpainting algorithm based on the fast marching method; when the inpainting area is small, the restoration effect is better, but when the inpainting area is large, a blurring effect will occur. The basic idea of ​​block-based image restoration technology is to complete the image by searching for a block similar to a small block in the inpainting area to be filled. Its essence is to use the information of the known part of the image to complete the unknown part; the representative algorithm of this type of method is the example-based inpainting algorithm. When the image color is monotonous, the restoration effect is better, and when the color difference in the image is large, the restoration effect is very poor.

[0076] Extracting effective features of background area images is a key step in generating high-quality inpainted images. In image inpainting tasks, currently, some methods try to maintain the resolution of the input image and use dilated convolution to encode sufficient spatial information, while other methods try to capture sufficient receptive fields through pyramid modules or larger convolution kernels. However, both sufficient spatial information and a larger receptive field are crucial to achieving high-precision inpainted images. However, current methods find it difficult to simultaneously meet the requirements of sufficient spatial information and a larger receptive field.

[0077] In order to solve the above technical defects, refer to Figure 1 The present invention provides an image restoration method, comprising:

[0078] Step S101: Acquire an image to be repaired.

[0079] Step S102: input the image to be repaired into an encoder for feature conversion to obtain a coded feature image.

[0080] Step S103: extracting spatial information features from the coded feature image according to the spatial path to obtain a first feature image.

[0081] Step S104: extract context information features from the encoded feature image according to the context path to obtain a second feature image.

[0082] Step S105: perform feature fusion on the first feature image and the second feature image to obtain a feature fusion image.

[0083] Step S106: extract features from the encoded feature image according to the plurality of residual blocks to obtain a third feature image.

[0084] Step S107: connect the feature fusion image and the third feature image to obtain a connected fusion feature image.

[0085] Step S108: input the connected fused feature image into a decoder for decoding to obtain a restored image.

[0086] The method obtains an image to be repaired, inputs the image to be repaired into an encoder for feature conversion to obtain a coded feature image, extracts spatial information features of the coded feature image according to a spatial path to obtain a first feature image, extracts context information features of the coded feature image according to a context path to obtain a second feature image, performs feature fusion of the first feature image and the second feature image to obtain a feature fusion image; extracts features of the coded feature image according to a plurality of residual blocks to obtain a third feature image, connects the feature fusion image with the third feature image to obtain a connected fused feature image; inputs the connected fused feature image into a decoder for decoding to obtain a repaired image, more effective feature information can be obtained by extracting spatial information features of the coded feature image according to the spatial path, a maximum receptive field with global context information can be provided by extracting context information features of the coded feature image according to the context path, and the quality of the repaired image is improved by connecting the fusion features with the features of a plurality of residual blocks.

[0087] Specifically, refer to Figure 2 , where Dualatt represents a dual attention mechanism based on channels and spaces, FFM represents a feature fusion module, and jump connections of context paths and spatial paths are added to the 8 residual blocks of the encoder-decoder network, and a dual attention scheme based on channels and positions is added to the context path of the jump connection. For the spatial path, the present invention only stacks three convolutional layers to obtain 1 / 8 of the feature map, and the obtained 1 / 8 feature map retains rich spatial details. For the context path, the present invention uses a global average pooling layer, in which the receptive field is the maximum value of the backbone network. In the context path, the present invention adds a global average pooling at the tail of the model, which can provide a maximum receptive field with global context information.

[0088] In order to better fuse the features extracted by the spatial path and the context path, the present invention also studies a fusion refinement module of the two paths, namely, a feature fusion module. In order to better extract the contextual information features of the image, the present invention adds an attention refinement module to the context path. The attention module performs weighted processing on the image in the channel dimension and the spatial dimension respectively, so as to obtain more effective feature information. The use of two additional components, the feature fusion module and the dual attention module based on position and channel, can further generate high-quality repaired images.

[0089] In some embodiments, the spatial path includes three convolutional layers with a step size of 2, and spatial information features are extracted from the encoded feature image according to the spatial path to obtain a first feature image, including:

[0090] The spatial information features of the coded feature image are extracted according to the first convolutional layer with a step size of 2 to obtain a feature image that is half of the coded feature image.

[0091] The spatial information features of one-half of the feature image are extracted according to the second convolutional layer with a step size of 2 to obtain a feature image of one-quarter of the encoded feature image.

[0092] The spatial information features of one quarter of the feature images are extracted according to the third convolutional layer with a step size of 2 to obtain the first feature image.

[0093] Specifically, refer to Figure 3 , the present invention proposes a spatial path branch (Spatial Path) to maintain the spatial information of the original input image and encode the rich spatial information. The spatial path contains three layers. Each layer contains a convolution with a stride of 2. The output feature map extracted by this path is 1 / 8 of the original image. Due to the large spatial size of the feature map, it encodes rich spatial information. Figure 3 The detailed information of the spatial path and context path branches is shown. The feature map input to the repair network passes through 3 convolutional layers with a stride of 2 in the spatial path. The size of the feature map becomes 1 / 2 of the original after each convolutional layer. When passing through 3 convolutional layers, the size of the feature map output by this path becomes 1 / 8 of the input image size. This feature map retains the spatial information of the image.

[0094] In some embodiments, the context path includes four global average poolings, and context information features are extracted from the encoded feature image according to the context path to obtain a second feature image, including:

[0095] The context information features are extracted from the encoded feature image according to the first global average pooling to obtain a feature image that is 4 times the size of the encoded feature image.

[0096] The context information feature is extracted from the 4-fold feature image according to the second global average pooling to obtain a feature image that is 8 times the size of the encoded feature image.

[0097] The context information feature is extracted from the 8-fold feature image according to the third global average pooling to obtain a 16-fold feature image of the encoded feature image.

[0098] The context information feature is extracted from the 16-fold feature image according to the fourth global average pooling to obtain a 32-fold feature image of the encoded feature image.

[0099] An attention operation is performed on the 16-fold feature image to obtain a 16-fold feature map after the attention operation; an attention operation is performed on the 32-fold feature image to obtain a 32-fold feature map after the attention operation.

[0100] The feature map that is 16 times larger after the attention operation is concatenated with the feature map that is 32 times larger after the attention operation to obtain the second feature image.

[0101] Specifically, refer to Figure 3 , the context path is designed to provide a sufficient receptive field. In the image restoration task, the size of the receptive field has an important impact on feature extraction. In order to expand the receptive field, some methods use pyramid modules or larger convolution kernels to achieve the purpose of expanding the receptive field. However, these operations are computationally intensive, resulting in a slow network speed. Considering both a large receptive field and efficient calculation, the present invention proposes a context path. The context path performs downsampling through pooling to provide a larger receptive field. Figure 3 As shown, in the context path, through 4 layers of global average pooling, the present invention obtains feature maps that are downsampled by 4 times, 8 times, 16 times, and 32 times, respectively, and connects the feature map that is 16 times after the attention operation with the feature map that is 32 times after the attention operation to obtain a second feature image.

[0102] In some embodiments, performing an attention operation on the 16-fold feature image to obtain a 16-fold feature image after the attention operation includes:

[0103] The 16-fold feature image is subjected to an attention operation through the channel information attention mechanism and the spatial information attention mechanism to obtain a 16-fold feature map after the attention operation; wherein, the calculation formula for obtaining the 16-fold feature map after the attention operation based on the 16-fold feature image is:

[0104]

[0105]

[0106] F∈R C×H×W

[0107] M c ∈R C×1×1

[0108] M s ∈R 1×H×W

[0109] Among them, F is the 16-fold feature image, M c is a one-dimensional channel attention feature map, M sis a two-dimensional spatial attention feature map, F ′ is the feature map 16 times after the channel attention operation, and F″ is the feature map 16 times after the attention operation.

[0110] Specifically, refer to Figure 3 and Figure 5 , where Channel Attention represents channel attention, SpatialAttention represents spatial attention, Input Feature represents input feature, and Refined Feature represents feature after attention operation. In order to better extract context information with more image features, the present invention adds a dual attention mechanism (Dualatt) based on channel and space after 16 times and 32 times downsampling, so that the context path can provide the maximum receptive field with global context information.

[0111] In some embodiments, performing feature fusion on the first feature image and the second feature image to obtain a feature fusion image includes:

[0112] The first feature image is connected with the second feature image to obtain a connected feature image.

[0113] Calculate the feature weights of the concatenated feature image.

[0114] The connected feature images are fused according to the feature weights to obtain a feature fused image.

[0115] In some embodiments, calculating the feature weights of the connected feature images includes:

[0116] The scale of the concatenated feature image is balanced through a batch normalization operation to obtain a balanced concatenated feature map.

[0117] The balanced connected feature map is pooled to obtain a pooled connected feature map.

[0118] The pooled connected feature map is convolved to obtain the convolved connected feature map.

[0119] The convolutional connected feature map is pooled to obtain a pooled connected feature map.

[0120] The balanced connected feature map is subjected to activation function operation to obtain the feature weight of the connected feature image.

[0121] Specifically, in some embodiments, the features extracted by the spatial path and the context path are different in the feature representation level. Therefore, the present invention cannot simply summarize these features. The spatial information captured by the spatial path encodes most of the rich detail information, and the output features of the context path mainly encode the context information. In other words, the output features of the spatial path are low-level, while the output features of the context path are high-level. Therefore, the present invention proposes a specific feature fusion module to fuse these features.

[0122] Reference Figure 4 , the present invention first connects the output features of the spatial path and the context path, then uses batch normalization to balance the scale of the features, merges the connected features into the feature vector, and calculates the weight vector. The weight vector can re-weight the features, which is equivalent to feature selection and combination.

[0123] Figure 4 The detailed information of the feature fusion module is shown. Concatenate means that the output feature maps of the spatial path and the context path are first connected, and the batch normalization operation (BN) is used to balance the scale of the obtained feature map. Next, the connected features in the previous step are calculated through pooling (global pool) operation, convolution operation (conv) operation and activation function (relu) operation. The weight vector can re-weight the features, which is equivalent to the selection and combination of features.

[0124] In some embodiments, the number of residual blocks is 8, and the feature fusion image is connected with the third feature image to obtain a connected fusion feature image, including:

[0125] Skip connections between context paths and spatial paths are added to the 8 residual blocks.

[0126] The feature fusion image is connected to the third feature image according to the jump connection to obtain a connected fusion feature image.

[0127] Reference Figure 2 In order to facilitate understanding by those skilled in the art, a set of optimal embodiments are provided below:

[0128] 1. Image Coding:

[0129] Get the image to be repaired.

[0130] The image to be repaired is input into the encoder for feature conversion to obtain a coded feature image.

[0131] 2. Feature Fusion

[0132] The spatial information features of the coded feature image are extracted according to the first convolutional layer with a step size of 2 to obtain a feature image that is half of the coded feature image.

[0133] The spatial information features of one-half of the feature image are extracted according to the second convolutional layer with a step size of 2 to obtain a feature image of one-quarter of the encoded feature image.

[0134] The spatial information features of one quarter of the feature images are extracted according to the third convolutional layer with a step size of 2 to obtain the first feature image.

[0135] The context information features are extracted from the encoded feature image according to the first global average pooling to obtain a feature image that is 4 times the size of the encoded feature image.

[0136] The context information feature is extracted from the 4-fold feature image according to the second global average pooling to obtain a feature image that is 8 times the size of the encoded feature image.

[0137] The context information feature is extracted from the 8-fold feature image according to the third global average pooling to obtain a 16-fold feature image of the encoded feature image.

[0138] The context information feature is extracted from the 16-fold feature image according to the fourth global average pooling to obtain a 32-fold feature image of the encoded feature image.

[0139] Attention operation is performed on the 16-fold feature image through the channel information attention mechanism and the spatial information attention mechanism to obtain a 16-fold feature map after the attention operation; attention operation is performed on the 32-fold feature image through the channel information attention mechanism and the spatial information attention mechanism to obtain a 32-fold feature map after the attention operation.

[0140] The feature map that is 16 times larger after the attention operation is concatenated with the feature map that is 32 times larger after the attention operation to obtain the second feature image.

[0141] The first feature image is connected with the second feature image to obtain a connected feature image.

[0142] The balanced connected feature map is pooled to obtain a pooled connected feature map.

[0143] The pooled connected feature map is convolved to obtain the convolved connected feature map.

[0144] The convolutional connected feature map is pooled to obtain a pooled connected feature map.

[0145] The balanced connected feature map is subjected to activation function operation to obtain the feature weight of the connected feature image.

[0146] The connected feature images are fused according to the feature weights to obtain a feature fused image.

[0147] 3. Residual block feature extraction:

[0148] The number of residual blocks is 8. Feature extraction is performed on the encoded feature image according to a number of residual blocks to obtain a third feature image.

[0149] 4. Image decoding:

[0150] Skip connections between context paths and spatial paths are added to the 8 residual blocks.

[0151] The feature fusion image is connected to the third feature image according to the jump connection to obtain a connected fusion feature image.

[0152] The concatenated fused feature image is input into the decoder for decoding to obtain the restored image.

[0153] In order to facilitate understanding by those skilled in the art, a set of experimental data is provided below:

[0154] The present invention is evaluated based on Places2, CelebA and Paris Street View datasets. The irregular mask dataset contains 12,000 irregular masks, and the masked area accounts for 0-60% of the total image size. In addition, the irregular dataset is divided into six intervals of 0-10%, 10-20%, 20-30%, 40-50%, 30-40%, and 50-60% according to the masked area. There are 2,000 irregular masks in each interval.

[0155] Dataset introduction:

[0156] Places2 Challenge Dataset: A dataset released by MIT containing more than 8 million images from 365 scenes, which is very suitable for building inpainting models because it enables the model to learn distributions from many natural scenes.

[0157] CelebA dataset: A dataset focused on face images, containing more than 180,000 training images. Models trained on this dataset can be easily converted to face editing / completion tasks.

[0158] Paris Street Scene Dataset: A dataset containing 14,900 training images and 100 test images from Paris street scenes. This dataset mainly focuses on buildings in the city.

[0159] The present invention is compared with several state-of-the-art methods. These models are trained until convergence, and the experimental settings are the same as the present invention. These models are: PIC, PConv, GatedConv, EdgeConnect and PRVS.

[0160] Experiments were conducted on these three datasets and qualitative and quantitative results were measured while comparing the proposed method with previous methods. Then, the bare network (without attention module) was compared with the existing backbone network in terms of model size and quantitative performance to demonstrate the efficiency of the proposed method.

[0161] Reference Figure 6 , Figure 6 The present invention (ours) is compared with 4 state-of-the-art methods on the dataset. In most cases, the restoration results of ours have much fewer obvious inconsistencies than the state-of-the-art methods, especially for large holes. Compared with other methods, the algorithm proposed in ours produces more semantically reasonable and elegant results.

[0162] The present invention quantitatively compares the models based on the structural similarity index (SSIM), peak signal-to-noise ratio (PSNR) and mean l1 (MeanL1) loss. Tables 1, 2, and 3 list the results of the quantitative indicators on different proportions of irregular occlusions in the three datasets. As shown in Tables 1, 2, and 3, the present invention produces excellent results on the Places2, CelebA, and Paris StreetView datasets.

[0163] Table 1 shows the data comparison of different methods on the Places2 dataset. * indicates that the higher the better, and + indicates that the lower the better.

[0164] Table 1

[0165]

[0166] Table 2 shows the data comparison of different methods on the CelebA dataset.

[0167] Table 2

[0168]

[0169] Table 3 shows the data comparison of different methods on the Paris Street View dataset.

[0170] Table 3

[0171]

[0172] In addition, refer to Figure 7 One embodiment of the present invention provides an image restoration system, including a data acquisition module 1100, an encoding module 1200, a first feature image extraction module 1300, a second feature image extraction module 1400, a feature fusion module 1500, a third feature image extraction module 1600, a feature connection module 1700 and a decoding module 1800, wherein:

[0173] The data acquisition module 1100 is used to acquire the image to be restored.

[0174] The encoding module 1200 is used to input the image to be repaired into the encoder for feature conversion to obtain an encoded feature image.

[0175] The first feature image extraction module 1300 is used to extract spatial information features from the coded feature image according to the spatial path to obtain a first feature image.

[0176] The second feature image extraction module 1400 is used to extract context information features from the encoded feature image according to the context path to obtain a second feature image.

[0177] The feature fusion module 1500 is used to perform feature fusion on the first feature image and the second feature image to obtain a feature fusion image.

[0178] The third feature image extraction module 1600 is used to extract features from the encoded feature image according to a number of residual blocks to obtain a third feature image.

[0179] The feature connection module 1700 is used to connect the feature fusion image with the third feature image to obtain a connected fusion feature image.

[0180] The decoding module 1800 is used to input the connected fused feature image into the decoder for decoding to obtain a restored image.

[0181] The system obtains an image to be repaired, inputs the image to be repaired into an encoder for feature conversion to obtain a coded feature image, extracts spatial information features of the coded feature image according to a spatial path to obtain a first feature image, extracts contextual information features of the coded feature image according to a context path to obtain a second feature image, fuses the first feature image with the second feature image to obtain a feature fused image; extracts features of the coded feature image according to a number of residual blocks to obtain a third feature image, connects the feature fused image with the third feature image to obtain a connected fused feature image; inputs the connected fused feature image into a decoder for decoding to obtain a repaired image, more effective feature information can be obtained by extracting spatial information features of the coded feature image according to the spatial path, and a maximum receptive field with global context information can be provided by extracting contextual information features of the coded feature image according to the context path, and the quality of the repaired image is improved by connecting the fused features with the features of a number of residual blocks.

[0182] It should be noted that the present system embodiment and the above-mentioned system embodiment are based on the same inventive concept, so the relevant contents of the above-mentioned method embodiment are also applicable to the present system embodiment and will not be repeated here.

[0183] The present application also provides an image restoration electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image restoration method as described above is implemented.

[0184] The processor and the memory may be connected via a bus or other means.

[0185] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The non-transient software program and instructions required to implement the image restoration method of the above embodiment are stored in the memory. When executed by the processor, the image restoration method of the above embodiment is executed, for example, the above described Figure 1 The method comprises steps S101 to S108.

[0187] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute: the image restoration method as described above.

[0188] The computer-readable storage medium stores computer-executable instructions, which are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the processor can execute the image restoration method in the above embodiment, for example, execute the above described Figure 1 The method comprises steps S101 to S108.

[0189] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program units or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer readable instructions, data structures, program elements, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0190] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. An image restoration method, characterized in that: The image restoration method comprises: Obtain the image to be repaired; Inputting the image to be repaired into an encoder for feature conversion to obtain a coded feature image; Extracting spatial information features from the encoded feature image according to the spatial path to obtain a first feature image; the spatial path includes three convolutional layers with a step size of 2, and the spatial information feature map extracted by the spatial path is ultimately 1 / 8 of the original image; According to the context path, context information features are extracted from the encoded feature image to obtain a second feature image; the context path includes 4 global average poolings, and feature images of 4 times, 8 times, 16 times, and 32 times are obtained through the 4 global average poolings respectively, and an attention operation is performed on the 16-fold feature image to obtain a 16-fold feature map after the attention operation; an attention operation is performed on the 32-fold feature image to obtain a 32-fold feature map after the attention operation; the 16-fold feature map after the attention operation is connected with the 32-fold feature map after the attention operation to obtain the second feature image; wherein the attention operation is specifically performed through a channel information attention mechanism and a spatial information attention mechanism; Performing feature fusion on the first feature image and the second feature image to obtain a feature fusion image; Extracting features from the encoded feature image according to a plurality of residual blocks to obtain a third feature image; the number of the residual blocks is 8; Connecting the feature fusion image with the third feature image to obtain a connected fusion feature image; specifically comprising: adding a jump connection of the context path and the spatial path to the 8 residual blocks; connecting the feature fusion image with the third feature image according to the jump connection to obtain a connected fusion feature image; The connected fused feature image is input into a decoder for decoding to obtain a restored image.

2. The image restoration method according to claim 1, characterized in that: The step of extracting spatial information features from the coded feature image according to the spatial path to obtain a first feature image includes: Performing spatial information feature extraction on the coded feature image according to a first convolutional layer with a step size of 2 to obtain a feature image that is half of the coded feature image; Performing spatial information feature extraction on the half of the feature image according to a second convolutional layer with a step size of 2 to obtain a feature image of one quarter of the encoded feature image; The spatial information features of the one-quarter feature images are extracted according to the third convolutional layer with a step size of 2 to obtain the first feature image.

3. An image restoration method according to claim 2, characterized in that: The extracting context information features from the coded feature image according to the context path also includes: Performing context information feature extraction on the encoded feature image according to a first global average pooling to obtain a feature image that is four times the size of the encoded feature image; Performing context information feature extraction on the 4-fold feature image according to the second global average pooling to obtain a feature image 8 times the size of the encoded feature image; Performing context information feature extraction on the 8-fold feature image according to a third global average pooling to obtain a 16-fold feature image of the encoded feature image; Context information features are extracted from the 16-fold feature image according to the fourth global average pooling to obtain a 32-fold feature image of the encoded feature image.

4. The image restoration method according to claim 3, characterized in that: The step of performing an attention operation on the 16-fold feature image to obtain a 16-fold feature image after the attention operation comprises: The 16-fold feature image is subjected to an attention operation through a channel information attention mechanism and a spatial information attention mechanism to obtain a 16-fold feature map after the attention operation; wherein, the calculation formula for obtaining the 16-fold feature map after the attention operation based on the 16-fold feature image being subjected to an attention operation through a channel information attention mechanism and a spatial information attention mechanism is: F∈R C×H×W M c ∈R C×1×1 M s ∈R 1×H×W Among them, F is the 16-fold feature image, M c is a one-dimensional channel attention feature map, M s is a two-dimensional spatial attention feature map, F' is a feature map 16 times larger than the channel attention operation, and F'' is a feature map 16 times larger than the attention operation.

5. The image restoration method according to claim 4, characterized in that: The step of fusing the first feature image with the second feature image to obtain a feature fused image includes: Connecting the first feature image and the second feature image to obtain a connected feature image; Calculating feature weights of the connected feature images; The connected feature images are subjected to feature fusion according to the feature weights to obtain the feature fusion image.

6. The image restoration method according to claim 5, characterized in that: The calculating the feature weight of the connected feature image comprises: Performing scale balancing on the connected feature image through a batch normalization operation to obtain a balanced connected feature map; Performing a pooling operation on the balanced connected feature map to obtain a pooled connected feature map; Performing a convolution operation on the pooled connected feature map to obtain a convolutional connected feature map; Performing a pooling operation on the convolutional connected feature map to obtain a pooled connected feature map; An activation function operation is performed on the balanced connected feature map to obtain a feature weight of the connected feature image.

7. An image restoration system, characterized in that: The image restoration system: A data acquisition module, used for acquiring the image to be restored; An encoding module, used for inputting the image to be repaired into an encoder for feature conversion to obtain an encoded feature image; A first feature image extraction module is used to extract spatial information features from the coded feature image according to a spatial path to obtain a first feature image; the spatial path includes three convolution layers with a step size of 2, and the spatial information feature map extracted by the spatial path is ultimately 1 / 8 of the original image; A second feature image extraction module is used to extract context information features from the encoded feature image according to the context path to obtain a second feature image; the context path includes 4 global average poolings, through which 4 global average poolings are used to obtain feature images of 4 times, 8 times, 16 times, and 32 times, respectively, and an attention operation is performed on the 16-fold feature image to obtain a feature map of 16 times after the attention operation; an attention operation is performed on the 32-fold feature image to obtain a feature map of 32 times after the attention operation; the feature map of 16 times after the attention operation is connected with the feature map of 32 times after the attention operation to obtain the second feature image; wherein the attention operation is specifically performed through a channel information attention mechanism and a spatial information attention mechanism; A feature fusion module, used for performing feature fusion on the first feature image and the second feature image to obtain a feature fusion image; A third feature image extraction module, used for performing feature extraction on the coded feature image according to a plurality of residual blocks to obtain a third feature image; the number of the residual blocks is 8; A feature connection module, used to connect the feature fusion image with the third feature image to obtain a connected fusion feature image; specifically comprising: adding a jump connection of the context path and the spatial path to the 8 residual blocks; connecting the feature fusion image with the third feature image according to the jump connection to obtain a connected fusion feature image; The decoding module is used to input the connected fused feature image into a decoder for decoding to obtain a restored image.

8. An image restoration device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute an image repair method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute an image restoration method as described in any one of claims 1 to 6.

Images