Image inpainting method and system based on parallel multi-scale multi-stage fusion

Abstract

The application discloses an image repairing method and system based on parallel multi-scale multi-stage fusion, extracts image features according to an image to be repaired and a corresponding mask image, obtains four sub-image features of different scales after down-sampling the image features, controls the repairing speed of a missing area in each scale of the sub-image features by using a mask image-based repairing priority method, fuses the four repaired sub-images, and then performs down-sampling and repairing processing multiple times, and finally obtains a final repairing image by using a self-attention-guided feature fusion method. The application can maintain high-resolution feature repairing at all times, and can better fuse structure and texture information and obtain a more natural and reasonable repairing result by using a multi-stage multi-scale fusion technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image inpainting method and system based on parallel multi-scale multi-stage fusion. BACKGROUND

[0002] The existing image inpainting technology mostly uses background information to fill in the missing area, and can inpaint simple low-resolution images, but it is difficult to process high-resolution missing images. In addition, these methods perform poorly on images with complex scenes or large missing areas. In recent years, deep learning has greatly improved image inpainting technology. Previous image inpainting methods using deep learning are all based on autoencoder networks. In this network structure, the resolution of image features is first sampled down layer by layer from high to low, and then sampled up layer by layer from low to high. In the down-sampling process, the spatial details of the image are often lost, leading to degradation of the inpainting result. In the autoencoder structure, the high-level structure information and the low-level content texture information cannot be well fused, so that the inpainting result lacks semanticity, resulting in that the existing methods still cannot well process high-resolution missing images, and the inpainting result is not strong in semanticity. SUMMARY

[0003] The present application proposes an image inpainting method and system based on parallel multi-scale multi-stage fusion to solve the above problems in the prior art, which can maintain high-resolution features for inpainting at all times, and use multi-stage multi-scale fusion technology to better fuse structure and texture information and obtain more natural and reasonable inpainting results.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to an image inpainting method based on parallel multi-scale multi-stage fusion. After extracting image features from the image to be inpainted and its corresponding mask image, four sub-image features of different scales are obtained by down-sampling the image features. The repair priority method based on the mask image is used to control the repair speed of the missing area in each scale of the sub-image features. After the four repaired sub-images are fused and down-sampled again, the final inpainting image is obtained by using the feature fusion based on self-attention guidance after multiple repair processing.

[0006] The application relates to a system for implementing the above method, comprising an image feature processing unit and a multi-scale feature fusion unit composed of four parallel branches, wherein: the image feature processing unit performs feature extraction processing on an input picture to be repaired and a corresponding mask picture, obtains a multi-channel feature map of an original size, and performs 1, 2, 4 and 8 times down-sampling on the multi-channel feature map and the mask picture respectively to obtain feature maps and mask pictures of different resolution sizes; each branch of the multi-scale feature fusion unit receives feature maps and mask pictures of different resolutions respectively, controls the repair speed of a missing area of the feature map in each branch by using a repair priority method based on the mask picture, performs feature fusion of four scales after each sub-processing stage, fuses feature maps of different resolutions based on a multi-scale attention mechanism, and finally outputs a result with the largest resolution in the four-scale repaired pictures as the final output.

[0007] The repair speed refers to that a branch processing the feature map and the mask picture with the smallest resolution is a low-resolution branch, and other branches are high-resolution branches, the low-resolution branch focuses on the priority repair of structures, and the high-resolution branch focuses on the priority repair of textures.

[0008] Technical effects

[0009] The application uses a high-resolution and low-resolution repair priority method based on a mask picture to control the image repair speed, focuses on the repair of textures and results respectively, realizes more reasonable repair, and fuses multi-scale features by using a multi-scale attention fusion mechanism. Compared with the prior art, the application obtains good effects under the conditions of four image repair indexes, high-resolution images and large-area missing. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 It is a flowchart of the application;

[0011] Figure 2 It is a mask effect diagram of the embodiment;

[0012] Figure 3 It is an image repair neural network diagram of the embodiment;

[0013] Figure 4 It is a repair diagram of the embodiment.

[0014] Figure 5 It is an effect diagram of the embodiment. DETAILED DESCRIPTION

[0015] As Figure 1As shown, the embodiment relates to an image inpainting method based on parallel multi-scale multi-stage fusion. After extracting a multi-channel feature map from an image to be inpainted and its corresponding mask map, the multi-channel feature map and the mask map are down-sampled by 1, 2, 4 and 8 times to obtain four feature maps and mask maps of different resolutions. Four parallel branches process feature maps of different resolutions respectively. The repair speed of the missing area in the feature map in each branch is controlled by using a mask map-based repair priority method. Then, the feature maps of the four branches are subjected to multi-scale feature fusion operation for multiple times. Finally, a final inpainting image is obtained by using a self-attention-guided feature fusion method. Specifically, the method comprises the following steps:

[0016] Step 101) obtaining an image to be inpainted and a corresponding mask map, wherein the mask map is used to indicate the position of the area to be inpainted. Specifically, an image segmentation algorithm is used to obtain the mask map.

[0017] As shown in Figure 2 , the image segmentation algorithm uses, but is not limited to, a mask map provided in an image segmentation data set, a randomly generated irregular line map and a rectangular region mask map, or a modified position recorded by a user when editing an image, to obtain the mask map.

[0018] The training process of the embodiment uses the first two ways to obtain the mask map. In the test process, the mask map of the last two ways is used. The mask Figure 1 map generally has two numerical values of 0 and 1, wherein 0 is the area to be inpainted, and 1 is the area that does not need to be inpainted.

[0019] Step 102) extracting image features of the first stage by using an inpainting network, wherein the size of the image features is consistent with that of the input image.

[0020] As shown in Figure 3 , the inpainting network comprises a six-layer convolutional network, and each square in the figure is a feature map output by one layer of convolution.

[0021] Step 103) down-sampling the feature map of the first stage to obtain four sub-feature maps of scales in proportion to each other, wherein the ratio relationship between the size of the first feature map and the size of the second feature map is consistent with the ratio relationship between the size of the second feature map and the size of the third feature map, and so on.

[0022] For example, the size of the feature map obtained in the first stage is 256*256. Then, the sizes of the four scale feature maps obtained by the down-sampling stage are 256*256, 128*128, 64*64 and 32*32 respectively.

[0023] Step 104) inputting the four scale feature maps obtained in step 103 into four parallel sub-branches respectively, and controlling the repair speed of the missing area in the feature map in each scale branch by using a mask map-based repair priority method.

[0024] As shown in Figure 3 each parallel sub-branch is composed of multiple sub-stages, each sub-stage includes four convolutional layers and a multi-scale feature fusion layer, and the last feature fusion stage uses a feature fusion method based on a self-attention mechanism.

[0025] The sub-image repair speed, i.e., the image repair speed controlled by the priority relationship, includes repair under a high-resolution feature map and repair under a low-resolution feature map, and specifically includes:

[0026] a) updating the mask map at the corresponding position of each pixel point in the feature map: wherein m' is the value of the position on the mask map corresponding to the current first-stage pixel point, m is the value on the mask map at the corresponding position of the previous stage, and p is a priority value, which has a value when the pixel point is located at the edge position of the missing area in the mask map, otherwise the value of p is 0. A pixel point located at the edge position of the missing area needs to satisfy m = 0 and 0 ≤ sum(M p ) ≤ Ω p , M p is the number of 1s contained in a convolution kernel in the mask map, Ω p is the product of the length and width of the convolution kernel, μ is a hyperparameter, p max is the maximum value of the p values of all boundary pixel points, p = sum(M p )·h l (x), l ∈ {3, 2, 1, 0} is the resolution level, corresponding to {256 2 , 128 2 , 64 2 , 32 2} resolution feature maps, respectively, sum(M p ) is a common priority degree, used to calculate the number of effective pixels in the current convolution window, and the more the number of effective pixels around a point, the higher the priority degree of the repair, h l (x) is the priority degree of different scales.

[0027] In this embodiment, l ∈ {3, 2, 1} is a high-resolution scale, and l ∈ {0} is a low-resolution scale. For the low-resolution scale l ∈ {0}, wherein n p is the normal vector of a point on the boundary of the missing area on the mask map, is the missing boundary isophote direction at the corresponding position on the feature map, i.e., a vector perpendicular to the gradient direction; for the high-resolution scale l ∈ {3, 2, 1}, wherein X p↑↓The feature map is downsampled and then upsampled back to the original image size. This process results in the loss of some high-frequency information. (The original feature map X is then used for this purpose.) p Subtract X p↑↓ This allows us to obtain this high-frequency information. Calculating the gradient on this high-frequency information map is to remove structural information from the image, thus enabling the network to focus on texture restoration at high-resolution scales.

[0028] b) Perform convolution calculations controlled by a mask-based inpainting priority method: Where: x p Let W be a pixel on the feature map, W be the convolution kernel, b be the bias term, and X be a pixel. p For x p Convolutional sliding window on the feature map at the location, M p is the convolutional sliding window on the mask image, and m′ is the corresponding position on the updated mask image.

[0029] Step 105) Perform multi-scale fusion, i.e., the fused feature map at the current scale includes:

[0030] a) For feature maps with the same scale as the current scale, simply add them together;

[0031] b) For feature maps with a scale smaller than the current scale, upsample them to the same scale before adding them together;

[0032] c) For feature maps with a scale larger than the current scale, perform a 2D convolution operation with a kernel of 3 to obtain feature maps with the same scale as the current scale, and then add them together.

[0033] Step 106) Figure 3 As shown by the dashed arrow, feature fusion based on a self-attention mechanism is performed, specifically including:

[0034] i) At the smallest resolution scale (32 2 Calculate attention score on ) s ij =f(x) i ) T g(x j ), where: x is the feature map obtained after stacking, x i x j Let f(x) be a point on the feature map, N be the total number of pixels on the feature map, and f(x) = W. f x, g(x) = W g x is the convolution operation on x, and the resulting attention score a ij The weights are used by the model to reference the pixel value at position j on the feature map when synthesizing pixel i.

[0035] ii) Calculate the output y after fusing the pixels at position i. i= γ o i + x i wherein: γ is a hyper-parameter to balance the importance between o i and x i .

[0036] iii) the attention score map a is used to guide the global feature fusion at each scale: wherein: l e {3, 2, 1}, a i，j is the (i, j)th position in the attention score map a, the ith feature block is synthesized by the jth feature block at the resolution l scale. For example: for different resolution scale feature maps (256 2 , 128 2 , 64 2 , 32 2 ), each corresponding feature block size is resolved as 8 2 , 4 2 , 2 2 , 1 2 .

[0037] The attention score map a e R N×N is obtained by using, but not limited to, the self-attention mechanism proposed in Self-Attention Generative Adversarial Networks.

[0038] Step 107) the fused image is subjected to a layer of convolution to obtain the final repair result.

[0039] Through specific actual experiments, under the specific environment setting of Linux version 4.15 and Pytorch 1.5, on a single NVIDIA RTX TITAN 24G graphics card, with a learning rate of 0.0001, an input image size of 256, β1 = 0.001, and a β2 = 0.99 adam optimizer to train the above model, and test the method on the same single graphics card. 10000 pictures are randomly selected on the Places2 dataset to test the effect of the method and compare it with the best image repair algorithms at present. Irregular mask graphs are used to simulate missing areas, a total of six groups of missing mask graphs, the missing proportions are: 0-10%, 10%-20%, 20%-30%, 30%-40%, 40%-50%, 50-60%. Four evaluation indexes are used: relative L1, structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and Frechet perceptual distance (FID). The experimental data obtained are shown in Table 1.

[0040]

[0041] Table 1

[0042] Compared with the prior art, as shown in Table 1, the method achieves the best results in all indicators, especially in the experiment of large area missing (Mask = 50%-60%), the results of the method are much better than other methods, which proves that the method has special advantages under the condition of large area missing. As shown in Figure 4 , the repair effect under different missing areas is shown, the image size is 256*256, and the first two columns are the repair effect under large area missing, it can be seen that the method can obtain relatively ideal results under the condition of large area missing. As shown in Figure 5 , the repair effect of high resolution image (1024*1024) is shown, it can be seen that the method also has advantages in the repair of high resolution image.

[0043] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation scheme within the scope is subject to the present application.

Claims

1. An image inpainting method based on parallel multi-scale multi-stage fusion, characterized in that, After extracting image features from the image to be repaired and its corresponding mask, the image features are downsampled to obtain four sub-image features at different scales. A mask-based repair priority method is used to control the repair speed of missing regions in the sub-image features at each scale. The four repaired sub-images are then fused and downsampled and repaired multiple times. Finally, a self-attention-guided feature fusion method is used to obtain the final repaired image, which includes: Step 101) Obtain the image to be repaired and the corresponding mask. The mask is used to indicate the location of the area to be repaired. Specifically, the mask is obtained by using an image segmentation algorithm. Step 102) Use the repair network to extract the image features of the first stage, the size of which is consistent with the input image; Step 103) Downsample the feature map of the first stage to obtain four sub-feature maps with proportionally decreasing scales. Specifically, the ratio of the size of the first feature map to the size of the second feature map is consistent with the ratio of the size of the second feature map to the size of the third feature map, and so on. Step 104) Input the four scale feature maps obtained in step 103 into the four parallel sub-branches respectively, and use the mask map-based repair priority method to control the repair speed of the missing feature map regions in each scale branch. Step 105) Perform multi-scale fusion, i.e., the fused feature map at the current scale includes: a) For feature maps with the same scale as the current scale, simply add them together; b) For feature maps with a scale smaller than the current scale, upsample them to the same scale before adding them together; c) For feature maps with a scale larger than the current scale, perform a 2D convolution operation with a kernel of 3 to obtain feature maps with the same scale as the current scale, and then add them together. Step 106) Perform feature fusion based on the self-attention mechanism, specifically including: i) At the smallest resolution scale Calculate attention score ,in: The feature map obtained after stacking. , Let N be a point on the feature map, and N be the total number of pixels on the feature map. , The attention score obtained from the convolution operation on x. The weights of the pixel value at position j on the feature map are used by the model when synthesizing pixel i. ii) Calculate the output result after pixel fusion at position i. ,in: , These are hyperparameters used for balancing. and The proportion between them; iii) Use the attention score map a to guide the fusion of global features across various scales: ,in: , It is an attention score graph The i-th feature block at position (i, j) in the middle From the j-th feature block At resolution Scale synthesis is obtained; for feature maps of different resolution scales (256) 2 128 2 64 2 32 2 Each of its corresponding feature blocks has a resolution of 8. 2 4 2 ,2 2 1 2 ; Step 107) The fused image is processed through a convolution layer to obtain the final repair result.

2. The image inpainting method based on parallel multi-scale multi-stage fusion according to claim 1, characterized in that, The image segmentation algorithm described herein obtains a mask image by using a mask image provided in the image segmentation dataset, a mask image of a randomly generated irregular line image and a mask image of a rectangular region, or by recording the location of the modification when the user edits the image. The training process uses the first two methods to obtain the mask image, while the testing process uses the mask images obtained by the latter two methods.

3. The image inpainting method based on parallel multi-scale multi-stage fusion according to claim 1, characterized in that, The repair network includes a six-layer convolutional network, where each square in the diagram is a feature map output after one convolutional layer.

4. The image inpainting method based on parallel multi-scale multi-stage fusion according to claim 1, characterized in that, Each parallel sub-branch consists of multiple sub-stages, each of which includes four convolutional layers and a multi-scale feature fusion layer. The final feature fusion stage uses a feature fusion method based on a self-attention mechanism.

5. The image inpainting method based on parallel multi-scale multi-stage fusion according to claim 1, characterized in that, The repair speed of missing regions in the sub-image features, i.e., the image repair speed is controlled by priority relationships, includes: repair under high-resolution feature maps and repair under low-resolution feature maps, specifically: a) For each pixel in the feature map, update the mask image at its corresponding location: ,in: `m` represents the value of the position on the mask corresponding to the current pixel in the first stage, `m` represents the value on the mask corresponding to the previous position, and `p` is the priority value. A pixel only has a value if it is located at the edge of a missing region in the mask; otherwise, `p` is 0. For a pixel to be located at the edge of a missing region, `m` must be 0 and... , The number of 1s contained in a convolution kernel in the mask image. It is the product of the length and width of the convolution kernel. For hyperparameters, The maximum value of p among all pixels on the boundary. , These are resolution levels, corresponding to { Feature maps at} resolution The priority level is used to calculate the number of valid pixels in the current convolutional window. The more valid pixels around a point, the higher its repair priority. To prioritize different scales, For high resolution scale, For low-resolution scales; , ,in: It is the normal vector of a point on the boundary of the missing region in the mask image. It is the direction of the isoluminance line at the corresponding location on the feature map, i.e., a vector perpendicular to the gradient direction; for high-resolution scales , ,in: The feature map is downsampled and then upsampled back to the original image size. This process results in the loss of some high-frequency information. minus This allows us to obtain this high-frequency information. Calculating the gradient on this high-frequency information map is to remove structural information from the image, thus enabling the network to focus on texture restoration at high-resolution scales. b) Perform convolution calculations controlled by a mask-based inpainting priority method: ,in: For a pixel on the feature map, For convolution kernel, For bias terms, for Convolutional sliding window on the feature map at the location, For the convolution sliding window on the mask image, The corresponding position on the updated mask map.

6. A system for implementing the image inpainting method based on parallel multi-scale multi-stage fusion as described in any one of claims 1-5, characterized in that, include: The image feature processing unit and the multi-scale feature fusion unit consisting of four parallel branches are provided. The image feature processing unit performs feature extraction processing on the input image to be repaired and the corresponding mask image to obtain a multi-channel feature map of the original size. The multi-channel feature map and the mask image are downsampled by 1, 2, 4 and 8 times respectively to obtain feature maps and mask images of different resolution sizes. Each branch of the multi-scale feature fusion unit receives feature maps and mask maps of different resolutions. The repair priority method based on the mask map is used to control the repair speed of the missing regions of the feature maps in each branch. After each sub-processing stage, feature fusion of four scales is performed, and feature maps of different resolutions are fused based on a multi-scale attention mechanism. Finally, the result with the highest resolution among the repaired images of the four scales is output as the final output.

7. The system according to claim 6, characterized in that, The repair speed refers to the following: the branch that processes the feature map and mask map with the lowest resolution is the low-resolution branch, and the other branches are the high-resolution branches. For the low-resolution branches, the focus is on the priority repair of the structure; for the high-resolution branches, the focus is on the priority repair of the texture.

Images