A unified method of combining image inpainting and extrapolation

Abstract

The application provides a unified method combining image inpainting and extrapolation, and belongs to the field of deep learning. The application adopts a joint CNN and a transformer to construct a deep neural network, establishes a unified processing method for image inpainting and extrapolation, decomposes the solving process of the image inpainting and extrapolation problem into three parts of "representation, prediction and synthesis", and connects the three parts to form a unified method for solving the image inpainting and extrapolation problem. Finally, an adversarial learning strategy is introduced to improve the performance of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, convolutional neural network (CNN), self-attention transform network (Transformer), and adversarial learning, and particularly relates to a unified method of combining image inpainting and outpainting. BACKGROUND

[0002] Image inpainting and outpainting problems are caused by incomplete image information due to environmental obstruction and device field of view limitation during data collection, and incomplete image content caused by image damage, human smearing, and added obstruction during transmission. These problems seriously affect image understanding and visual effects. High-quality inpainting and outpainting results can greatly reduce the cognitive difficulty of images and also help to provide a complete and rich data basis for subsequent visual tasks such as image understanding. Image inpainting refers to supplementing the missing areas in the image according to the visual rationality principle based on the neighborhood information of the missing part and the overall information of the known area. Image outpainting refers to extending the observation field of view by extending the known area of the image to infer and draw the content outside the boundary. Image inpainting and outpainting methods are widely used in computer vision tasks, such as field of view expansion, texture synthesis, image editing, object removal, etc.

[0003] Recently, deep learning based methods have been developed rapidly and become the mainstream method to solve the problems of image inpainting and image extrapolation due to their superior performance. Most of these image inpainting and image extrapolation methods directly input sub-images into CNN to synthesize the whole image by learning the knowledge of the dataset. For image inpainting, the unknown area is usually small in size and located inside the image. Due to the strong correlation between adjacent pixels in the image, the missing pixels in the unknown area can be completed by convolving the pixels around the unknown area. This idea and method can achieve good performance in image inpainting tasks. However, for image extrapolation, the unknown area is usually large in size and located outside the known area, which has the characteristics of less available information and large synthesis area, making the image extrapolation task tend to be a more difficult image generation problem. Therefore, current image inpainting and image extrapolation methods are difficult to be mutually applicable, and image inpainting methods are difficult to complete the drawing of a large range of unknown areas around the periphery, while image extrapolation methods are also difficult to handle various different morphological known areas. Compared with using CNN to predict the content of the unknown area, Transformer has more advantages due to its strong ability to model long-distance relationships. Transformer was first applied to the field of natural language processing and has entered the field of computer vision in the past two years as a new type of network structure. Today, Transformer has been proven to be a basic network structure that can replace CNN in many computer vision problems, such as image recognition, object detection, and low-level image processing problems. Unlike CNN, Transformer does not have local correlation and spatial invariance induction bias, and has a strong ability to capture long-distance context relationships. However, Transformer mostly adopts an autoregressive-based approach in prediction problems. Although this approach can solve the problem of predicting unknown pixels from known pixels, in the case of a large number of missing pixels in image extrapolation problems, the pixel-by-pixel prediction method will inevitably consume a lot of time and computing resources. SUMMARY

[0004] To solve the above technical problems, the present application provides a unified method for combining image inpainting and image extrapolation. Using an autoencoder structure, image inpainting and image extrapolation problems can be solved more efficiently.

[0005] The technical scheme of the present application is:

[0006] A unified method for combining image inpainting and image extrapolation, combining CNN and Transformer deep neural network models, can complete various types of image inpainting and extrapolation scenarios, including image inpainting in regular and irregular cases, and four-side, double-side and single-side extrapolation scenarios.

[0007] The system design separates the "representation, prediction, and synthesis" parts. In this separate system design, the CNN encoder and decoder respectively implement the representation and synthesis parts, and the Transformer encoder completes the parallel feature prediction function, which together form an end-to-end model.

[0008] In the end-to-end model, partial convolution, pixel normalization, and dimension transformation techniques are used to construct the network structure; multiple loss functions are superimposed for model training, including adversarial loss, reconstruction loss, and perceptual loss, and the Adam optimizer is used during training.

[0009] Furthermore,

[0010] After each partial convolution, a pixel normalization method is used to balance the feature values ​​at different locations, so that all features fed into the prediction stage have the same data distribution to facilitate more thorough model training.

[0011] The model is constructed using several residual blocks and upsampling intervals. The images are converted into pixel values ​​using a tangent function and then output as composite images. Finally, a discriminant network is introduced to improve the model's performance.

[0012] Furthermore,

[0013] The method involves reducing the image dimension to a feature space, and then using a Transformer to predict features in unknown regions.

[0014] When using Transformer to handle image interpolation and extrapolation problems, it is necessary to represent the two-dimensional data as a one-dimensional sequence data for input, and each sequence element is called a token.

[0015] To address the problems of image inpainting and extrapolation, the original input, which covers the location of unknown regions, is arranged sequentially according to pixel order, thus converting it into one-dimensional sequence data. In this case, each token in the sequence consists of one pixel, and pixels in unknown regions can be filled with special values ​​for labeling.

[0016] The beneficial effects of this invention are

[0017] This invention utilizes open-source libraries related to deep learning. It can be applied to image processing problems such as image inpainting, achieving high-quality restoration and prediction results, and can help improve the perception capabilities of machine vision methods. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall architecture and workflow of the present invention;

[0019] Figure 2 This is a schematic diagram of the parallel feature prediction structure of the present invention;

[0020] Figure 3Here is an example of the image interpolation comparison evaluation results;

[0021] Figure 4 This is an example of the image extrapolation comparison evaluation results. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] This invention provides a unified method for joint image inpainting and extrapolation. The process of solving the image inpainting and extrapolation problem is explicitly decomposed into three stages: representation, prediction, and synthesis. Based on these three stages, a unified method of CNN representation synthesis and joint Transformer iterative prediction is proposed. Specifically, the representation stage employs a partially convolutional CNN encoder to better handle edge information of known regions; the prediction stage designs a Transformer encoder, which, in conjunction with coordinate information, predicts features of unknown regions based on known region features; and the synthesis stage uses a CNN decoder constructed from residual blocks and upsampling overlaps to finally map the predicted features to the complete image.

[0024] Image inpainting and extrapolation can be decomposed into three stages: representation, prediction, and synthesis. Based on this, this invention constructs a unified framework for image inpainting and extrapolation. The representation and synthesis stages are implemented by a CNN encoder and a CNN decoder, respectively, while the prediction stage is completed by a Transformer encoder. This constructs a network architecture that combines a Transformer and a CNN autoencoder. The overall architecture of the unified framework is as follows: Figure 1 As shown. Representation and synthesis can be viewed as mapping processes from image space to feature space and from feature space to image space, respectively. Specifically, the CNN encoder reduces the dimensionality of the input known region image to the feature space, where the feature size is a fraction of the input image. Size. To avoid introducing invalid information from unknown regions, partial convolution is used instead of regular convolution to handle the boundary information between known and unknown regions. Furthermore, after each partial convolution layer, pixel normalization is used to balance feature values ​​at different locations, ensuring all features fed into the prediction stage have the same data distribution for more thorough model training. The feature-to-image mapping process uses a CNN decoder composed of multiple residual blocks and upsampling intervals. These blocks are converted into pixel values ​​using a tangent function to output a synthesized image. Finally, a discriminant network is introduced to improve model performance.

[0025] The prediction phase is the core of the entire model. This method uses a Transformer encoder, whose main function is to predict unknown regions in parallel based on all known region features. Since the Transformer design is suitable for processing one-dimensional sequence data, the one-dimensional data entering and leaving the Transformer encoder needs to be transformed with the two-dimensional features. Considering that the dimensionality processed by the Transformer should not be too large, the features are reduced from 512 to 256 dimensions through a 1×1 convolution before entering the prediction process.

[0026] When using Transformer to handle image inpainting and extrapolation problems, the two-dimensional data needs to be represented as a one-dimensional sequence of input, with each element called a token. For image inpainting and extrapolation, the original input, covering unknown regions, can be arranged pixel-wise, thus converting it into a one-dimensional sequence. In this sequence, each token consists of one pixel, and pixels in unknown regions can be labeled with special values. However, converting the image pixel-wise into a one-dimensional sequence results in an excessively long sequence for the Transformer. For example, a 256×256 image stretched into a one-dimensional sequence would be 65536 pixels long. Since the computational complexity of Transformer is the square of the number of tokens, it requires massive computing power and makes it difficult for the model to extract correlation features between pixels. Therefore, to avoid an excessively long one-dimensional sequence after conversion, the original image is typically reduced to a feature space or the image is split into image patches. Both essentially convert a region of the image into a token. The difference is that, compared to directly splitting the original image into multiple equal-sized image patches, the method of reducing to a feature space allows for further extraction of image features. In the problem of image inpainting and extrapolation, the model usually needs to first analyze the contents contained in the known region. Therefore, in this invention, the image is reduced to a feature space, and then the Transformer is used to predict the features of the unknown region.

[0027] Figure 2The diagram shows the structure of the Transformer for feature prediction in image inpainting and extrapolation tasks. On the left are the two-dimensional features of the input and output. Unknown regions in the input features are represented by blank spaces (white squares). After the Transformer predicts the features, these blank spaces are filled in to obtain the output. Before entering the Transformer encoder, the input two-dimensional sequence features are expanded from left to right and top to bottom, transforming them into a one-dimensional sequence. Each element in the sequence is a token, and the token's corresponding position encoding is added before being fed into the Transformer encoder for computation. The Transformer encoder employs an N-layer self-attention network, using an additional mask to label the unknown regions, ensuring that the self-attention layers only use known region features to predict unknown regions, preventing the introduction of invalid information from unknown regions.

[0028] Let I represent the input two-dimensional feature and 0 represent the output two-dimensional feature, then this prediction process can be represented as:

[0029]

[0030] in, A function that represents the conversion of two-dimensional data into one-dimensional data; I represents the function that converts one-dimensional data into two-dimensional data; E represents the position coordinates, which are usually set in advance according to the number of features; M is a two-dimensional mask downsampled to the same size as I, where 0 is used to mark unknown positions and 1 is used to mark known positions; TRE represents the Transformer encoder.

[0031] This approach allows for the simultaneous prediction of features in all unknown regions, with each unknown region's location feature derived from the combined prediction of all known region features. As the network depth increases, the relationship between known and unknown regions is explored more thoroughly, contributing to improved feature prediction accuracy. In this process, location encoding compensates for the lost location information when converting two-dimensional features to a one-dimensional sequence, playing an indispensable role in helping the network extract two-dimensional spatial relationships from the features.

[0032] The entire deep learning model is trained using a combination of three loss functions: adversarial loss, reconstruction loss, and perceptual loss. The adversarial loss is optimized using an additional discriminant network D, which judges the difference between the output W and the ground truth G. The adversarial loss function is:

[0033] L adv (F, D) = E G [log(D(G))]+E W [log(1-D(W))]

[0034] In the formula, E represents the expectation, F is the model representation function, which is constrained to minimize this loss during training, while D is constrained to maximize this loss during training; the two are adversarial in training. The reconstruction loss is achieved by minimizing the L1 distance between W and G, and its formula is:

[0035] L L1 =E[||WG||]1

[0036] Perceptual loss is used to penalize the perceptual difference between the output image and the real image. It is calculated through the distance difference between the activation layers of a pre-trained classification network, using the following formula:

[0037]

[0038] In the formula, N u σ is the total number of elements in the u-th layer of the network. u This is the activation map of layer u. In practice, a pre-trained VGG-19 classification network is used to obtain the activation maps of each layer.

[0039] The implementation of this invention includes model training, data selection and processing, system testing and performance evaluation.

[0040] During model training, the model structure and training parameters must be specified. In implementation, the model consists of four deep networks: a CNN encoder, a decoder, a Transformer encoder, and a discriminator network. The CNN encoder comprises three partial convolutional layers with a stride of 2, each followed by a pixel normalization layer, and finally reduced to 256 dimensions using a 1×1 convolution. During prediction, the Transformer encoder uses a default configuration of 4 heads and 6 layers. The CNN decoder consists of five residual blocks and three alternating upsampled blocks, followed by a tangent function layer outputting to the image space. Adversarial loss is calculated using a discriminator network, specifically implemented using a multi-scale discriminator network. The CNN encoder, Transformer encoder, CNN decoder, and discriminator network are all trained using the Adam optimizer, with all networks trained at a learning rate α = 0.0001 and optimizer parameters set to β1 = 0.0 and β2 = 0.9.

[0041] The datasets used in the implementation included the CelebA-HQ face dataset from the object category and the Places2 and Cityscape multi-type scene datasets from the scene category. Image interpolation comparison evaluation used the face dataset and the multi-type scene dataset; image extrapolation comparison evaluation used the Cityscape dataset, evaluating bilateral extrapolation. Image interpolation was tested using the mask dataset provided by NVIDIA. Ten randomly selected mask images were used to cover each test image, serving as known regions input to different comparison methods to calculate the average index. The division of training and testing sample sizes is shown in Table 1. During training and testing, the image interpolation and surrounding extrapolation images were scaled proportionally to 256×256 resolution, while the bilateral and unilateral extrapolation images were scaled proportionally to 512×256 resolution.

[0042] In the tests, commonly used image interpolation methods PC, GC, MEDFE, and PUT were selected, along with image extrapolation methods SRN and SpiralNet. The evaluation metrics included Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Frechet Perceptual Distance (FID). Higher PSNR and SSIM values ​​indicate better model performance, while lower FID values ​​indicate superior model performance.

[0043] Table 1. Dataset Training and Test Sample Count Division

[0044] Dataset Training Testing Total CelebA-HQ 28000 2000 30000 Places2 50000 100 50100 Cityscapes 2975 1525 4500

[0045] Table 2. Results of the comparative experiment on quantitative in-image supplementation.

[0046] Metrics (CelebA-HQ) PC GC MEDFE PUT The invention PSNR 27.19 27.44 26.82 24.93 27.52 SSIM 0.9283 0.9347 0.9265 0.8701 0.9318 FID 6.23 5.83 5.48 5.59 5.34 Metrics (Places2) PC GC MEDFE PUT The invention PSNR 26.62 27.36 27.17 27.79 27.82 SSIM 0.8635 0.8813 0.8755 0.8832 0.8845 FID 41.57 30.49 35.57 36.25 30.44

[0047] Table 3. Results of the Quantitative Comparison Experiment on Image Extrapolation

[0048] Method PSNR SSIM FID SRN 20.33 0.6980 28.90 SpiralNet 20.43 0.7125 22.34 The invention 20.47 0.7143 19.57

[0049] The following are the performance evaluation results. Table 2 and Figure 3 The comparison results between the proposed method and other methods are presented. It can be seen that the present invention outperforms other methods in all indicators. Figure 3 The image shows visual comparison results of various methods. The first two rows are examples of results from the CelebA-HQ face dataset, and the last two rows are examples of results from the Places2 dataset. Input is the input sub-image, and GT is the real image. It is evident from the images of the face's eye area and the dam region that the synthesized result of this invention is more realistic and shows no obvious smearing. Table 3 presents the extrapolation quantitative evaluation results, showing that the proposed method improves upon other methods in almost all metrics, demonstrating better performance. Figure 4The comparison of the image results shows that the extrapolation effect of the present invention is more realistic than other methods.

[0050] By combining two deep neural network models, CNN and Transformer, it can complete various types of image inpainting and extrapolation, including image inpainting in regular and irregular cases, and extrapolation in four-sided, two-sided and one-sided cases.

[0051] The above description is merely a preferred embodiment of the present invention and is used only to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A unified method for combining image interpolation and extrapolation, characterized in that, The process of solving the image inpainting and extrapolation problem is explicitly decomposed into three stages: representation, prediction, and synthesis. The representation stage is implemented using a CNN encoder based on partial convolution to process the edge information of known regions. The prediction stage designs a Transformer encoder to predict the features of unknown regions based on known region features, in conjunction with coordinate information. The synthesis stage uses a CNN decoder constructed by overlapping residual blocks and upsampling to finally map the predicted features into a complete image.

2. The method according to claim 1, characterized in that, The Transformer encoder performs parallel feature prediction and combines them into an end-to-end model.

3. The method according to claim 2, characterized in that, In the end-to-end model, partial convolution, pixel normalization, and dimension transformation techniques are used to construct the network structure; several loss functions are superimposed for model training, and the Adam optimizer is used during training.

4. The method according to claim 1 or 3, characterized in that, After each partial convolution, a pixel normalization method is used to balance the feature values ​​at different locations, so that all features fed into the prediction stage have the same data distribution to facilitate more thorough model training.

5. The method according to claim 4, characterized in that, The model is constructed using several residual blocks and upsampling intervals. The images are converted into pixel values ​​using a tangent function and then output as composite images. Finally, a discriminant network is introduced to improve the model's performance.

6. The method according to claim 1, characterized in that, The CNN encoder reduces the dimensionality of the image to a feature space, and then uses the Transformer to predict features in unknown regions.

7. The method according to claim 6, characterized in that, When using Transformer to handle image interpolation and extrapolation problems, it is necessary to represent the two-dimensional data as a one-dimensional sequence data for input, and each sequence element is called a token.

8. The method according to claim 7, characterized in that, The original input, which covers the unknown regions, is arranged in pixel order to be converted into one-dimensional sequence data. In this sequence, each token is composed of one pixel, and pixels in the unknown regions can be filled with special values ​​for labeling.

Images