An image inpainting method, device, terminal equipment and storage medium
By combining an autoencoder module and a non-quantized Transformer module with FFC residual blocks, an image inpainting method is developed that solves the problem of limited receptive field in traditional convolution methods, achieving higher quality image inpainting results.
Patent Information
- Application Number
- CN202310488070.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2043-04-28
Smart Images
Figure CN116739950B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of image / video inpainting, and in particular to an image inpainting method and device, a terminal device and a storage medium. BACKGROUND
[0002] The process of image inpainting is a task process of predicting and inpainting the missing regions in an image by using available information (such as texture, structure and other effective data) in the image to create a high-quality image that is visually similar to the original image.
[0003] In recent years, great progress has been made in the field of image inpainting using a convolutional neural network (CNN) optimized Transformer (also known as a hybrid model). However, when processing an image to be inpainted with a large missing region, the prior art selects a traditional convolution method for the convolution module of the image inpainting model, and the traditional convolution has the problem of limited receptive field. In particular, when processing an image with high resolution or a complex scene, the traditional convolution can only perform linear operations in the spatial domain, and each convolution operation can only process a small local receptive field, resulting in problems such as the image inpainting model being unable to capture global information and missing details, which reduces the quality of image inpainting. SUMMARY
[0004] The embodiments of the present application provide an image inpainting method, device, terminal device and storage medium, which can effectively solve the problem that the traditional convolution can only perform linear operations in the spatial domain, and each convolution operation can only process a small local receptive field, resulting in problems such as the image inpainting model being unable to capture global information and missing details, which reduces the quality of image inpainting.
[0005] An embodiment of the present application provides an image inpainting method, comprising:
[0006] obtaining an image to be inpainted; wherein the image to be inpainted comprises a known region and a missing region;
[0007] inputting the image to be inpainted into an image inpainting model to enable the image inpainting model to reconstruct the missing region in the image to be inpainted and generate an inpainted image;
[0008] wherein the image inpainting model comprises a self-encoder module and a non-quantized Transformer module;
[0009] the encoder in the self-encoder module is configured to extract feature vectors of the image to be inpainted and combine a plurality of feature vectors to generate a target feature map;
[0010] The non-quantized Transformer module is configured to predict potential vectors of the missing region in the target feature map according to feature vectors corresponding to the missing region in the target feature map.
[0011] The vector quantization double codebook module in the autoencoder module is configured to quantize the feature vectors of the known region and quantize the potential vectors, to obtain first quantized feature vectors corresponding to the feature vectors of the known region and second quantized feature vectors corresponding to the potential vectors.
[0012] The decoder in the autoencoder module that fuses the FFC residual block is configured to reconstruct the missing region in the image to be inpainted according to the first quantized feature vectors and the second quantized feature vectors stored in the vector quantization double codebook module, to generate the inpainted image corresponding to the image to be inpainted.
[0013] Preferably, before the image to be inpainted is input into the image inpainting model, the method further comprises:
[0014] Dividing all regions of the image to be inpainted into a plurality of patch regions.
[0015] The encoder in the autoencoder module is configured to extract feature vectors of the image to be inpainted, and combine the feature vectors to generate a target feature map, specifically comprising:
[0016] The encoder in the autoencoder module is configured to process the plurality of patch regions in a non-overlapping block manner and through a plurality of linear layers, to obtain feature vectors corresponding to each patch region, and combine the feature vectors to generate a target feature map.
[0017] Preferably, the autoencoder module further comprises a vector quantization double codebook module.
[0018] After obtaining the first quantized feature vectors corresponding to the feature vectors of the known region and the second quantized feature vectors corresponding to the potential vectors, the method further comprises:
[0019] The vector quantization double codebook module respectively labels the first quantized feature vectors corresponding to the known region in the image to be inpainted and the second quantized feature vectors corresponding to the missing region.
[0020] Preferably, the reconstructing the missing region in the image to be inpainted according to the first quantized feature vectors and the second quantized feature vectors stored in the vector quantization double codebook module, to generate the inpainted image corresponding to the image to be inpainted, specifically comprises:
[0021] The FFC residual block in the decoder extracts local features of all quantized feature vectors by using a traditional convolution through a local branch, and extracts features of all quantized feature vectors in the global context in the spectral domain through a global branch to obtain global features;
[0022] The local features and the global features are fused to obtain fused features;
[0023] The decoder reconstructs the missing region in the to-be-repaired image according to the fused features to generate a repaired image corresponding to the to-be-repaired image.
[0024] Preferably, the training process of the image repairing model comprises:
[0025] The training operation of the image repairing model is repeatedly performed until it is determined that the image repairing model converges:
[0026] A sample to-be-repaired image and a sample repaired image corresponding to the sample to-be-repaired image are obtained;
[0027] The sample to-be-repaired image is input into the image repairing model, so that the encoder in the auto-encoder module of the image repairing model extracts feature vectors of the sample to-be-repaired image and combines a plurality of feature vectors to generate a sample feature map;
[0028] The non-quantized Transformer module in the image repairing model predicts a sample latent vector of the missing region in the sample feature map according to a feature vector corresponding to the missing region in the sample feature map;
[0029] The vector quantization double codebook module in the auto-encoder module of the image repairing model quantizes the feature vectors of the known region in the sample feature map and quantizes the sample latent vector to obtain a sample first quantized feature vector corresponding to the feature vectors of the known region in the sample feature map and a sample second quantized feature vector corresponding to the sample latent vector;
[0030] The decoder fused with the FFC residual block reconstructs the missing region in the sample to-be-repaired image according to the sample first quantized feature vector and the sample second quantized feature vector stored in the vector quantization double codebook module to generate a predicted repaired image corresponding to the sample to-be-repaired image;
[0031] The predicted repaired image is compared with the sample repaired image, and when it is determined according to the comparison result that the image repairing model has not converged, an updated sample to-be-repaired image and a corresponding sample repaired image are obtained.
[0032] Preferably, when the image repairing model is trained, it further comprises:
[0033] The encoder module in the image inpainting model is updated according to a reconstruction loss function; wherein the reconstruction loss function is composed of a pixel loss function, a gradient loss function, an adversarial loss function, a perceptual loss function and a style loss function;
[0034] The pixel loss function is calculated according to the following formula:
[0035]
[0036] Wherein, is the pixel loss function, is the sample to be inpainted image, is the sample to be inpainted image corresponding to the sample repaired image, represents the element subtraction operation, represents the mean operation;
[0037] The gradient loss function is calculated according to the following formula:
[0038]
[0039] Wherein, is the gradient loss function, and grad[·] represents a function for calculating image gradient;
[0040] The adversarial loss function is calculated according to the following formula:
[0041]
[0042] Wherein, is the adversarial loss function, D adv (·) is the function corresponding to the discriminator network;
[0043] The perceptual loss function is calculated according to the following formula:
[0044]
[0045] Wherein, is the perceptual loss function;
[0046] The style loss function is calculated according to the following formula:
[0047]
[0048] Wherein, is the style loss function, and G(·) is the Gram matrix of the obtained parameters;
[0049] The reconstruction loss function is calculated according to the following formula:
[0050]
[0051] wherein, is a reconstruction loss function, λ g is a first preset parameter, λ a is a second preset parameter, λ p is a third preset parameter, λ s is a fourth preset parameter.
[0052] Preferably, when training the image inpainting model, further comprising:
[0053] updating the non-quantized Transformer module in the image inpainting model according to a cross-entropy loss function;
[0054] wherein, the cross-entropy loss function is represented as follows:
[0055]
[0056] wherein, L trans is a cross-entropy loss function, is a distribution probability of the latent vector of the missing region in the target feature map predicted by the non-quantized Transformer module, is an encoder function, and m is a binary mask, when m i,j = 0 represents that the pixel at (i,j) is missing, and when m i,j = 1 represents that the pixel at (i,j) is valid.
[0057] On the basis of the method embodiments described above, the present application correspondingly provides device embodiments.
[0058] An embodiment of the present application provides an image inpainting device, comprising: a to-be-inpainted image acquisition module and an inpainted image generation module;
[0059] The to-be-inpainted image acquisition module is configured to acquire a to-be-inpainted image; wherein the to-be-inpainted image comprises a known region and a missing region.
[0060] The inpainted image generation module is configured to input the to-be-inpainted image into an image inpainting model, so that the image inpainting model reconstructs the missing region in the to-be-inpainted image and generates an inpainted image.
[0061] Wherein, the image inpainting model comprises a self-encoder module and a non-quantized Transformer module.
[0062] The encoder in the self-encoder module is configured to extract feature vectors of the to-be-inpainted image, and combine a plurality of feature vectors to generate a target feature map.
[0063] The non-quantized Transform module is configured to predict the latent vector of the missing region in the target feature map according to the feature vector corresponding to the missing region in the target feature map.
[0064] The vector quantization double codebook module in the auto-encoder module is configured to quantize the feature vector of the known region and quantize the latent vector, to obtain a first quantized feature vector corresponding to the feature vector of the known region and a second quantized feature vector corresponding to the latent vector.
[0065] The decoder in the auto-encoder module, which is fused with the FFC residual block, is configured to reconstruct the missing region in the image to be repaired according to the first quantized feature vector and the second quantized feature vector stored in the vector quantization double codebook module, to generate the repaired image corresponding to the image to be repaired.
[0066] On the basis of the method embodiments described above, the application correspondingly provides terminal device embodiments.
[0067] Another embodiment of the application provides a terminal device, which comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the image repairing method described in the above-mentioned embodiments of the application when executing the computer program.
[0068] On the basis of the method embodiments described above, the application correspondingly provides storage medium embodiments.
[0069] Another embodiment of the application provides a storage medium, which comprises a stored computer program, and when the computer program is running, the computer readable storage medium controls the device where the computer readable storage medium is located to execute the image repairing method described in the above-mentioned embodiments of the application.
[0070] By implementing the application, the following beneficial effects are achieved:
[0071] This invention provides an image restoration method, apparatus, terminal device, and storage medium. The invention inputs the image to be restored into an image restoration model, whereby the encoder in the autoencoder module extracts feature vectors from the image and combines these feature vectors to generate a target feature map. Further, the non-quantized Transformer module in the image restoration model predicts the latent vectors of the missing regions in the target feature map based on the feature vectors corresponding to the missing regions. After obtaining the first quantized feature vector corresponding to the known region's feature vector and the second quantized feature vector corresponding to the latent vector, the decoder in the autoencoder module reconstructs the missing regions in the image to be restored based on the first and second quantized feature vectors. The decoder in the autoencoder module not only recovers the content of the missing regions based on the latent vectors repaired by the non-quantized Transformer module but also maintains the content of the known regions of the image to be restored unchanged. Furthermore, based on the quantized feature vectors in the image to be restored... When reconstructing missing regions, the decoder of this invention introduces FFC residual blocks to reconstruct the image to be repaired. The FFC residual blocks can perform Fourier transforms on the feature maps through global branches and update them in the spectral domain that affects the global context. Compared with traditional convolution, which can only perform linear operations in the spatial domain and each convolution operation can only process a small local receptive field, the FFC residual blocks of this invention can perform linear operations in the frequency domain (complex space). Especially when processing images with high resolution or complex scenes, it allows convolution operations to more effectively process the global receptive field, enabling the model to obtain a receptive field covering the entire image in a shallower layer of the network. The increase in the receptive field can capture richer image information, that is, more accurately capture the detailed information in the image. Thus, the FFC residual blocks of this invention can improve the accuracy of missing region prediction when reconstructing the image to be repaired, making the final repaired image more restored and realistic. In other words, the image repair model of this invention can more accurately capture global information and missing detailed information, improving the quality of image repair. Attached Figure Description
[0072] Figure 1 This is a schematic flowchart of an image restoration method provided in an embodiment of the present invention.
[0073] Figure 2 This is a schematic diagram of the structure of an image restoration model provided in an embodiment of the present invention.
[0074] Figure 3 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of the present invention. Detailed Implementation
[0075] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.
[0076] As shown in Figure 1 is a flowchart of an image repairing method provided by an embodiment of the present application;
[0077] An image repairing method provided by an embodiment of the present application comprises:
[0078] Step S1: obtaining a to-be-repaired image; wherein the to-be-repaired image comprises a known region and a missing region;
[0079] Step S2: inputting the to-be-repaired image into an image repairing model, so that the image repairing model reconstructs the missing region in the to-be-repaired image to generate a repaired image; wherein the image repairing model comprises a self-encoder module and a non-quantized Transformer module;
[0080] The encoder in the self-encoder module is configured to extract feature vectors of the to-be-repaired image and combine a plurality of feature vectors to generate a target feature map;
[0081] The non-quantized Transformer module is configured to predict a latent vector of the missing region in the target feature map according to a feature vector corresponding to the missing region in the target feature map;
[0082] The vector quantization double codebook module in the self-encoder module is configured to quantize the feature vectors of the known region and quantize the latent vector to obtain a first quantized feature vector corresponding to the feature vectors of the known region and a second quantized feature vector corresponding to the latent vector;
[0083] The decoder in the self-encoder module, which fuses the FFC residual block, is configured to reconstruct the missing region in the to-be-repaired image according to the first quantized feature vector and the second quantized feature vector stored in the vector quantization double codebook module to generate the repaired image corresponding to the to-be-repaired image.
[0084] For step S1, in a preferred embodiment, a to-be-repaired image comprising a known region and a missing region is obtained; it should be noted that the missing region is a region shielded by a mask;
[0085] In a preferred embodiment, before the to-be-repaired image is input into the image repairing model, the method further comprises:
[0086] All regions of the image to be inpainted are divided into several patch regions. Illustratively, the pixels of each patch region are 8*8 in size.
[0087] For step S2, in one preferred embodiment, the image to be inpainted is input into the image inpainting model to enable the image inpainting model to reconstruct the missing region in the image to be inpainted, generating an inpainted image.
[0088] Specifically, as shown in Figure 2 The image inpainting model includes a self-encoder module and a non-quantitative Transformer module.
[0089] The encoder in the self-encoder module includes an encoder P-Enc (Patch-based Encoder), a vector quantization dual codebook module D-Codes, and a decoder RR-Dec (Resolution-Robust Decoder).
[0090] The encoder P-Enc in the self-encoder module is used to extract feature vectors of the image to be inpainted and combine several feature vectors to generate a target feature map.
[0091] Wherein, the encoder P-Enc processes several patch regions in a non-overlapping block manner through multiple linear layers to obtain a feature vector corresponding to each patch region. Traditional CNN encoders use multiple convolution kernels to process input images in a sliding window manner. Compared with traditional CNN-based encoders, the encoder P-Enc of the present application adopts a unique way to process input images, which processes input images in a non-overlapping block manner through multiple linear layers. This design is more suitable for image inpainting because it avoids introducing interference between the missing region and the known region, making the image inpainting effect better.
[0092] The vector quantization dual codebook module D-Codes of the self-encoder module labels the quantized feature vectors corresponding to the known region and the quantized feature vectors corresponding to the missing region in the image to be inpainted, respectively. Specifically, the vector quantization dual codebook module D-Codes is an effective method for representing high-dimensional vectors. In the model, the quantized feature vectors are represented as a combination of two codebooks, and the labels of the quantized feature vectors in the known region and the missing region are stored, i.e. e and e ' This further eliminates the difference between the missing patch and the known patch, so that a more reasonable result can be generated when the model predicts the features of the missing patch.
[0093] In a preferred embodiment, the non-quantized Transformer module is configured to predict a latent vector of a missing region in the target feature map according to a feature vector corresponding to the missing region in the target feature map; and the non-quantized Transformer module is also referred to as a UQ-Transformer model.
[0094] Specifically, directly taking the target feature map generated by the autoencoder module as the input of the non-quantized Transformer module can avoid the information loss caused by the down-sampling and quantization processes, so that the UQ-Transformer model can more accurately predict the corresponding latent vector when using the patch-based feature vector from the P-Enc as the input to predict the latent vector of the missing region; that is, directly taking the unquantized image feature as the input of the UQ-Transformer model ensures that the damaged image completes the image reconstruction while retaining as much information as possible.
[0095] The core mechanism of the UQ-Transformer model of the present application is the self-attention mechanism, which allows the model to weigh the importance of different elements in the feature vector and predict these elements according to the weighted sum. The self-attention mechanism is also implemented as a series of parallel multi-head attention layers that focus on different parts of the input sequence, and then the outputs of these attention layers are concatenated and transformed through a feed-forward layer to produce the final output representation.
[0096] In a preferred embodiment, the decoder RR-Dec in the autoencoder module quantizes the latent vector output by the UQ-Transformer model to obtain a quantized feature vector corresponding to the latent vector, quantizes the feature vector of the known region to obtain a first quantized feature vector, and quantizes the latent vector to obtain a second quantized feature vector.
[0097] The first quantized feature vector and the second quantized feature vector are input into the decoder to allow the decoder to reconstruct the missing region in the image to be repaired according to the first quantized feature vector and the second quantized feature vector, and generate a repaired image corresponding to the image to be repaired.
[0098] Illustratively, the decoder RR-Dec not only recovers the content in the missing region from the repair mask, but also keeps the content in the known region unchanged. The decoder is composed of a main branch and a reference branch structure: the main branch uses multiple deconvolution layers to generate the repaired image, while the reference branch extracts multi-scale feature maps from the image to be repaired. The feature maps obtained by the reference branch are fused with the features of the main branch through a mask-guided addition module (MGA), so that the known pixels in the image to be repaired can be used to recover the missing region. where the predicted quantized vectors in the missing region are used to recover the unknown region masked by m.
[0099]
[0100] where and denote the features on the main branch and the reference branch respectively, with spatial size (0≤l≤log2r), and m l denotes the binary mask corresponding to the above spatial size, with value 1 indicating the known region and value 0 indicating the existence of missing pixels in the region.
[0101] The FFC residual block is introduced in the decoder RR-Dec, so that the decoder can obtain the global receptive field of the coverage image in the shallow layer of the network compared with the original, and the increase of the effective receptive field makes the image information captured by the decoder more abundant, and the details of the image are captured more accurately, which will improve the accuracy of predicting the content in the missing region during image inpainting.
[0102] The FFC residual block is based on channel fast Fourier transform (FFT), which divides the channel into two parallel branches: the local branch uses traditional convolution to process local features, and the global branch uses FFT to analyze global context features. An important difference between FFC and traditional convolution is that they process data differently: traditional convolution is a linear operation in the spatial domain, and each convolution operation can only process a small local receptive field; while FFC is a linear operation in the frequency domain (complex space), which can make the convolution operation more effectively process the global receptive field. This is because in the frequency domain, all information is efficiently represented as a series of complex components, and FFC operations can combine these components through simple multiplication and addition operations, thereby simplifying the convolution operation process. Therefore, the decoder of the present application can convert data from the time domain to the frequency domain through fast Fourier transform, then perform convolution operation, and obtain the global receptive field more quickly than traditional convolution. Finally, the local branch and the global branch are fused and output to obtain the fused features, so that the decoder RR-Dec can reconstruct the missing region in the image to be inpainted according to the fused features, and generate the inpainted image corresponding to the image to be inpainted.
[0103] The prior art Transformer will downsample the input image to a lower resolution and quantize it to a smaller pixel space for efficiency, both of which will cause severe information loss in the image, and this information loss is difficult to compensate for by subsequent refinement networks, causing serious harm to image inpainting. Although there is already an auto-encoder optimized Transformer that no longer needs to downsample and quantize the image, the traditional convolution method is still used, which has a limited receptive field problem, which to some extent limits the potential of the hybrid model, causing the model to lack an effective learning process for both global structure and detail information. The present application can improve the receptive field limitation problem caused by traditional convolution in the hybrid model while maintaining the efficiency of Transformer computation. The model of the present application designs a self-encoder module (P-VQVAE+) module, which introduces FFC residual blocks into the original auto-encoder architecture, abolishes the original traditional convolution residual block, and combines the self-encoder module with the UQ-Transformer, so that the final image inpainting model realizes a receptive field covering the entire image in the early network layer, so that the inpainting model obtains a large and effective receptive field, effectively reducing the information loss when reconstructing the image, and outputting a restored image with a repaired image. At the same time, the model uses less training data and computation to improve performance.
[0104] In a preferred embodiment, the training process of the image inpainting model comprises:
[0105] The training operation of the image inpainting model is repeatedly performed until the image inpainting model converges:
[0106] Obtain a sample to-be-repaired image and a sample repaired image corresponding to the sample to-be-repaired image;
[0107] Input the sample to-be-repaired image into the image inpainting model, so that the encoder in the self-encoder module of the image inpainting model extracts the feature vectors of the sample to-be-repaired image, and combines a plurality of feature vectors to generate a sample feature map;
[0108] The non-quantized Transformer module in the image inpainting model predicts the sample latent vector of the missing region in the sample feature map according to the feature vector corresponding to the missing region in the sample feature map;
[0109] The vector quantization double codebook module in the auto-encoder module of the image inpainting model quantizes the feature vectors of the known region in the sample feature map and quantizes the sample latent vector, to obtain a sample first quantized feature vector corresponding to the feature vectors of the known region in the sample feature map and a sample second quantized feature vector corresponding to the sample latent vector;
[0110] The decoder fused with the FFC residual block reconstructs the missing region in the sample to-be-inpainted image according to the sample first quantized feature vector and the sample second quantized feature vector stored in the vector quantization double codebook module, to generate a predicted repaired image corresponding to the sample to-be-inpainted image;
[0111] The predicted repaired image is compared with the sample repaired image, and when it is determined according to the comparison result that the image inpainting model has not converged, an updated sample to-be-inpainted image and a corresponding sample to-be-inpainted image are obtained.
[0112] In a preferred embodiment, when the image inpainting model is trained, the following steps are further included:
[0113] The encoder module in the image inpainting model is updated according to a reconstruction loss function; wherein the reconstruction loss function is composed of a pixel loss function, a gradient loss function, an adversarial loss function, a perceptual loss function and a style loss function;
[0114] The pixel loss function is calculated according to the following formula:
[0115]
[0116] wherein, the pixel loss function is, the sample to-be-inpainted image is, the sample repaired image corresponding to the sample to-be-inpainted image is, the element subtraction operation is represented by, the mean operation is represented by;
[0117] The gradient loss function is calculated according to the following formula:
[0118]
[0119] wherein, the gradient loss function is grad[·], and grad[·] represents a function for calculating image gradient;
[0120] The adversarial loss function is calculated according to the following formula:
[0121]
[0122] wherein, the adversarial loss function is, is a function corresponding to the discriminator network;
[0123] The perceptual loss function is calculated according to the following formula:
[0124]
[0125] wherein, is a perceptual loss function;
[0126] The style loss function is calculated according to the following formula:
[0127]
[0128] wherein, is a style loss function, G(·) is a Gram matrix for obtaining parameters;
[0129] The reconstruction loss function is calculated according to the following formula:
[0130]
[0131] wherein, is a reconstruction loss function, λ g is a first preset parameter, λ a is a second preset parameter, λ p is a third preset parameter, λ s is a fourth preset parameter;
[0132] The reconstruction loss is a function for calculating the difference between the input image and the reconstructed image, which consists of five parts, including L1 loss gradient of the two images adversarial loss perceptual loss and style loss In a preferred embodiment, λ g = 5, λ a = 0.1, λ p = 0.1, λ s = 250.
[0133] The final loss function of the encoder module is:
[0134]
[0135] L vae is a final loss function, is a codebook loss vector for optimizing the latent vector, For the commitment loss of gradient information from the decoder to the encoder, alpha is a weight parameter.
[0136] In a preferred embodiment, when training the image inpainting model, further comprising:
[0137] According to the cross-entropy loss function, the non-quantized Transformer module in the image inpainting model is updated;
[0138] Wherein, the cross-entropy loss function is represented as follows:
[0139]
[0140] Wherein, L trans is the cross-entropy loss function, is the distribution probability of the latent vector predicted by the non-quantized Transformer module for the missing region in the target feature map, is the encoder function, and m is a binary mask, when m i,j = 0 indicates that the pixel at (i,j) is missing, and when m i,j = 1 indicates that the pixel at (i,j) is valid. Indicates the token of the first parameter, which is obtained by obtaining the index of all quantized vectors in , and O(·) sets the given parameter value to 1. The present application can simultaneously integrate the powerful modeling capability of CNN in texture structure and other information and the powerful modeling capability of Transformer in long-distance relationship, so that the model can restore more detailed image information when repairing high-resolution images with large missing regions, and also ensure that the repair result is reasonable in vision.
[0141] The present application can use auto-encoder to abolish the operation of down-sampling and quantization of the image, i.e. the unquantized image features are directly used as the input of the UQ-Transformer model, ensuring that the damaged image input completes the reconstruction of the image on the premise of preserving as much information as possible. The present application can solve the problem of serious information loss caused by using Transformer for image inpainting to improve the calculation efficiency, and can ensure that the overall model can still improve the repair effect while using less data amount and lower calculation complexity.
[0142] The FFC residual block is introduced in the mixed model, so that the model can obtain a global receptive field covering an image in a shallow layer of the network, the increased receptive field can capture more rich image information, more accurately capture the detailed information in the image, and improve the accuracy of the missing area prediction. Meanwhile, all the training data and test data are selected in the data set, which proves that the repair results of the model in various scenes are improved, especially when there is a large missing area in the image, the improvement effect is particularly prominent. Since the FFC residual block is introduced, for some pictures with periodic structure, the model can better extract effective information for image inpainting, so that the repair result is more in line with the visual perception of the human eye, that is, the model can also obtain excellent results when repairing the repetitive structure in the artificial environment.
[0143] As Figure 3 shown, on the basis of the embodiments of the above various image inpainting methods, the application correspondingly provides device embodiments;
[0144] An embodiment of the application provides an image inpainting device, comprising: a to-be-repaired image acquisition module and a repaired image generation module.
[0145] The to-be-repaired image acquisition module is configured to acquire a to-be-repaired image; wherein the to-be-repaired image comprises a known area and a missing area.
[0146] The repaired image generation module is configured to input the to-be-repaired image into an image inpainting model, so that the image inpainting model reconstructs the missing area in the to-be-repaired image and generates a repaired image.
[0147] The image inpainting model comprises a self-encoder module and a non-quantized Transformer module.
[0148] The encoder in the self-encoder module is configured to extract feature vectors of the to-be-repaired image and combine a plurality of feature vectors to generate a target feature map.
[0149] The non-quantized Transformer module is configured to predict a latent vector of the missing area in the target feature map according to a feature vector corresponding to the missing area in the target feature map.
[0150] The vector quantization double codebook module in the self-encoder module is configured to quantize the feature vectors of the known area and quantize the latent vector, to obtain a first quantized feature vector corresponding to the feature vectors of the known area and a second quantized feature vector corresponding to the latent vector.
[0151] The decoder of the FFC residual block is fused in the auto-encoder module, and is used for reconstructing the missing region in the to-be-repaired image according to the first quantized feature vector and the second quantized feature vector stored in the vector quantization double codebook module, to generate a repaired image corresponding to the to-be-repaired image.
[0152] It should be noted that the apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, which can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiments provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.
[0153] Those skilled in the art can clearly understand that, for the convenience and brevity, the specific working process of the apparatus described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.
[0154] On the basis of the embodiments of the various image repairing methods described above, the present application correspondingly provides terminal device item embodiments.
[0155] An embodiment of the present application provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, when the computer program is executed by the processor, realizing an image repairing method according to any one of the method item embodiments of the present application.
[0156] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing terminal devices. The terminal device can include, but is not limited to, a processor and a memory.
[0157] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the terminal device, and connects all parts of the terminal device through various interfaces and lines.
[0158] The memory can be used to store the computer program, and the processor realizes various functions of the terminal device by running or executing the computer program stored in the memory and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function, etc. The data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device or other volatile solid-state memory device.
[0159] On the basis of the above-mentioned various image repairing method embodiments, the application correspondingly provides a storage medium embodiment.
[0160] An embodiment of the application provides a storage medium, which comprises a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located performs an image repairing method according to any one of the method embodiments of the application.
[0161] The storage medium is a computer readable storage medium, and the computer program is stored in the computer readable storage medium. When the computer program is executed by a processor, steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
[0162] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.
Claims
1. An image restoration method, characterized in that, include: Obtain the image to be repaired; wherein the image to be repaired includes known regions and missing regions; The image to be repaired is input into the image repair model so that the image repair model can reconstruct the missing areas in the image to be repaired and generate a repaired image; The image restoration model includes an autoencoder module and a non-quantized Transformer module. The encoder in the autoencoder module is used to extract the feature vectors of the image to be repaired, and to combine the extracted feature vectors to generate a target feature map. The non-quantized Transformer module is used to predict the potential vector of the missing region in the target feature map based on the feature vector corresponding to the missing region in the target feature map. The vector quantization dual codebook module in the autoencoder module is used to quantize the feature vector of the known region and the latent vector to obtain the first quantized feature vector corresponding to the feature vector of the known region and the second quantized feature vector corresponding to the latent vector. The autoencoder module includes a decoder that incorporates FFC residual blocks. The FFC residual blocks in the decoder extract local features from all quantized feature vectors using traditional convolution through local branches, and extract global features from all quantized feature vectors in the spectral domain of the global context through global branches. The local and global features are then fused to obtain fused features. The decoder reconstructs the missing regions in the image to be repaired based on the fusion features, generating a repaired image corresponding to the image to be repaired.
2. The image restoration method as described in claim 1, characterized in that, Before inputting the image to be repaired into the image restoration model, the following steps are also included: The entire region of the image to be repaired is divided into multiple patch regions; The encoder in the autoencoder module is used to extract feature vectors from the image to be repaired, and to combine the extracted feature vectors to generate a target feature map, specifically including: The encoder in the autoencoder module is used to process multiple patch regions in a non-overlapping block manner through multiple linear layers to obtain the feature vector corresponding to each patch region, and to combine the feature vectors corresponding to each patch region to generate a target feature map.
3. The image restoration method as described in claim 1, characterized in that, After obtaining the first quantized feature vector corresponding to the feature vector of the known region and the second quantized feature vector corresponding to the latent vector, the process also includes: The vector quantization dual codebook module marks the first quantization feature vector corresponding to the known region and the second quantization feature vector corresponding to the missing region in the image to be repaired.
4. The image restoration method as described in claim 1, characterized in that, The training process of the image restoration model includes: Repeat the following training operations for the image restoration model until it is determined that the image restoration model has converged: Obtain the sample image to be repaired and the corresponding repaired sample image; The sample image to be repaired is input into the image repair model, so that the encoder in the autoencoder module of the image repair model extracts the feature vector of the sample image to be repaired, and combines the extracted feature vectors to generate a sample feature map. The non-quantized Transformer module in the image inpainting model predicts the latent vector of the missing region in the sample feature map based on the feature vector corresponding to the missing region in the sample feature map. The vector quantization dual codebook module in the autoencoder module of the image restoration model quantizes the feature vector of the known region in the sample feature map and quantizes the sample latent vector to obtain the first quantized feature vector of the sample corresponding to the feature vector of the known region in the sample feature map and the second quantized feature vector of the sample corresponding to the sample latent vector. The decoder, which incorporates FFC residual blocks, reconstructs the missing regions in the sample image to be repaired based on the first quantized feature vector and the second quantized feature vector of the sample stored in the vector quantization dual codebook module, and generates the predicted repaired image corresponding to the sample image to be repaired. The predicted repaired image is compared with the sample repaired image. If the image repair model is determined not to have converged based on the comparison result, the updated sample image to be repaired and the sample image to be repaired corresponding to the updated sample image to be repaired are obtained.
5. The image restoration method as described in claim 4, characterized in that, Training the image restoration model also includes: The encoder module in the image inpainting model is updated according to the reconstruction loss function; wherein, the reconstruction loss function consists of pixel loss function, gradient loss function, adversarial loss function, perceptual loss function and style loss function; The pixel loss function is calculated using the following formula: in, For pixel loss function, The sample image to be repaired. The image to be repaired is the image that has already been repaired, corresponding to the image to be repaired. This represents element-wise subtraction. This represents the mean operation; The gradient loss function is calculated using the following formula: in, Let be the gradient loss function. This represents a function for calculating the gradient of an image. The adversarial loss function is calculated using the following formula: in, For adversarial loss function, The function corresponding to the discriminator network; The perceptual loss function is calculated using the following formula: in, For the perceptual loss function; The style loss function is calculated using the following formula: in, Let be the style loss function. To obtain the Gram matrix of the parameters; The reconstruction loss function is calculated using the following formula: in, To reconstruct the loss function, First preset parameter, This is the second preset parameter. This is the third preset parameter. This is the fourth preset parameter.
6. The image restoration method as described in claim 5, characterized in that, Training the image restoration model also includes: The non-quantized Transformer module in the image inpainting model is updated based on the cross-entropy loss function; The cross-entropy loss function is expressed as follows: in, Let cross-entropy be the loss function. The non-quantized Transformer module predicts the distribution probability of latent vectors for missing regions in the target feature map. For encoder functions, For binary masks, when express Pixels missing at that location, when express The pixels at that location are valid.
7. An image restoration device, characterized in that, include: The module for acquiring images to be repaired and the module for generating repaired images; The image to be repaired acquisition module is used to acquire the image to be repaired; wherein, the image to be repaired includes known regions and missing regions; The repaired image generation module is used to input the image to be repaired into the image repair model, so that the image repair model can reconstruct the missing areas in the image to be repaired and generate a repaired image. The image restoration model includes an autoencoder module and a non-quantized Transformer module. The encoder in the autoencoder module is used to extract the feature vectors of the image to be repaired, and to combine the extracted feature vectors to generate a target feature map. The non-quantized Transformer module is used to predict the potential vector of the missing region in the target feature map based on the feature vector corresponding to the missing region in the target feature map. The vector quantization dual codebook module in the autoencoder module is used to quantize the feature vector of the known region and the latent vector to obtain the first quantized feature vector corresponding to the feature vector of the known region and the second quantized feature vector corresponding to the latent vector. The autoencoder module includes a decoder that incorporates FFC residual blocks. The FFC residual blocks in the decoder extract local features from all quantized feature vectors using traditional convolution through local branches, and extract global features from all quantized feature vectors in the spectral domain of the global context through global branches. The local and global features are then fused to obtain fused features. The decoder reconstructs the missing regions in the image to be repaired based on the fusion features, generating a repaired image corresponding to the image to be repaired.
8. A terminal device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements an image restoration method as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to perform an image restoration method as described in any one of claims 1 to 6.