An Image Inpainting Method and System Guided by Semantic Information

Abstract

This invention discloses a semantic information-guided image restoration method and system, relating to the field of image processing. The invention comprises the following steps: extracting a semantic feature map from a masked image, integrating the semantic features into an image restoration network through a cross-attention mechanism and a gating mechanism; converting the semantic feature map input to the image restoration network from the spatial domain to the wavelet domain through a wavelet transform; performing a convolution operation on the wavelet domain, recursively decomposing low-frequency components, guiding the convolutional network to respond to low-frequency information, and employing an inverse wavelet transform to obtain enhanced image features; and performing comprehensive multi-scale extraction and fusion of the semantic feature map and refined and enhanced image features to generate a restoration result. This invention can achieve semantically consistent and detailed image restoration.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more particularly to an image restoration method and system guided by semantic information. Background Art

[0002] In image restoration tasks, convolution methods based on the Fast Fourier Transform (FFT) primarily achieve efficient image reconstruction and missing area filling through frequency domain transformation. First, the input image is converted from the spatial domain to the frequency domain using the Fast Fourier Transform (FFT). This transform transforms the image data representation from pixel-level spatial features to frequency domain features, decomposing the image information into high- and low-frequency components. After conversion to the frequency domain, the model performs frequency domain convolution. Finally, an inverse Fast Fourier Transform (IFT) is used to restore the frequency domain features back to the spatial domain to generate the final restored image.

[0003] Although image restoration methods based on fast Fourier transform (FFT) convolution excel in computational efficiency and global information capture, they still suffer from several technical limitations. First, during the frequency domain conversion process, some fine spatial information may be lost, resulting in insufficient detail restoration in the restored image, and prone to texture blurring or missing details. FFT convolution-based methods perform poorly when dealing with irregularly shaped defects or non-uniform image damage. This is because frequency domain feature processing is more global, making it difficult to effectively fill small or irregular defect areas, resulting in insufficient detail consistency in the restored image. FFT convolution is limited in its ability to restore high-frequency details, as it is more suitable for global feature extraction and relatively weak in capturing and enhancing specific high-frequency details. This can result in the restored image lacking clarity in texture and edge details, especially when high-precision restoration is required. Furthermore, the model's generalization ability is relatively limited when handling image restoration tasks in different scenarios and types.

[0004] Therefore, how to solve the above problems urgently needs to be studied by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an image restoration method and system based on semantic information guidance to solve the problems in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An image restoration method based on semantic information guidance includes the following steps:

[0008] Extract semantic feature maps of masked images and integrate the semantic features into the image restoration network through cross-attention and gating mechanisms;

[0009] The semantic feature map input to the image restoration network is converted from the spatial domain to the wavelet domain through wavelet transform. Convolution operation is performed on the wavelet domain to recursively decompose the low-frequency components, guide the convolution network to respond to low-frequency information, and use inverse wavelet transform to obtain enhanced image features.

[0010] The semantic feature map and the refined and enhanced image features are comprehensively extracted and fused at multiple scales to generate the restoration result.

[0011] Optionally, the image restoration network also includes the introduction of central differential convolution to capture intensity and gradient information and angular differential convolution to focus on angle changes. The calculation process is as follows:

[0012]

[0013] in, is the reparameterized weight tensor of the convolution kernel, is the convolution kernel of ordinary convolution, is the convolution kernel of the central difference convolution, is the convolution kernel of angular difference convolution.

[0014] Optionally, the image restoration network inputs the semantic feature map and converts the image from the spatial domain to the wavelet domain through wavelet transform. The calculation formula is as follows:

[0015]

[0016] in, is the low-frequency component of the semantic feature map, 、 、 are the horizontal, vertical, and diagonal high-frequency components of the semantic feature map, respectively, and X is the semantic feature map.

[0017] Optionally, the specific formula for inverse wavelet transform is as follows:

[0018]

[0019] Among them, X is the semantic feature map.

[0020] Optionally, a gating mechanism is used to dynamically adjust the role of CLIP semantic features in the image generation process according to image features to optimize the final restoration effect.

[0021] Optionally, the cross-attention mechanism is specifically as follows: the semantic features are projected into the corresponding dimension as the query Q, and the image features extracted by the convolutional network are used as the key K and value V. The calculation process is as follows:

[0022]

[0023] Among them, F represents the computational output of cross attention, is the scaling factor.

[0024] Optionally, the loss function includes Loss, Fighting Loss , Perceptual Loss and style loss , the total loss function is:

[0025] .

[0026] An image restoration system guided by semantic information, comprising:

[0027] Large-scale visual language model for extracting semantic feature maps from masked images;

[0028] Semantic guidance module: used to incorporate semantic features into the image restoration network through cross-attention mechanism and gating mechanism;

[0029] Wavelet convolution module: It is used to convert the semantic feature map input to the image restoration network from the spatial domain to the wavelet domain through wavelet transform, perform convolution operations on the wavelet domain, recursively decompose the low-frequency components, guide the convolution network to respond to low-frequency information, and use inverse wavelet transform to obtain enhanced image features;

[0030] Multi-scale fusion module: used to comprehensively extract and fuse the semantic feature map and the refined and enhanced image features at multiple scales to generate the restoration results.

[0031] The above technical solution demonstrates that, compared to existing technologies, the present invention provides a semantically guided image restoration method and system, enabling semantically consistent and detailed image restoration. Furthermore, the wavelet convolution module effectively extracts both global and low-level local information from the image, while the semantic guidance module provides semantic guidance to the model, improving its performance and generalization, resulting in more refined and coordinated image restoration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 It is a structural schematic diagram of the present invention;

[0034] Figure 2 This is a wavelet convolution module diagram of the present invention;

[0035] Figure 3 This is the semantic guidance module diagram of the present invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] The embodiment of the present invention discloses an image restoration method based on semantic information guidance, comprising the following steps:

[0038] Extract semantic feature maps of masked images and integrate the semantic features into the image restoration network through cross-attention and gating mechanisms;

[0039] The semantic feature map input to the image restoration network is converted from the spatial domain to the wavelet domain through wavelet transform. Convolution operation is performed on the wavelet domain to recursively decompose the low-frequency components, guide the convolution network to respond to low-frequency information, and use inverse wavelet transform to obtain enhanced image features.

[0040] The semantic feature map and the refined and enhanced image features are comprehensively extracted and fused at multiple scales to generate the restoration result.

[0041] Specifically, the overall process is as follows Figure 1 As shown in the figure, for each masked image input, the system first extracts semantic features using a large-scale visual language model to capture the rich semantic information contained in the image. In the wavelet convolution module, continuous downsampling and upsampling operations continuously refine and enhance image features. This combined with the extracted semantic information enables comprehensive extraction and fusion of multi-scale features, resulting in clear and natural restoration results.

[0042] The wavelet convolution module of the present invention is as follows Figure 2 As shown. Decompose the input feature map X into the wavelet domain, the formula is as follows:

[0043]

[0044] in, is the low-frequency component of the semantic feature map, 、 、 are the horizontal, vertical, and diagonal high-frequency components of the semantic feature map, and X is the semantic feature map.

[0045] Convolution in the wavelet domain can expand the receptive field of convolution, and recursively decompose the low-frequency components of the feature map before performing the convolution operation, so that CNN can better respond to the low-frequency information of the image, that is, the global structure.

[0046] Then the inverse wavelet transform is used to get the final output result. The calculation process is as follows:

[0047]

[0048] In addition, the central difference convolution (CDC) that captures intensity and gradient information and the angular difference convolution (ADC) that focuses on angle changes are introduced to enhance the network's ability to represent fine-grained textures.

[0049] In order to avoid increasing parameters and inference time, a reparameterization technique is used to avoid increasing parameters and inference time. The calculation process is as follows:

[0050]

[0051] in, is the reparameterized weight tensor of the convolution kernel, is the convolution kernel of ordinary convolution, is the convolution kernel of the central difference convolution, is the convolution kernel of angular difference convolution.

[0052] The semantic guidance module of the present invention is as follows Figure 3 As shown, classic visual language models (such as CLIP) aim to learn aligned features in an embedding space from image-text pairs through contrastive learning. Due to its powerful feature extraction capabilities, CLIP is well-suited for semantic exploration and can effectively provide semantic priors to improve inpainting performance. This paper uses the pre-trained CLIP "RN50" model as a semantic feature extractor. Considering that CLIP's semantic extraction when processing mask images may have certain inaccuracies, the extracted semantic features are passed to a channel attention module to enhance useful semantic information and suppress irrelevant or distracting signals. Furthermore, the system introduces a gating mechanism that dynamically adjusts the role of CLIP semantic features in the image generation process based on image characteristics to optimize the final inpainting effect.

[0053] Then, a cross-attention mechanism is used to exchange information and embed semantic information into the image rendering network. The semantic features are projected into the corresponding dimensions as the query, and the image features extracted by the convolutional network are used as the key and value. The calculation process is as follows:

[0054]

[0055] Among them, F represents the computational output of cross attention, is the scaling factor.

[0056] Finally, the image features with semantic information obtained by cross-attention are concatenated with the original enhanced features in the channel dimension to form the final feature representation.

[0057] The present invention is trained on a joint loss, which includes Loss, Fighting Loss , Perceptual Loss and style loss , the total loss function is:

[0058] .

[0059] We use the widely accepted Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Perceptual Similarity Index (LPIPS) to evaluate the performance of our network. Our approach performs well across multiple evaluation metrics, significantly outperforming other state-of-the-art techniques. Specifically, on the CelebA-HQ dataset, our approach achieves an average 2.8% improvement in PSNR and an average 1.1% improvement in SSIM over the state-of-the-art, demonstrating significant improvement in preserving image structure and detail similarity.

[0060] This embodiment also discloses an image restoration system guided by semantic information, which is as follows:

[0061] Large-scale visual language model for extracting semantic feature maps from masked images;

[0062] Semantic guidance module: used to incorporate semantic features into the image restoration network through cross-attention mechanism and gating mechanism;

[0063] Wavelet convolution module: It is used to convert the semantic feature map input to the image restoration network from the spatial domain to the wavelet domain through wavelet transform, perform convolution operations on the wavelet domain, recursively decompose the low-frequency components, guide the convolution network to respond to low-frequency information, and use inverse wavelet transform to obtain enhanced image features;

[0064] Multi-scale fusion module: used to comprehensively extract and fuse the semantic feature map and the refined and enhanced image features at multiple scales to generate the restoration results.

[0065] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0066] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic information-guided image restoration method, characterized in that: The following steps are involved: Extract semantic feature maps of masked images and integrate the semantic features into the image restoration network through cross-attention and gating mechanisms; The semantic feature map input to the image restoration network is converted from the spatial domain to the wavelet domain through wavelet transform. Convolution operation is performed on the wavelet domain to recursively decompose the low-frequency components, guide the convolution network to respond to low-frequency information, and use inverse wavelet transform to obtain enhanced image features. The semantic feature map and the refined and enhanced image features are comprehensively extracted and fused at multiple scales to generate the restoration result; It also includes the introduction of central differential convolution to capture intensity and gradient information in the image restoration network, as well as angular differential convolution that focuses on angle changes. The calculation process is as follows: Output=Conv(W Rep ,Input), IN Rep =In VC +W CDC +W ADC , Among them, W Rep is the reparameterized weight tensor of the convolution kernel, W VC is the convolution kernel of ordinary convolution, W CDC is the convolution kernel of the central difference convolution, W ADC is the convolution kernel of angular difference convolution; The image restoration network inputs the semantic feature map and converts the image from the spatial domain to the wavelet domain through wavelet transform. The calculation formula is as follows: [X LL ,X LH X HL ,X HH ]=WT(X) Among them, X LL is the low-frequency component of the semantic feature map, X LH 、X HL 、X HH are the horizontal, vertical, and diagonal high-frequency components of the semantic feature map, respectively, and X is the semantic feature map; The specific formula of inverse wavelet transform is as follows: O = IWT(Conv(WT(X))) Among them, X is the semantic feature map.

2. The image restoration method based on semantic information guidance according to claim 1, characterized in that: The gating mechanism is used to dynamically adjust the role of CLIP semantic features in the image generation process according to image features, thereby optimizing the final restoration effect.

3. The image restoration method based on semantic information guidance according to claim 1, characterized in that: The cross-attention mechanism is specifically as follows: the semantic features are projected into the corresponding dimension as the query Q, and the image features extracted by the convolutional network are used as the key K and value V. The calculation process is as follows: Among them, F represents the computational output of cross attention, is the scaling factor.

4. The image restoration method based on semantic information guidance according to claim 1, characterized in that: Loss functions include l1 loss and adversarial loss Perceptual loss and style loss The total loss function is:

5. An image restoration system guided by semantic information, characterized in that: include: Large-scale visual language model for extracting semantic feature maps from masked images; Semantic guidance module: used to incorporate semantic features into the image restoration network through cross-attention mechanism and gating mechanism; Wavelet convolution module: It is used to convert the semantic feature map input to the image restoration network from the spatial domain to the wavelet domain through wavelet transform, perform convolution operations on the wavelet domain, recursively decompose the low-frequency components, guide the convolution network to respond to low-frequency information, and use inverse wavelet transform to obtain enhanced image features. It also includes the introduction of central difference convolution to capture intensity and gradient information and angular difference convolution to focus on angle changes in the image restoration network. The calculation process is as follows: Output=Conv(W Rep ,Input), IN Rep =In VC +W CDC +W ADC , Among them, W Rep is the reparameterized weight tensor of the convolution kernel, W VC is the convolution kernel of ordinary convolution, W CDC is the convolution kernel of the central difference convolution, W ADC is the convolution kernel of angular difference convolution; The image restoration network inputs the semantic feature map and converts the image from the spatial domain to the wavelet domain through wavelet transform. The calculation formula is as follows: [X LL ,X LH ,X HL ,X HH ]=WT(X) Among them, X LL is the low-frequency component of the semantic feature map, X LH 、X HL 、X HH are the horizontal, vertical, and diagonal high-frequency components of the semantic feature map, respectively, and X is the semantic feature map; The specific formula of inverse wavelet transform is as follows: O = IWT(Conv(WT(X))) Among them, X is the semantic feature map; Multi-scale fusion module: used to comprehensively extract and fuse the semantic feature map and the refined and enhanced image features at multiple scales to generate the restoration results.

Images