Blind image restoration method and system based on self-learning features and hierarchical similarity

Abstract

The invention discloses a blind image restoration method and system based on self-learning features and hierarchical similarity, and belongs to the field of blind image restoration. The problem that a human intervention mechanism is still introduced to different degrees in the existing blind image restoration process, and the network cannot really autonomously learn features required by restoration from a damaged image is solved. The method comprises the following steps: automatically extracting semantic features from a damaged image through a self-learning module; a hierarchical similarity block is introduced into the self-learning module; designing a self-learning semantic loss function, and guiding semantic feature extraction through feature-level soft constraint; the semantic features extracted by the self-learning module and the damaged image are spliced and then input into a restoration module; and carrying out joint training on the self-learning module and the repairing module by adopting a comprehensive loss function. The method is used for pre-binding algorithm selection and image self-adaptive restoration in a scene where a restoration mask is difficult to acquire or unknown when a damaged image is restored.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of blind image inpainting, and particularly relates to a blind image inpainting method based on self-learning features and hierarchical similarity. BACKGROUND

[0002] As a basic problem in the field of image processing and computer vision, the core goal of image inpainting is to reconstruct the damaged area of an image. In the traditional technical path, most image inpainting methods are heavily dependent on a key premise: accurate mask information must be provided to clearly demarcate the damaged and undamaged areas in the image, thereby guiding the subsequent inpainting process. However, this strong dependence on known masks constitutes a significant limitation in real-world application scenarios, because in actual situations, damaged areas often have irregular and unpredictable characteristics, and it is often unrealistic to obtain accurate mask information in advance. Based on this challenge, the task of blind image inpainting, which involves inpainting an image without any known mask, has become an important research direction that needs to be addressed.

[0003] To address the challenge of blind image inpainting, the mainstream solution in existing technology is to decompose the problem into two sequential sub-problems: mask prediction and image inpainting. Specifically, this type of method first uses a mask prediction network to estimate the location of the contaminated area, and then inputs the predicted mask result as prior knowledge into the subsequent image inpainting network to guide the inpainting. However, the effectiveness of this technology is highly dependent on the accuracy of the mask prediction stage; even if there are inevitable minor errors in the predicted mask, they will be amplified in the inpainting stage, leading to a significant decline in inpainting performance. In particular, when the input image contains a large area of damaged regions, the inaccuracy of mask prediction will cause more severe interference, ultimately resulting in the inability of the inpainting result to maintain structural and semantic consistency in the global range.

[0004] To overcome the above-mentioned chain problems caused by inaccurate mask prediction, some research attempts to avoid explicit mask prediction stages and instead use techniques such as "soft mask" or wavelet transform, aiming to achieve end-to-end inpainting. However, these improved methods themselves have new inherent defects. For example, the "soft mask" method is difficult to capture the overall semantic dependency of the image, while the wavelet transform inevitably faces the risk of loss of image feature information during processing. The root cause of these problems lies in the fact that their inpainting processes still introduce artificial intervention mechanisms to varying degrees, failing to truly enable the network to autonomously and intelligently learn the required features from the damaged image.

[0005] The core technical challenges currently facing blind image restoration research can be summarized in three aspects. First, there is the challenge of semantic consistency reconstruction without explicit masking. Traditional methods rely on manual intervention methods such as mask prediction or "soft masking," while true blind restoration requires autonomously extracting effective semantic features from the damaged image without knowing the location and shape of the damaged area. How to avoid the restoration result becoming disconnected from the original image in key textures and structures due to feature extraction bias—for example, causing distortion of facial lip details or vehicle tire outlines—is a primary challenge. Second, there is the inadequacy of cross-layer feature association and effective transmission. The semantic information of damaged images is scattered across different layers of deep networks, and simply stacking network modules makes it difficult to capture potential feature associations between layers. If an effective inter-layer feature complementarity mechanism cannot be established, the restoration process will lack global consistency guidance, especially when the damaged area accounts for 40%-60%, easily leading to problems such as inconsistent colors, blurred details, or structural breaks. Third, there is the dilemma of balancing "weak intervention" with restoration effectiveness. To reduce errors introduced by human intervention, the ideal approach is to provide soft guidance at the feature level through a self-learning mechanism, rather than imposing pixel-level constraints. However, designing a reasonable self-learning loss function that ensures its guidance strength allows the network to learn features beneficial for restoration without excessively restricting the network's autonomous optimization space, thereby avoiding "template-like" restoration results lacking individual feature differences—for example, the loss of original unique features in the restored face is a core challenge in achieving high-quality blind image restoration. Summary of the Invention

[0006] In view of this, the present invention aims to propose a blind image restoration method and system based on self-learning features and hierarchical similarity, in order to solve the problem that existing blind image restoration processes still introduce human-designed intervention mechanisms to varying degrees, failing to truly enable the network to autonomously and intelligently learn the features required for restoration from damaged images.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a blind image inpainting method based on self-learning features and hierarchical similarity, the method comprising: Semantic features are automatically extracted from damaged images through a self-learning module based on the Transformer architecture, which is used to extract the semantic features required for repair from the undamaged areas of damaged images without mask prediction. A hierarchical similarity block is introduced into the self-learning module to calculate the similarity of features between consecutive layers of the Transformer; We design a self-learning semantic loss function to guide semantic feature extraction through feature-level soft constraints. The semantic features extracted by the self-learning module are concatenated with the damaged image and then input into the repair module; A comprehensive loss function is used to jointly train the self-learning module and the repair module. The comprehensive repair module includes reconstruction loss, perceptual loss, style loss and self-learning semantic loss.

[0008] Furthermore, a preferred method is proposed, wherein the construction of the self-learning module includes: The damaged image is encoded through an embedding layer and then fed into a Transformer block to extract semantic features; A hierarchical similarity block is inserted between every two Transformer blocks to calculate the deep similarity of features between layers; The self-learning semantic loss function is applied to the output of the self-learning module to optimize the feature extraction process.

[0009] Furthermore, a preferred method is proposed, wherein the construction of the hierarchical similarity block includes: Obtain features from two consecutive layers of a Transformer and Normalization is performed along the channel dimension, and features are calculated. and Cosine similarity in the spatial domain yields surface similarity. :

[0010] in, CN This indicates a channel normalization operation; Will The data is fed into a multilayer perceptron to extract deeper similarity features in the following ways:

[0011] in, MLP For a multilayer perceptron; With features Multiplication yields the latent similarity features between layers. This enhances the extraction of self-learning semantic features in the following ways:

[0012] at the same time, Added to In this process, the results are used as new features from the previous layer and then re-input into the current layer.

[0013] Furthermore, a preferred method is proposed, wherein the calculation of the self-learning semantic loss function includes: The output features of the self-learning module and real images Input to VGG network; Compare the feature differences between shallow convolutional layers and deep fully connected layers respectively; calculate and The stylistic differences between them are captured by using a Gram matrix to show color distribution and texture patterns:

[0014] in, It is a self-learning loss. This represents the nth layer of the VGG network. Indicates calculation The resulting Gram matrix.

[0015] Furthermore, a preferred approach is proposed, wherein the repair module employs a single-time-step denoising diffusion model, specifically the DDIM algorithm, to achieve image repair during the inference phase.

[0016] Furthermore, a preferred embodiment is proposed, wherein the expression for the comprehensive loss function is:

[0017] in, It is a reconstruction loss. It is perceived loss. It's a loss of style. It is the weight that balances the reconstruction loss. It is the weight of perceived loss. It is the weight of style loss. It is the weight of the self-learning loss.

[0018] Furthermore, a preferred method is proposed, wherein... It is 2. =1, It is 250. The value is 1.

[0019] Based on the same inventive concept, this invention also proposes a blind image inpainting system based on self-learning features and hierarchical similarity, the system comprising: A semantic feature extraction unit is used to automatically extract semantic features from damaged images through a self-learning module. The self-learning module is based on the Transformer architecture and is used to extract the semantic features required for repair from the undamaged areas of the damaged image without the need for mask prediction. Hierarchical similarity block insertion unit is used to introduce hierarchical similarity blocks into the self-learning module to calculate the similarity of features between consecutive layers of the Transformer. The guidance unit is used to design a self-learning semantic loss function, which guides semantic feature extraction through feature-level soft constraints; The stitching unit is used to stitch the semantic features extracted by the self-learning module with the damaged image and then input it into the repair module; The training unit is used to jointly train the self-learning module and the repair module using a comprehensive loss function. The comprehensive repair module includes reconstruction loss, perceptual loss, style loss, and self-learning semantic loss.

[0020] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a blind image restoration method based on self-learning features and hierarchical similarity according to any one of the above.

[0021] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of a blind image inpainting method based on self-learning features and hierarchical similarity as described in any of the preceding claims.

[0022] Compared with the prior art, the beneficial effects of the present invention are: The method proposed in this invention can reconstruct visually coherent and accurate images, especially when dealing with complex and irregular damaged areas, effectively maintaining the semantic consistency and structural integrity of the image. For example, for fine structures such as lip details in facial images and the outline of vehicle tires, this method can achieve more realistic and accurate reproduction, avoiding common problems in existing technologies such as texture blurring, structural distortion, or semantic disconnection.

[0023] The method proposed in this invention eliminates the traditional manual intervention stage such as mask prediction or soft masking, enabling the network to autonomously learn the semantic features required for repair from damaged images. This weak intervention mechanism not only reduces the risk of performance degradation due to mask estimation errors, but also makes the algorithm more robust and generalizable when facing unseen and unknown contamination conditions during training, such as solid color irregular damage or rectangular damage in specific regions.

[0024] Existing methods typically experience a sharp decline in repair effectiveness when the damaged area reaches 40%-60%. The hierarchical similarity blocks and self-learning feature extraction mechanism introduced in this invention can effectively capture and associate semantic information scattered across different layers of the deep network, thus providing globally consistent guidance for repairing large-area damage and effectively mitigating problems such as inconsistent color and loss of detail.

[0025] This invention uses a single-time-step diffusion model (DDIM) as the repair module. Compared with multi-step sampling schemes, it compresses the inference speed to a practical level while ensuring no significant decrease in repair performance through effective guidance of self-learning features, thus achieving a balance between high efficiency and high quality.

[0026] This invention is used for pre-binding algorithm selection and adaptive image restoration in scenarios where obtaining the restoration mask is difficult or unknown during damaged image restoration. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Fig. 1 This is a network framework diagram of the blind image inpainting method based on self-learning features and hierarchical similarity described in this invention; Fig. 2 The image shows the effect of the blind image restoration method based on self-learning features and hierarchical similarity described in this invention. Fig. 3 The image shows a comparison of the blind image restoration method based on self-learning features and hierarchical similarity described in this invention. From left to right, the images represent the damaged image, the image result processed by TransCNNHAE, the image result processed by WavNet, the image result processed by HAN, the image result processed by CoordFill, the image result processed by StrDiffusion, the image result processed by this method, and the real image. Fig. 4 This image shows the repair effect of the blind image restoration method based on self-learning features and hierarchical similarity described in this invention under unknown damage conditions. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other, and the described embodiments are only some embodiments of the present invention, not all embodiments.

[0029] Implementation Method 1: A blind image inpainting method based on self-learning features and hierarchical similarity, as described in this implementation method, includes: Semantic features are automatically extracted from damaged images through a self-learning module based on the Transformer architecture, which is used to extract the semantic features required for repair from the undamaged areas of damaged images without mask prediction. A hierarchical similarity block is introduced into the self-learning module to calculate the similarity of features between consecutive layers of the Transformer; We design a self-learning semantic loss function to guide semantic feature extraction through feature-level soft constraints. The semantic features extracted by the self-learning module are concatenated with the damaged image and then input into the repair module; A comprehensive loss function is used to jointly train the self-learning module and the repair module. The comprehensive repair module includes reconstruction loss, perceptual loss, style loss and self-learning semantic loss.

[0030] Traditional methods rely on manually defined masks (whether hard or soft) to delineate repair boundaries. Their feature extraction process is limited by human prior knowledge and cannot autonomously adapt to ever-changing damage patterns. The method proposed in this implementation utilizes the Transformer architecture's powerful modeling capabilities for long-distance dependencies and designs a self-learning module. The core principle of this module is to abandon the dependence on masks and directly and intelligently mine and extract the semantic features required for repair from undamaged areas. This makes the feature extraction process no longer based on human assumptions but driven by the data itself, fundamentally avoiding biases introduced by humans.

[0031] Simply stacking network layers is insufficient to effectively capture and utilize the potential correlations between features at different depth levels, resulting in inefficient transmission of deep semantic information and shallow detail information, and underutilization of inter-layer features. This implementation introduces a hierarchical similarity block in the self-learning module. This module works by calculating the cosine similarity of the output features of consecutive network layers in the spatial domain, mining their deep similarities through a multilayer perceptron, and finally feeding the obtained similarity features back into the network stream in a weighted summation. The principle behind this mechanism is to proactively establish dialogue and complementary channels between inter-layer features, enhancing the interconnectivity and richness of self-learned semantic features, and ensuring that feature information beneficial for repair can be effectively transmitted and enhanced within the network.

[0032] Existing restoration methods often use loss functions that directly apply to pixel-level output, creating a "hard constraint" that strongly guides the network to output results that conform to human pre-defined intentions. This easily leads to templated restoration results and a lack of diversity. This implementation proposes a self-learning semantic loss function. Its technical principle is to apply constraints at the feature level rather than the pixel level. By utilizing a pre-trained VGG network to calculate the differences between the real image and the self-learned features in multi-layer feature maps and styles (Gram matrices), a flexible and guiding constraint is provided for the feature extraction process. This "soft constraint" principle ensures that the network, while acquiring features beneficial to restoration, retains sufficient autonomous optimization space, thereby generating high-quality restoration results that are both reasonable and rich in subtle differences.

[0033] Existing methods either separate restoration from mask prediction or attempt an end-to-end approach but lack effective feature guidance, resulting in a blind restoration process or insufficient guidance information. This invention proposes a novel workflow that integrates a self-learning module with a diffusion model. The principle is to concatenate semantically rich features extracted by the self-learning module with the damaged image, using both as input to the diffusion model. This allows the diffusion model to receive not only pixel information from the damaged image during denoising and generation but also continuous high-level semantic guidance from the self-learning module, thus ensuring, in principle, a high degree of semantic and textural consistency between the restored content and the undamaged parts of the image. The single-timestep strategy is an efficiency optimization based on this effective guidance principle.

[0034] Implementation Method Two: This implementation method further defines the blind image inpainting method based on self-learning features and hierarchical similarity described in Implementation Method One. The construction of the self-learning module includes: The damaged image is encoded through an embedding layer and then fed into a Transformer block to extract semantic features; A hierarchical similarity block is inserted between every two Transformer blocks to calculate the deep similarity of features between layers; The self-learning semantic loss function is applied to the output of the self-learning module to optimize the feature extraction process.

[0035] Implementation Method 3: This implementation method further defines the blind image inpainting method based on self-learning features and hierarchical similarity described in Implementation Method 1. The construction of the hierarchical similarity block includes: Obtain features from two consecutive layers of a Transformer and Normalization is performed along the channel dimension, and features are calculated. and Cosine similarity in the spatial domain yields surface similarity. :

[0036] in, CN This indicates a channel normalization operation; Will The data is fed into a multilayer perceptron to extract deeper similarity features in the following ways:

[0037] in, MLP For a multilayer perceptron; With features Multiplication yields the latent similarity features between layers. This enhances the extraction of self-learning semantic features in the following ways:

[0038] at the same time, Added to In this process, the results are used as new features from the previous layer and then re-input into the current layer.

[0039] Implementation Method Four: This implementation method further defines the blind image inpainting method based on self-learning features and hierarchical similarity described in Implementation Method One. The calculation of the self-learning semantic loss function includes: The output features of the self-learning module and real images Input to VGG network; Compare the feature differences between shallow convolutional layers and deep fully connected layers respectively; calculate and The stylistic differences between them are captured by using a Gram matrix to show color distribution and texture patterns:

[0040] in, It is a self-learning loss. This represents the nth layer of the VGG network. Indicates calculation The resulting Gram matrix.

[0041] Implementation Method 5: This implementation method further defines the blind image inpainting method based on self-learning features and hierarchical similarity described in Implementation Method 1. The inpainting module adopts a single-time-step denoising diffusion model, specifically the DDIM algorithm, to achieve image inpainting during the inference stage.

[0042] Implementation Method Six: This implementation method further defines the blind image inpainting method based on self-learning features and hierarchical similarity described in Implementation Method One. The expression for the comprehensive loss function is:

[0043] in, It is a reconstruction loss. It is perceived loss. It's a loss of style. It is the weight that balances the reconstruction loss. It is the weight of perceived loss. It is the weight of style loss. It is the weight of the self-learning loss.

[0044] Implementation Method Seven: This implementation method further defines the blind image inpainting method based on self-learning features and hierarchical similarity described in Implementation Method One. It is 2. =1, It is 250. The value is 1.

[0045] Implementation Method Eight: A blind image inpainting system based on self-learning features and hierarchical similarity as described in this implementation method, the system comprising: A semantic feature extraction unit is used to automatically extract semantic features from damaged images through a self-learning module. The self-learning module is based on the Transformer architecture and is used to extract the semantic features required for repair from the undamaged areas of the damaged image without the need for mask prediction. Hierarchical similarity block insertion unit is used to introduce hierarchical similarity blocks into the self-learning module to calculate the similarity of features between consecutive layers of the Transformer. The guidance unit is used to design a self-learning semantic loss function, which guides semantic feature extraction through feature-level soft constraints; The stitching unit is used to stitch the semantic features extracted by the self-learning module with the damaged image and then input it into the repair module; The training unit is used to jointly train the self-learning module and the repair module using a comprehensive loss function. The comprehensive repair module includes reconstruction loss, perceptual loss, style loss, and self-learning semantic loss.

[0046] Implementation Method Nine: A computer device according to this implementation method includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a blind image restoration method based on self-learning features and hierarchical similarity according to any one of the above embodiments.

[0047] Implementation Method 10: A computer-readable storage medium according to this embodiment stores a computer program, which, when executed by a processor, performs the steps of a blind image inpainting method based on self-learning features and hierarchical similarity as described in any of the preceding embodiments.

[0048] Implementation Method 10, see below Figs. 1 to 4 This embodiment describes a specific example of the blind image inpainting method based on self-learning features and hierarchical similarity described in Embodiment 1. It also serves to explain Embodiments 2 through 7. Specifically: The proposed method in this embodiment first utilizes the Transformer's ability to model long-range dependencies to design a self-learning semantic feature acquisition module. This module automatically extracts the semantic features needed to repair damaged images from undamaged regions. Secondly, a hierarchical similarity block is introduced into this module, which effectively provides semantic features derived from surrounding layers for self-learning. Finally, leveraging the powerful image generation capabilities of the diffusion model, a single-timestep diffusion model is used as the repair module to achieve image repair through self-learning semantic features. Simultaneously, this embodiment redesigns the self-learning loss function to optimize the self-learning semantic feature extraction process in the model, employing a weakly guided approach to minimize human interference during the repair process. The overall network architecture is based on self-learning semantic features, such as... Fig. 1 As shown, it is divided into self-learning components and repair components.

[0049] Step 1: Build a self-learning module The goal of this method is to improve the blind image inpainting task by retrieving the image from a damaged input image. Reconstructing visually coherent and content-accurate images .in h , w These represent the height and width of the image, respectively. R This is a collection of images. Considering the complexity of real-world scenes, the input of damaged images is generated as follows: (1-1) in, It is a real image; noise is a visual signal of noise, not a simple assumed noise distribution. It's a binary mask indicating damaged areas in the image; 1 represents a damaged area, and 0 represents an undamaged area. ⊙ represents element-wise multiplication. Since this method primarily focuses on blind image inpainting, the mask... This module is used only for synthesizing damaged images and does not participate in the restoration process itself. In this module, the image is encoded through an embedding layer and then fed into a Transformer block to facilitate the extraction of self-learned semantic features. Hierarchical similarity blocks are interspersed between each Transformer block; these blocks compute the deep similarity of features between different layers, thereby enhancing the interconnectivity of self-learned semantic features between layers. Simultaneously, a self-learning loss is applied to the output of this segment. This is done to obtain the semantic information desired by the network itself through soft constraints rather than forced constraints, rather than information subject to artificial constraints.

[0050] Step 2: Construct hierarchical similarity blocks Most existing blind image inpainting methods incorporate a human intervention stage into the network, such as mask prediction and soft masking techniques, and use the results for inpainting. This method strives to avoid this intervention, enabling the network to learn semantic features beneficial for inpainting and those needed by the network itself. To help it more skillfully acquire the semantic features it inherently needs, this method considers implicit features beneficial for inpainting present in different layers of the Transformer. Specifically, this module utilizes features obtained from two consecutive layers of the Transformer. and Normalization is performed along the channel dimension, and their cosine similarity in the spatial domain is calculated to derive the surface similarity between these features. As shown below: (2-1) in, CN This indicates a channel normalization operation. This is used to uncover deeper similarities between layers. They are fed into a multilayer perceptron to extract deeper similarity features in the following ways: (2-2) in, MLP It is a multilayer perceptron. Ultimately, It acts similarly to a weighting factor, combining it with features Multiplication yields the latent similarity features between layers. This enhances the extraction of self-learning semantic features in the following ways: (2-3) at the same time, Added to In this process, the results are used as new features from the previous layer and then re-inputted into the current layer, which also enhances feature acquisition.

[0051] Step 3: Construct self-learning semantic loss To ensure that the semantic features acquired during the self-learning phase are useful and not irrelevant in the repair phase, this method designs a self-learning semantic loss. This loss function guides the feature extraction process to some extent while striving to avoid interference. Inspired by the excellent feature extraction capabilities of the VGG network, the results of the self-learning module are used... and The input network is then compared with the feature differences between shallow convolutional layers and deep fully connected layers. Furthermore, the role of style loss in capturing the color distribution and texture patterns of the target is utilized, while simultaneously calculating... and The stylistic differences between them are as follows: (3-1) in, It is a self-learning loss. This represents the nth layer of the VGG network. Indicates calculation The resulting Gram matrix. Unlike popular inpainting methods, this loss function does not impose constraints on pixel-level results, but only provides guidance at the feature level, thus the results are not strongly influenced by human intent. Therefore, the comprehensive loss function of this method is: (3-2) in, , and Through and O These are calculated losses, namely reconstruction loss, perceptual loss, and style loss. The formulas are as follows: (3-3) (3-4) (3-5) in, The mean, To extract features using the VGG network, It is a Gram matrix. Through and Calculated , , and It is the weight of the balancing reconstruction loss, perception loss, style loss, and self-learning loss.

[0052] Step 4: Build the repair module During the restoration process, the results generated by the self-learning module are stitched together with the damaged image, and this combined data serves as input. Leveraging the powerful generative capabilities of the diffusion model, this process employs the original diffusion model with single-time-step denoising. This approach facilitates loss calculation and results in faster inference speeds compared to multi-time-step diffusion models. The complete training process is summarized in Algorithm 1.

[0053]

[0054] In the actual design of this implementation, there are two main coordination difficulties: First, the input adaptation problem between the Transformer self-learning module and the diffusion model. The self-learning semantic features extracted by the Transformer module need to be concatenated with the damaged image in the channel dimension before being input into the diffusion model. Since the feature dimensions and semantic granularity of the two differ, if feature alignment cannot be achieved, the diffusion model will be unable to effectively utilize the guidance information of the self-learning features during the denoising process, and may even introduce noise interference. To address the problem of the difficulty in directly adapting the semantic features extracted by the Transformer self-learning module to the input of the diffusion model, this implementation adopts a feature concatenation strategy and uses this combined data as the input of the repair module. To achieve effective collaboration, the input structure of the original diffusion model, such as DDIM, was specifically modified to make it compatible with and utilize additional semantic feature channels. Simultaneously, by redesigning the training loss function of the diffusion model, it was made to work in conjunction with the self-learning loss function, ensuring that the self-learning features can be effectively decoded and utilized during the denoising process of the diffusion model, avoiding noise interference or guidance failure caused by feature mismatch.

[0055] To address the computational overhead of the complex combination of "Transformer + hierarchical similarity block + diffusion model," this implementation method employs a meticulous trade-off design. On one hand, while the introduction of the hierarchical similarity block slightly increases computation, its inter-layer feature enhancement effect is significant, avoiding the need for more complex repair modules or multiple iterations due to insufficient feature extraction, thus improving overall efficiency. On the other hand, using a single-timestep diffusion model for inference, rather than traditional multi-step sampling, while placing higher demands on the model's one-time generation capability, combined with powerful self-learning feature guidance, achieves an order-of-magnitude improvement in inference speed with minimal accuracy loss. Table 1 shows the experimental results demonstrating the effectiveness of the method. ↓ and ↑ represent lower values / higher performance.

[0056] Table 1. Model performance and inference speed at different time steps.

[0057] The method proposed in this invention can be tested on public image datasets and can be plug-and-play optimized for some existing methods. The specific repair effects of this method are as follows: Fig. 2As shown, the left side of each image group represents the damaged image, the middle side represents the result of the self-learning module, providing corresponding features for the restoration module, and the right side represents the restoration result. To ensure fair comparison and demonstrate the superiority of the proposed method, this invention is compared with publicly available image restoration algorithms in existing literature on public datasets such as CelebA-HQ and Places2. Furthermore, the datasets are trained using masks with proportions between 0% and 60%, rather than splitting these masks for training. In our proposed method, the self-learning module is carefully constructed by stacking Transformers containing a dual-head self-attention mechanism, and the restoration module uses the original DDIM algorithm. To train the model designed for this method, the Adam optimizer is used, with configured parameters... and and make the learning rate The training and testing images were both resized to 256×256. Additionally, the weights in the loss function were adjusted. , , and The values ​​are set to 2, 1, 250, and 1 respectively. This setting ensures that the method can repair images with high quality while balancing the quality of self-learned features acquired during the self-learning process.

[0058] Fig. 3 The image presented here compares the proposed method with state-of-the-art methods. The images generated by this method exhibit high visual credibility and contain plausible content. Notably, in terms of fine detail... Fig. 3 (The part enclosed in red in the image) The existence of self-learned semantic features enables images to be restored more realistically in terms of both texture and semantic nuances. For example, in the generation of the human face lips and car tires shown in the image, this method shows meticulous attention to both texture and semantic nuances, resulting in results that are more consistent with the features of the original image.

[0059] Fig. 4 This paper demonstrates a comparison between the proposed method and state-of-the-art methods under unknown damage conditions, including irregular solid-color damage and rectangular damage in a specific area. The results show that, when faced with unknown contamination conditions, the proposed method outperforms other methods in terms of image quality and visual impact, exhibiting a higher degree of robustness. This is particularly evident in the subtle differences in structure and semantics, such as the realistic reproduction of human fingers and the intricate details between train carriages.

[0060] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Fig. 1 One or more processes and / or boxes Fig. 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Fig. 1 One or more processes and / or boxes Fig. 1 The function specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Fig. 1 One or more processes and / or boxes Fig. 1 The steps of the function specified in one or more boxes.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit its protection scope. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading this disclosure, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the published pending claims.

Claims

1. A blind image inpainting method based on self-learning features and hierarchical similarity, characterized in that, The method includes: Semantic features are automatically extracted from damaged images through a self-learning module based on the Transformer architecture, which is used to extract the semantic features required for repair from the undamaged areas of damaged images without mask prediction. A hierarchical similarity block is introduced into the self-learning module to calculate the similarity of features between consecutive layers of the Transformer; We design a self-learning semantic loss function to guide semantic feature extraction through feature-level soft constraints. The semantic features extracted by the self-learning module are concatenated with the damaged image and then input into the repair module; A comprehensive loss function is used to jointly train the self-learning module and the repair module. The comprehensive repair module includes reconstruction loss, perceptual loss, style loss and self-learning semantic loss.

2. The blind image inpainting method based on self-learning features and hierarchical similarity according to claim 1, characterized in that, The construction of the self-learning module includes: The damaged image is encoded through an embedding layer and then fed into a Transformer block to extract semantic features; A hierarchical similarity block is inserted between every two Transformer blocks to calculate the deep similarity of features between layers; The self-learning semantic loss function is applied to the output of the self-learning module to optimize the feature extraction process.

3. The blind image inpainting method based on self-learning features and hierarchical similarity according to claim 1, characterized in that, The construction of the hierarchical similarity blocks includes: Obtain features from two consecutive layers of a Transformer and Normalization is performed along the channel dimension, and features are calculated. and Cosine similarity in the spatial domain yields surface similarity. : in, CN This indicates a channel normalization operation; Will The data is fed into a multilayer perceptron to extract deeper similarity features in the following ways: in, MLP For a multilayer perceptron; With features Multiplication yields the latent similarity features between layers. This enhances the extraction of self-learning semantic features in the following ways: at the same time, Added to In this process, the results are used as new features from the previous layer and then re-input into the current layer.

4. The blind image inpainting method based on self-learning features and hierarchical similarity according to claim 1, characterized in that, The calculation of the self-learning semantic loss function includes: The output features of the self-learning module and real images Input to VGG network; Compare the feature differences between shallow convolutional layers and deep fully connected layers respectively; calculate and The stylistic differences between them are captured by using a Gram matrix to show color distribution and texture patterns: in, It is a self-learning loss. This represents the nth layer of the VGG network. Indicates calculation The resulting Gram matrix.

5. The blind image inpainting method based on self-learning features and hierarchical similarity according to claim 1, characterized in that, The repair module employs a single-time-step denoising diffusion model, specifically the DDIM algorithm, to perform image repair during the inference phase.

6. The blind image inpainting method based on self-learning features and hierarchical similarity according to claim 1, characterized in that, The expression for the comprehensive loss function is: in, It is a reconstruction loss. It is perceived loss. It's a loss of style. It is the weight that balances the reconstruction loss. It is the weight of perceived loss. It is the weight of style loss. It is the weight of the self-learning loss.

7. The blind image inpainting method based on self-learning features and hierarchical similarity according to claim 1, characterized in that, The It is 2. =1, It is 250. The value is 1.

8. A blind image inpainting system based on self-learning features and hierarchical similarity, characterized in that, The system includes: A semantic feature extraction unit is used to automatically extract semantic features from damaged images through a self-learning module. The self-learning module is based on the Transformer architecture and is used to extract the semantic features required for repair from the undamaged areas of the damaged image without the need for mask prediction. Hierarchical similarity block insertion unit is used to introduce hierarchical similarity blocks into the self-learning module to calculate the similarity of features between consecutive layers of the Transformer. The guidance unit is used to design a self-learning semantic loss function, which guides semantic feature extraction through feature-level soft constraints; The stitching unit is used to stitch the semantic features extracted by the self-learning module with the damaged image and then input it into the repair module; The training unit is used to jointly train the self-learning module and the repair module using a comprehensive loss function. The comprehensive repair module includes reconstruction loss, perceptual loss, style loss, and self-learning semantic loss.

9. A computer device, characterized in that: The device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a blind image inpainting method based on self-learning features and hierarchical similarity according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of a blind image inpainting method based on self-learning features and hierarchical similarity as described in any one of claims 1-7.

Images