Image restoration method, device, electronic device and computer-readable storage medium

Abstract

The present disclosure relates to the field of image restoration technology, and provides an image restoration method, device, electronic device and medium. The method can perform image structure extraction processing on an occluded image to obtain the structural features of the occluded image; perform image encoding processing on the occluded image to obtain the visual features of the occluded image; perform text encoding processing on the prompt text of the occluded image to obtain the semantic features of the prompt text; perform decoding processing on the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text to obtain the pixel-level features corresponding to the occluded image and the prompt text; perform denoising processing on the initial noise map based on the visual features of the occluded image, the pixel-level features, and the semantic features of the prompt text through a diffusion model. The present disclosure overcomes the shortcomings of the prior art by utilizing the structural features, visual features of the image and the semantic features of the prompt text, and significantly improves the effect of image restoration.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image restoration, and in particular to an image restoration method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of Internet technology, the field of image processing has also been significantly improved, especially in image restoration. Image restoration algorithms can effectively fill in missing areas in images according to user needs through intelligent algorithms, aiming to maintain the structural rationality and texture consistency of the restored areas. These algorithms have shown their potential application value when dealing with large missing areas and high-resolution images. However, existing image restoration models still face multiple challenges. First, the limited receptive field makes it difficult for the model to learn semantically consistent textures in large missing areas or high-resolution images. This limitation causes the model to be unable to effectively capture global context information when restoring the image, thus affecting the naturalness and coherence of the restoration effect. Second, existing models lack an understanding of the overall structure, which is particularly prominent when restoring key edges and wireframes in high-resolution weak-texture scenes. Due to the lack of in-depth analysis of the overall structure of the image, the model may not be able to accurately reconstruct important details during the restoration process, resulting in unsatisfactory restoration results, or even obvious artifacts or inconsistent textures. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide an image restoration method, an apparatus, an electronic device, and a computer-readable storage medium to solve the technical problem of inaccurate image restoration results in the prior art.

[0004] According to a first aspect of an embodiment of the present disclosure, an image restoration method is provided, comprising: obtaining an occluded image and a prompt text of the occluded image; performing image structure extraction processing on the occluded image to obtain structural features of the occluded image; performing image encoding processing on the occluded image to obtain visual features of the occluded image; performing text encoding processing on the prompt text of the occluded image to obtain semantic features of the prompt text; decoding processing on the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text to obtain pixel-level features corresponding to the occluded image and the prompt text; denoising an initial noise map based on the visual features of the occluded image, the pixel-level features, and the semantic features of the prompt text by a diffusion model to obtain a target restoration image corresponding to the occluded image.

[0005] According to a second aspect of the disclosed embodiments, an image restoration device is provided, comprising: an acquisition module for acquiring an occluded image and a prompt text of the occluded image; a structural feature extraction module for performing image structure extraction processing on the occluded image to obtain the structural features of the occluded image; a visual feature encoding module for performing image encoding processing on the occluded image to obtain the visual features of the occluded image; a semantic feature encoding module for performing text encoding processing on the prompt text of the occluded image to obtain the semantic features of the prompt text; a decoding module for decoding the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text to obtain the pixel-level features corresponding to the occluded image and the prompt text; and a denoising module for denoising an initial noise map based on the visual features, pixel-level features, and semantic features of the prompt text through a diffusion model to obtain a target restoration image corresponding to the occluded image.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the disclosed embodiment has the following beneficial effects: the disclosed embodiment can better understand the overall structure of the image by extracting structural features of the occluded image. This process ensures that the repaired image is structurally reasonable and avoids common artifacts and unnatural repair results. Visual feature encoding of the occluded image can capture the details and texture information in the image. This meticulous feature extraction helps to maintain the original style and details of the image during the repair process and improves the visual quality of the repaired image. By encoding the semantic features of the prompt text, the text information can be effectively integrated into the image repair process. This combination of semantic information makes the repair result more in line with the needs of users and improves the intelligent level of repair. The structural features, visual features and semantic features are converted into pixel-level features through decoding processing, so that each pixel in the repair process can be accurately processed. This feature ensures the consistency and coherence of the repaired image in details. Denoising the initial noise map using a diffusion model can effectively reduce noise interference in the repair process, improve the clarity and quality of the repaired image, and make the repair result more realistic and in line with the visual perception of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0011] Figure 2 is a flowchart of an image restoration method provided by an embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram of an application framework of an image restoration method provided by an embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram of the structure of a submodule in a diffusion module provided in an embodiment of the present disclosure;

[0014] Figure 5 is a block diagram of an image restoration device provided by an embodiment of the present disclosure;

[0015] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0017] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0018] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.

[0019] like Figure 1As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0020] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0021] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0022] The server 105 may be a server that provides various services. For example, the server 105 may obtain the occluded image and the prompt text of the occluded image from the first terminal device 101 (or the second terminal device 102 or the third terminal device 103); perform image structure extraction processing on the occluded image to obtain the structural features of the occluded image; perform image encoding processing on the occluded image to obtain the visual features of the occluded image; perform text encoding processing on the prompt text of the occluded image to obtain the semantic features of the prompt text; perform decoding processing on the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text to obtain the pixel-level features corresponding to the occluded image and the prompt text; perform denoising processing on the initial noise image based on the visual features of the occluded image, the pixel-level features, and the semantic features of the prompt text through a diffusion model to obtain the target repaired image corresponding to the occluded image.

[0023] In some embodiments, the image restoration method provided by the embodiment of the present invention is generally executed by the server 105, and accordingly, the image restoration device is generally disposed in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the image restoration method provided by the embodiment of the present invention is not limited to being executed on the server side.

[0024] The image restoration method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 2 is a flowchart of an image restoration method provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0026] like Figure 2 As shown, the image restoration method includes steps S210 to S260.

[0027] In step S210, the blocked image and the prompt text of the blocked image are obtained.

[0028] Step S220: performing image structure extraction processing on the occluded image to obtain structural features of the occluded image.

[0029] In step S230, image coding processing is performed on the occluded image to obtain visual features of the occluded image.

[0030] In step S240, text encoding is performed on the prompt text of the blocked image to obtain semantic features of the prompt text.

[0031] In step S250, the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text are decoded to obtain pixel-level features corresponding to the occluded image and the prompt text.

[0032] In step S260, the initial noise image is denoised by a diffusion model based on visual features, pixel-level features, and semantic features of the prompt text of the occluded image to obtain a target repaired image corresponding to the occluded image.

[0033] This method can better understand the overall structure of the image by extracting structural features from the occluded image. This process ensures that the repaired image is structurally reasonable and avoids common artifacts and unnatural repair results. Visual feature encoding of the occluded image can capture the details and texture information in the image. This meticulous feature extraction helps to maintain the original style and details of the image during the repair process and improves the visual quality of the repaired image. By encoding the semantic features of the prompt text, the text information can be effectively integrated into the image repair process. This combination of semantic information makes the repair result more in line with user needs and improves the intelligent level of repair. The structural features, visual features and semantic features are converted into pixel-level features through decoding processing, so that each pixel in the repair process can be accurately processed. This feature ensures the consistency and coherence of the repaired image in details. Using the diffusion model to denoise the initial noise map can effectively reduce the noise interference in the repair process, improve the clarity and quality of the repaired image, and make the repair result more realistic and in line with the visual perception of the human eye.

[0034] In some embodiments of the present disclosure, an occluded image may refer to an image annotated by a user according to his or her actual needs. Such an image usually contains information that some areas are missing due to occlusion, damage or other reasons. When submitting an occluded image, the user can directly mark the area that needs to be repaired on the image. These annotations can be simple rectangular boxes, free-form polygons, or graffiti to indicate the missing part. The user can attach text descriptions to describe the characteristics or background information of the annotated area. For example, the user can describe the content, color, texture, etc. originally contained in the area to help the algorithm better understand the need for repair. The user can selectively block certain parts of the image while retaining other parts. This flexible annotation method enables the user to express his or her repair intention more accurately. In this way, the user not only provides the image to be repaired, but also clarifies the expected repair effect, thereby providing a valuable information basis for subsequent image processing. The prompt text may refer to the description content of the image that the user actually wants to generate for the occluded image. These text information can provide important semantic guidance for the image repair process, specifically including: the user can describe the content of the occluded area in natural language. For example, "there should be a blue sky and white clouds here" or "the missing part is a red car". Such descriptions can help the algorithm understand the expected characteristics of the repaired area. Users can add requirements for image style or emotion in the prompt text, such as: "I hope the repaired image will present a warm and sunny atmosphere" or "I hope the repaired image will have an artistic feel." This information can guide the algorithm to consider the overall visual effect during the repair process. Users can also put forward some specific repair requirements, such as details in color, shape, texture, etc. This specific guidance helps ensure that the repair results are more in line with user expectations. By combining the occluded image with the prompt text, users can express their repair requirements more effectively, and the algorithm can understand these requirements more accurately, thereby providing more accurate and satisfactory results during the image repair process. This interactive annotation and description mechanism not only improves the repair effect, but also enhances the user's sense of participation and satisfaction.

[0035] In some embodiments of the present disclosure, performing structural extraction processing on the occluded image to obtain the structural features of the occluded image includes performing structural extraction processing on the occluded image through a control network to obtain the structural features of the occluded image. For example, the occluded image is input into the Controlnet control network to extract the overall structural information of the image and obtain the structural features of the occluded image. These features include information such as shape, contour, edge and spatial layout in the image, which can help the subsequent image restoration process understand the basic structure of the image, so as to better fill the missing area. ControlNet is an advanced deep learning model specifically used to control and guide the image generation and restoration process. Its main function is to extract useful features and guide subsequent processing through the input image or condition information. Specifically, the occluded image is input into ControlNet. The image contains one or more occluded areas, and these areas may visually affect the overall structure. ControlNet contains multiple convolutional layers and feature extraction modules, which can identify edges, corners and other structural features in the image. Through the step-by-step processing of these layers, the model can extract the hierarchical features of the image. ControlNet can analyze images at different scales, ensuring that both global information and local details are captured when extracting features. This multi-scale processing method helps to understand the structure of the image more comprehensively. After processing, ControlNet outputs a feature map representing the structural features of the occluded image. These feature maps contain information about the overall layout of the image and can be used to guide the subsequent image restoration process. Through accurate structural feature extraction, the restored image can avoid common artifacts and unnatural effects, thereby improving the quality and realism of the final image. Structural features can be combined with subsequently extracted visual features to form a comprehensive feature representation. This comprehensive feature will help generate more accurate and natural restoration images. By performing structural extraction processing on the occluded image and using the ControlNet control network to extract the overall structural information, a solid foundation can be effectively provided for image restoration. This process not only improves the accuracy and consistency of image restoration.

[0036] In some embodiments of the present disclosure, performing image coding processing on the occluded image to obtain the visual features of the occluded image includes performing image coding processing on the occluded image through an image encoder to obtain the visual features of the occluded image. For example, the occluded image is input into an image encoder (such as a VIT image encoder), and the visual detail features of the image are extracted to obtain basic image features, that is, the visual features of the occluded image. These features include color, texture, shape and other detail information, which help to retain the visual consistency and realism of the image during the image restoration process. The image encoder is a model in deep learning, which is responsible for converting the input image into a high-dimensional feature vector. Taking the Vision Transformer (VIT) as an example, the occluded image is input into the image encoder. The image encoder processes the entire image, including the occluded part. VIT first divides the input image into multiple small patches (patches), and the size of each small patch is usually fixed. In this way, the image can be converted into a series of image patches of the same size, which is convenient for subsequent processing. Each image patch will be mapped to a high-dimensional space to form an embedded representation. This process can capture the local features of each patch. VIT uses a self-attention mechanism to capture the relationship between image patches. By calculating the similarity between patches, the model can effectively understand the global context information of the image. This mechanism can help the model better understand the relationship between the occluded area and the surrounding area. By stacking multiple encoding layers, VIT can gradually extract more complex visual features. These features include basic color and texture information, as well as higher-level shape and structural features. After encoding, VIT outputs a high-dimensional feature vector that represents the visual features of the occluded image. These feature vectors contain rich information about the image and can provide support for subsequent image restoration. By using visual features, the restored image can better maintain the visual style and details of the original image and avoid unnatural effects. Visual features can be combined with previously extracted structural features to form a comprehensive feature representation. This comprehensive feature can provide more comprehensive information for the restoration algorithm, thereby improving the restoration effect. By performing image encoding on the occluded image and extracting visual features using an image encoder (such as VIT), rich detail information can be effectively provided for image restoration. This process not only enhances the accuracy and naturalness of image restoration.

[0037] In some embodiments of the present disclosure, performing text encoding processing on the prompt text of the occluded image to obtain the semantic features of the prompt text includes: performing text encoding processing on the prompt text of the occluded image through a text encoder to obtain the semantic features of the prompt text. For example, the prompt text of the occluded image is input into a text encoder (such as a CLIP text encoder), and the semantic information of the text is extracted to obtain the semantic features of the prompt text. These features can help understand the information conveyed by the text, thereby providing context and guidance for image restoration. The text encoder is a model for converting natural language text into a high-dimensional vector representation. Taking the CLIP (Contrastive Language-Image Pre-training) text encoder as an example, the prompt text is input into the CLIP text encoder. The prompt text can be a description of the content of the occluded part, the desired restoration effect, or other related information. CLIP first performs word segmentation processing on the input text, decomposing the text into words or subwords for subsequent processing. Each word will be mapped to a high-dimensional space to form word embeddings. These embeddings can capture the semantics and context information of the words. CLIP uses the Transformer architecture to process text sequences, which can effectively capture the relationship between words. Through the self-attention mechanism, the overall semantic structure of the text can be understood. By stacking multiple encoding layers, CLIP can extract more complex semantic features. These features include not only word meanings, but also the overall meaning of the sentence. After encoding, CLIP outputs a high-dimensional feature vector that represents the semantic features of the prompt text. These features contain deep information about the text and can provide support for subsequent image restoration. By combining the semantic features of the prompt text, the restored image can better meet the user's expectations and enhance the relevance between the image and the text. Semantic features can be combined with previously extracted visual features and structural features to form a comprehensive feature representation. This comprehensive feature can provide more comprehensive information for the restoration algorithm, thereby improving the restoration effect. By performing text encoding on the prompt text of the occluded image and extracting semantic features using a text encoder (such as CLIP), the necessary contextual information can be effectively provided for image restoration. This process not only enhances the intelligence and personalization of image restoration.

[0038] In some embodiments of the present disclosure, decoding the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text to obtain the pixel-level features corresponding to the occluded image and the prompt text includes fusing the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text through the cross-attention mechanism in the Transformer decoder to obtain the pixel-level features corresponding to the occluded image and the prompt text. For example, the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text are input into the Transformer decoder (such as DeiT), and fused through the cross-attention mechanism to generate pixel-level features consistent with the text description, so that the image restoration algorithm can take into account the structure and semantic information of the image when filling the occluded area, thereby improving the accuracy and naturalness of the restoration. The Transformer decoder (such as DeiT, Data-efficient Image Transformer) is a model for converting a feature vector into a pixel-level representation of an image. Specifically, the structural features come from the structural features of the occluded image, which usually represent the overall layout and shape information of the image. The visual features come from the visual features of the occluded image, including detail information such as color and texture. Semantic features come from the semantic features of the prompt text, providing contextual information of the text description. The structural features, visual features, and semantic features are properly processed to ensure that they can be effectively fused in the decoder. The cross-attention mechanism is the core part of the Transformer decoder, which can effectively fuse features from different sources. For each input feature, the decoder calculates the attention weights with other features. The decoder can focus on the relationship between structural features, visual features, and semantic features.

[0039] For example, the decoder generates a query, key, and value for each feature: the query comes from the feature of the current decoding step. The key and value come from structural features, visual features, and semantic features, respectively. By calculating the attention weights, the decoder can determine which features are more important in generating pixel-level outputs. Finally, the decoder weights these features to form a fused feature representation. After the cross-attention mechanism, the pixel-level features generated by the decoder reflect the overall structure, detailed information, and semantic content of the occluded image. The decoder finally outputs a high-dimensional feature map, which can be directly used to generate the restored image. Each pixel in the feature map corresponds to the input prompt text and the features of the occluded image. The pixel-level features generated in this way can ensure that the restored image is not only visually consistent, but also meets the user's expectations and descriptions.

[0040] In some embodiments of the present disclosure, the initial noise image is denoised based on the visual features, pixel-level features, and semantic features of the prompt text of the occluded image through a diffusion model to obtain a target repair image corresponding to the occluded image, including: text and image alignment based on the initial noise image and the semantic features of the prompt text to obtain a first fusion feature; fusion based on the pixel-level features and the first fusion feature to obtain a second fusion feature; cross-attention based on the visual features of the occluded image and the second fusion feature through an image cross-attention mechanism to obtain a third fusion feature; the third fusion feature is processed through a feedforward network to obtain a target feature, and a target repair image corresponding to the occluded image is generated based on the target feature. In this embodiment, the diffusion model uses Stable Diffusion as a pre-trained model based on the basic model, and the main network uses a U-Net structure, which can effectively capture the local and global features of the image. One or more submodules can be deployed in the diffusion model, each submodule adopts the same structure, and the semantic features, visual features, and pixel-level features are fused into the network using a cross-attention mechanism to guide the image generation process.

[0041] Specifically, during the training of the diffusion model, the initial noise map is randomly generated as the starting point for denoising. The goal of the model is to gradually remove the noise and gradually transform it into the target image. For example, the semantic information extracted by the text encoder. The similarity between the initial noise map and the prompt text is calculated using the text-image alignment mechanism to generate the first fused feature. This process ensures that the generated image matches the prompt text semantically. The first fused feature is fused with the pixel-level feature to generate the second fused feature. This feature combines the semantic information of the text and the specific details of the image, providing rich information for subsequent image generation. Through the cross-attention mechanism, the relationship between the visual features of the occluded image and the second fused feature can be dynamically focused on during the generation process. This mechanism allows the model to consider the mutual influence of the two when generating each pixel to generate an image that is more consistent with visual and semantic consistency. The third fused feature is further processed by the feedforward network to extract the target feature. This feature will be used to generate the final inpainted image. Based on the target feature, the target inpainted image corresponding to the occluded image is generated. This process involves decoding the target feature into image pixels, usually using a U-Net structure to effectively capture local and global features. In this embodiment, the diffusion model uses Stable Diffusion as the basic pre-training model. The model can generate high-quality images through a multi-level denoising process. The main network adopts a U-Net structure, which can process image features at different resolutions to ensure the retention of details and contextual information. One or more submodules can be deployed in the diffusion model, each of which adopts the same structure, and uses a cross-attention mechanism to fuse semantic features, visual features, and pixel-level features into the network to guide the image generation process. The target repair image can be efficiently generated by combining the visual features, pixel-level features, and semantic features of the occluded image with the prompt text through the diffusion model. This process ensures that the generated image is consistent with the user's expectations both visually and semantically through multi-level feature processing and cross-attention mechanisms.

[0042] In some embodiments of the present disclosure, text and image alignment are performed based on the semantic features of the initial noise map and the prompt text to obtain the first fusion feature, including: processing the initial noise map through a self-attention mechanism to obtain the structural features of the initial noise map; and performing text and image alignment based on the structural features of the initial noise map and the semantic features of the prompt text through a text cross-attention mechanism to obtain the first fusion feature.

[0043] In this embodiment, when there is one submodule in the diffusion model, the initial noise map is input into the self-attention module in the submodule. When there are multiple submodules in the diffusion model, the output of the previous submodule is input into the attention module in the next submodule. When the next submodule is the first submodule, the output of the previous submodule is the initial noise map. Specifically, the output of the previous submodule is input into the self-attention module to enhance the model's understanding of the internal structure of the image. The semantic features extracted by the text encoder are combined with the output of the self-attention module, and the semantic information of the text is aligned with the image features through the cross-attention mechanism to ensure the consistency of the generated content with the text description.

[0044] In some embodiments of the present disclosure, the second fused feature is obtained by fusing the pixel-level feature and the first fused feature, including: processing the pixel-level feature through a learnable cross-attention mechanism to obtain a learnable cross-pixel-level feature; performing nonlinear transformation processing on the learnable cross-pixel-level feature through a multi-layer perceptron to obtain a nonlinearly transformed pixel-level feature; performing cross-attention processing on the nonlinearly transformed pixel-level feature and the first fused feature through pixel-level cross-attention to obtain the second fused feature. For example, the pixel-level feature is a fine-grained feature obtained by extracting features from each pixel of the image, which usually contains rich local information. The first fused feature is processed by the self-attention mechanism and the cross-attention mechanism, which integrates the structural information of the initial noise image and the semantic information of the prompt text. The pixel-level feature is processed by the learnable cross-attention mechanism to capture the relationship between the pixel-level features and effectively align with the first fused feature. The pixel-level feature is linearly transformed to generate a query, a key, and a value: the query represents the degree of attention of the current pixel to other pixels. The key represents the information of other pixels. The value contains the feature information of each pixel. The similarity between the query and the key is calculated to obtain the attention weight. These weights are then applied to the values ​​to generate learnable cross-pixel features. After being processed by the learnable cross-attention mechanism, the learnable cross-pixel features obtained contain rich local information and can reflect the relationship between different pixels in the image. The learnable cross-pixel features are nonlinearly transformed. MLP consists of multiple fully connected layers, which can introduce nonlinear features, allowing the model to learn more complex feature representations. The learnable cross-pixel features are input into the multi-layer perceptron, and after a series of linear transformations and activation functions (such as ReLU, Sigmoid, etc.), nonlinearly transformed pixel features are generated. After being processed by MLP, the nonlinearly transformed pixel features are more expressive and provide richer information for subsequent feature fusion. Through the pixel-level cross-attention mechanism, the nonlinearly transformed pixel features are cross-attended with the first fused features to further enhance the expressiveness of the features. The nonlinearly transformed pixel features and the first fused features are linearly transformed respectively to generate new queries, keys, and values: the query comes from the nonlinearly transformed pixel features. The key and value come from the first fused feature. The similarity between the query and the key is calculated to obtain the attention weight. These weights are then applied to the values ​​to generate the second fused feature. After pixel-level cross-attention processing, the second fusion feature obtained combines the fine-grained information of the pixel-level feature and the semantic information of the first fusion feature, further improving the quality and consistency of the generated image. The second fusion feature enhances the model's ability to understand the image by combining information at different levels (pixel-level features and first fusion features), making the generated image more detailed and accurate.This fusion process ensures that the generated image is consistent with the prompt text and the original image features in both visual and semantic aspects, thereby improving the overall quality of the generated image. The pixel-level features are processed through a learnable cross-attention mechanism, and then cross-processed with the first fused features through a multi-layer perceptron nonlinear transformation. The second fused features finally generated provide strong support for the subsequent image restoration and generation process. This process ensures the consistency of the generated image in details and semantics, and improves the effect of image restoration.

[0045] In some embodiments of the present disclosure, the above method also includes: performing convolution processing on pixel-level features to obtain pixel-level enhanced features; upsampling the pixel-level enhanced features, and generating a reconstructed line drawing of the occluded image based on the upsampling processing results.

[0046] For example, the pixel-level features output by the Transformer decoder are processed through a convolutional network to enhance the expressiveness of the features. The resolution of the feature map is increased through upsampling technology, and the reconstructed line drawing of the occluded image is predicted, which helps the model understand the image structure at the global level and provides a basis for generating images consistent with the text description. For example, the fine-grained features obtained after the pixel-level features are processed by the self-attention mechanism and the cross-attention mechanism contain rich local information. The expressiveness of the pixel-level features is enhanced through convolution operations to extract deeper feature information. Convolution can effectively capture local patterns and features, allowing the model to have a deeper understanding of the image. The pixel-level features are input into a convolutional neural network (CNN) and processed through multiple convolutional layers. Each convolutional layer usually includes: the convolution kernel performs a convolution operation on the input features to extract local features. Activation functions such as ReLU add nonlinear features. Batch normalization (optional) accelerates the training process and improves stability. After convolution processing, the obtained pixel-level enhanced features contain richer local information and higher-level feature representations, which lays the foundation for subsequent upsampling processing. Through convolution processing, the edge, texture and other information in the image can be better captured, thereby improving the expressiveness of the features. The enhanced features can help the model better understand the image structure. The resolution of the feature map is increased through upsampling technology, so that the generated feature map can better match the size of the original image, providing higher details for generating the reconstructed line drawing. Common upsampling methods include: Bilinear interpolation adds pixels to the feature map through linear interpolation. Transposed convolution (also known as deconvolution) upsamples the feature map through the learned convolution kernel, which can introduce more detailed information. After upsampling, the resolution of the obtained feature map is improved, which can better reflect the details and structure of the image. The reconstructed line drawing of the occluded image is generated based on the results of the upsampling process. The line drawing usually contains the main structure and contour information of the image, providing a basis for subsequent image generation. The upsampled feature map is input into a line drawing generation network, and a convolution layer is usually used to extract the main edge and contour information. The output feature is mapped to the pixel value range of the line drawing (usually 0 to 1) through a specific activation function (such as Sigmoid). The resulting reconstructed line drawing contains the main structural information of the occluded image, which helps the model understand the overall layout of the image at a global level. The reconstructed line drawing provides the model with a global understanding of the image structure, so that the subsequently generated images can better conform to the overall characteristics of the original image. The generation of the line drawing provides a clear structural foundation for the model, which helps to generate images consistent with the text description and ensure the accuracy and consistency of the generated content. Convolution processing is performed on pixel-level features to enhance the expressiveness of the features; then the resolution of the feature map is increased through upsampling technology, and finally a reconstructed line drawing of the occluded image is generated.This process ensures that the model understands the image structure at a global level, providing a solid foundation for generating images that are consistent with the text description. Through this series of processing, the model can better capture the details and structural information of the image, thereby improving the quality and accuracy of image generation.

[0047] In some embodiments of the present disclosure, the above-mentioned image restoration method is applied to an image restoration model. The image restoration model includes a control network, an image encoder, a text encoder, a decoder, a convolutional network, and a diffusion model. Figure 3 , the occluded image and text description (i.e. the above-mentioned prompt text) are input into the image restoration model. It should be noted that the text description is associated with the content of the occluded image, providing accurate semantic information for subsequent feature extraction and image reconstruction. The occluded image is input into the Controlnet control network to extract the overall structural information of the image and obtain the image control features, i.e. the above-mentioned structural features. The occluded image is input into the image encoder (such as the VIT image encoder) to extract the visual detail features of the image and obtain the basic image features, i.e. the above-mentioned visual features. The text description is input into the text encoder (such as the CLIP text encoder) to extract the semantic information of the text and obtain the text features, i.e. the above-mentioned semantic features. The text features, basic image features, and image control features are input into the Transformer decoder (such as DeiT) and fused through the cross-attention mechanism to generate pixel-level features consistent with the text description. The pixel-level features output by the Transformer decoder are processed through a convolutional network to enhance the expressive power of the features. The resolution of the feature map is increased by upsampling technology to predict the reconstructed line drawing of the occluded image. This step helps the model understand the image structure at the global level and provides a basis for generating images consistent with the text description.

[0048] The structure of the diffusion model above uses Stable Diffusion as the basic pre-training model, and the main network uses the U-Net structure, which can effectively capture the local and global features of the image. The diffusion model has multiple submodules, each of which uses the same structure and uses the cross-attention mechanism to fuse text features, basic image features, and pixel-level features into the network to guide the image generation process. The specific submodule structure is as follows Figure 4. In this embodiment, the initial noise map is superimposed on the occluded image to generate an initial initial noise map. The diffusion module may include one or more submodules, and the implementation process of the diffusion model may be to input the output of the previous submodule into the self-attention module to enhance the model's understanding of the internal structure of the image. The text features extracted by the text encoder are combined with the output of the self-attention module, and the semantic information of the text is aligned with the image features through the cross-attention mechanism to ensure the consistency of the generated content with the text description. The pixel-level features are input into the learnable cross-attention module, and then nonlinearly transformed through the MLP (multi-layer perceptron), and the transformed features are integrated into the submodule through the cross-attention module to further fuse the text and image features at the pixel level. The basic image features are fused into the network through cross-attention to strengthen the fusion of image features and ensure that the generated image is consistent with the style and content of the original image. The output features from the image cross-attention module are received, and the fused features are further processed through a feedforward network, which usually contains two layers of linear transformations with a ReLU activation function in the middle. The fused features are further processed to provide support for generating consistent images. The overall model starts with an initial noise map, gradually reduces the noise, refines the image details, and gradually builds a clear image until the noise is completely removed. After multiple iterations of the model, the completed image is finally output. The final output image should be consistent with the text description and visually match the style and content of the original image. Through the collaborative work of these sub-modules, the diffusion model can effectively utilize the information of the text description and the occluded image to gradually build a completed image that is consistent with the original image and has complete content. Each sub-module ensures the deep fusion of image features and text features during the generation process through a specific attention mechanism and network structure, thereby improving the accuracy and naturalness of image completion.

[0049] Figure 5 It is a block diagram of an image restoration device provided by an embodiment of the present disclosure.

[0050] like Figure 5 As shown, the image restoration device 500 includes an acquisition module 510, a structural feature extraction module 520, a visual feature encoding module 530, a semantic feature encoding module 540, a decoding module 550 and a denoising module 560.

[0051] Specifically, the acquisition module 510 is used to acquire the blocked image and the prompt text of the blocked image.

[0052] The structural feature extraction module 520 is used to perform image structure extraction processing on the occluded image to obtain the structural features of the occluded image.

[0053] The visual feature encoding module 530 is used to perform image encoding processing on the occluded image to obtain the visual features of the occluded image.

[0054] The semantic feature encoding module 540 is used to perform text encoding processing on the prompt text of the blocked image to obtain the semantic features of the prompt text.

[0055] The decoding module 550 is used to decode the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text to obtain pixel-level features corresponding to the occluded image and the prompt text.

[0056] The denoising module 560 is used to denoise the initial noise image based on the visual features, pixel-level features and semantic features of the prompt text of the occluded image through a diffusion model to obtain a target repaired image corresponding to the occluded image.

[0057] The image restoration device 500 can better understand the overall structure of the image by extracting structural features of the occluded image. This process ensures that the restored image is structurally reasonable and avoids common artifacts and unnatural restoration results. Visual feature encoding of the occluded image can capture the details and texture information in the image. This meticulous feature extraction helps to maintain the original style and details of the image during the restoration process and improves the visual quality of the restored image. By encoding the prompt text with semantic features, the text information can be effectively integrated into the image restoration process. This combination of semantic information makes the restoration result more in line with the needs of the user and improves the intelligent level of restoration. The structural features, visual features and semantic features are converted into pixel-level features through decoding processing, so that each pixel in the restoration process can be accurately processed. This feature ensures the consistency and coherence of the restored image in details. Using the diffusion model to perform denoising on the initial noise map can effectively reduce the noise interference in the restoration process, improve the clarity and quality of the restored image, and make the restoration result more realistic and in line with the visual perception of the human eye.

[0058] In some embodiments of the present disclosure, the structural feature extraction module 520 is configured to: perform structural extraction processing on the occluded image through a control network to obtain the structural features of the occluded image; the visual feature encoding module 530 is configured to perform image encoding processing on the occluded image through an image encoder to obtain the visual features of the occluded image.

[0059] In some embodiments of the present disclosure, the semantic feature encoding module 540 is configured to perform text encoding processing on the prompt text of the occluded image through a text encoder to obtain the semantic features of the prompt text; the decoding module 550 is configured to fuse the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text through the cross-attention mechanism in the Transformer decoder to obtain pixel-level features corresponding to the occluded image and the prompt text.

[0060] In some embodiments of the present disclosure, the denoising module 560 is configured to perform text and image alignment processing based on the initial noise map and the semantic features of the prompt text to obtain a first fused feature; perform fusion processing based on the pixel-level features and the first fused features to obtain a second fused feature; perform cross-attention processing based on the visual features of the occluded image and the second fused features through an image cross-attention mechanism to obtain a third fused feature; process the third fused feature through a feedforward network to obtain a target feature, and generate a target repair image corresponding to the occluded image based on the target feature.

[0061] In some embodiments of the present disclosure, text and image alignment are performed based on the semantic features of the initial noise map and the prompt text to obtain the first fusion feature, including: processing the initial noise map through a self-attention mechanism to obtain the structural features of the initial noise map; and performing text and image alignment based on the structural features of the initial noise map and the semantic features of the prompt text through a text cross-attention mechanism to obtain the first fusion feature.

[0062] In some embodiments of the present disclosure, a fusion process is performed based on pixel-level features and first fused features to obtain a second fused feature, including: processing pixel-level features through a learnable cross-attention mechanism to obtain learnable cross-pixel-level features; performing nonlinear transformation processing on the learnable cross-pixel-level features through a multi-layer perceptron to obtain nonlinearly transformed pixel-level features; performing cross-attention processing on the nonlinearly transformed pixel-level features and the first fused features through pixel-level cross-attention to obtain the second fused feature.

[0063] In some embodiments of the present disclosure, the image restoration device 500 is also used to perform convolution processing on pixel-level features to obtain pixel-level enhanced features; upsample the pixel-level enhanced features, and generate a reconstructed line drawing of the occluded image based on the upsampling processing results.

[0064] Figure 6 Schematic diagram of an electronic device 6 provided in an embodiment of the present disclosure. Figure 6As shown, the electronic device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules in the above-mentioned device embodiments are implemented.

[0065] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.

[0066] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0067] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0068] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0069] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0070] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image restoration method, characterized in that: include: Acquire an obscured image and a prompt text of the obscured image; the prompt text includes the content of the obscured image and / or a desired restoration effect; Performing image structure extraction processing on the obscured image to obtain structural features of the obscured image; the structural features include shape, contour, edge and spatial layout information of the obscured image; Performing image coding processing on the occluded image to obtain visual features of the occluded image; the visual features include basic color and texture information, and global context information reflecting the relationship between the occluded area and the surrounding area; Performing text encoding processing on the prompt text of the blocked image to obtain semantic features of the prompt text; The structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text are fused through the cross attention mechanism in the Transformer decoder to obtain pixel-level features corresponding to the occluded image and the prompt text. The pixel-level features are a high-dimensional feature map finally output by the Transformer decoder, and each pixel in the high-dimensional feature map corresponds to the features of the input prompt text and the occluded image. The initial noise image is denoised by using a diffusion model based on the visual features of the occluded image, the pixel-level features, and the semantic features of the prompt text to obtain a target repaired image corresponding to the occluded image.

2. The method according to claim 1, characterized in that Performing image structure extraction processing on the occluded image to obtain structural features of the occluded image, including: performing structure extraction processing on the occluded image through a control network to obtain structural features of the occluded image; Performing image coding processing on the occluded image to obtain visual features of the occluded image includes: performing image coding processing on the occluded image through an image encoder to obtain visual features of the occluded image.

3. The method according to claim 1, characterized in that Performing text encoding processing on the prompt text of the occluded image to obtain semantic features of the prompt text includes: performing text encoding processing on the prompt text of the occluded image by a text encoder to obtain semantic features of the prompt text.

4. The method according to claim 1, characterized in that De-noising the initial noise image based on the visual features of the occluded image, the pixel-level features, and the semantic features of the prompt text by using a diffusion model to obtain a target repaired image corresponding to the occluded image includes: Performing text and image alignment processing based on the initial noise map and the semantic features of the prompt text to obtain a first fusion feature; Performing a fusion process based on the pixel-level feature and the first fusion feature to obtain a second fusion feature; Performing cross-attention processing based on the visual features of the occluded image and the second fused features through an image cross-attention mechanism to obtain a third fused feature; The third fusion feature is processed through a feedforward network to obtain a target feature, and a target repaired image corresponding to the occluded image is generated based on the target feature.

5. The method according to claim 4, characterized in that Based on the semantic features of the initial noise map and the prompt text, text and image alignment processing is performed to obtain the first fusion feature, including: Processing the initial noise map through a self-attention mechanism to obtain structural features of the initial noise map; The text and image are aligned based on the structural features of the initial noise image and the semantic features of the prompt text through a text cross-attention mechanism to obtain a first fusion feature.

6. The method according to claim 4, characterized in that The second fusion feature is obtained by performing fusion processing based on the pixel-level feature and the first fusion feature, including: Processing the pixel-level features through a learnable cross-attention mechanism to obtain learnable cross-pixel-level features; Performing nonlinear transformation processing on the learnable cross-pixel features through a multi-layer perceptron to obtain nonlinearly transformed pixel-level features; The pixel-level features of the nonlinear transformation and the first fused features are cross-attention processed through pixel-level cross-attention to obtain the second fused features.

7. The method according to claim 3, characterized in that The method further comprises: Performing convolution processing on the pixel-level features to obtain pixel-level enhanced features; The pixel-level enhanced features are upsampled, and a reconstructed line drawing of the occluded image is generated based on the upsampling result.

8. An image restoration device, characterized in that: include: An acquisition module, used for acquiring an obscured image and a prompt text of the obscured image; the prompt text includes the content of the obscured image and / or a desired restoration effect; A structural feature extraction module, used to perform image structure extraction processing on the occluded image to obtain structural features of the occluded image; the structural features include shape, contour, edge and spatial layout information of the occluded image; A visual feature encoding module, used to perform image encoding processing on the occluded image to obtain visual features of the occluded image; the visual features include basic color and texture information, and global context information reflecting the relationship between the occluded area and the surrounding area; A semantic feature encoding module, used for performing text encoding processing on the prompt text of the blocked image to obtain the semantic features of the prompt text; A decoding module, used for fusing the structural features of the occluded image, the visual features of the occluded image, and the semantic features of the prompt text through a cross-attention mechanism in a Transformer decoder to obtain pixel-level features corresponding to the occluded image and the prompt text, wherein the pixel-level features are a high-dimensional feature map finally output by the Transformer decoder, and each pixel in the high-dimensional feature map corresponds to features of the input prompt text and the occluded image; The denoising module is used to denoise the initial noise image based on the visual features of the occluded image, the pixel-level features, and the semantic features of the prompt text through a diffusion model to obtain a target repaired image corresponding to the occluded image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images