Image restoration method and apparatus, electronic device and storage medium

Abstract

The present application relates to the technical field of image processing, and provides an image restoration method and apparatus, an electronic device and a storage medium. The method comprises: obtaining an image to be restored, and inputting the image to be restored into an image restoration network, wherein the image restoration network comprises a restoration stage and a generation stage; in the restoration stage, removing degradation in the image to be restored, so as to obtain a restored image; and in the generation stage, performing image detail supplementation on the restored image to obtain a target image. The present application solves the problem in image restoration related art that an image restoration method has insufficient generalization and poor robustness because the image restoration method cannot simultaneously realize image restoration on any image containing any degradation.

Description

Image restoration method, device, electronic device and storage medium Technical Field

[0001] The present application relates to the field of image processing technology. Specifically, the present application relates to an image restoration method, device, electronic device and storage medium. Background Art

[0002] Image inpainting aims to reconstruct a high-quality restored image from a low-quality image to be inpainted. To process real-world images to achieve realistic image inpainting under arbitrary image degradation conditions, blind image inpainting (BIR) has emerged.

[0003] At present, the research on blind image inpainting is still in its early stages, and the existing blind image inpainting methods can be roughly divided into three research topics, namely blind image super-resolution (BSR), zero-shot image inpainting (ZIR) and blind face inpainting (BFR).

[0004] Blind image super-resolution (BSR) formulates the image restoration problem as a supervised large-scale degradation fitting problem. Degradation reordering strategies and high-order degradation models are proposed to simulate real-world degradation. However, the degradation setting for BSR is limited to 4x / 2x super-resolution, which is incomplete for arbitrary images and arbitrary degradation conditions.

[0005] Zero-shot image restoration (ZIR) uses a powerful diffusion model as additional prior information. However, ZIR can only handle well-defined degradations (linear or nonlinear) but has poor generalization capabilities for unknown degradations. In other words, ZIR can achieve realistic restoration on any image, but cannot perform image restoration under arbitrary degradation conditions.

[0006] Blind face restoration (BFR) has a similar solution to blind image super-resolution (BSR), but differs in the degradation model and generative network. It can be implemented on real-world face images using VQGAN and Transformer. However, Blind Face Restoration (BFR) usually assumes a fixed input size and a constrained image space, so it cannot be applied to arbitrary images for image restoration.

[0007] In summary, the problem of how to improve the generalization and robustness of image restoration methods and realize image restoration on arbitrary images with arbitrary degradation remains to be solved.

[0008] Summary of the Invention

[0009] This application provides an image restoration method, device, electronic device, and storage medium that can address the problems in related technologies such as the inability to simultaneously perform image restoration on any image containing any degradation, the lack of generalization, and poor robustness of image restoration methods. The technical solution is as follows:

[0010] According to one aspect of the present application, an image restoration method includes: obtaining an image to be restored and inputting the image to be restored into an image restoration network; the image restoration network includes a restoration stage and a generation stage; in the restoration stage, removing degradation from the image to be restored to obtain a restored image; in the generation stage, supplementing image details of the restored image to obtain a target image.

[0011] According to one aspect of the present application, an image restoration device is characterized in that the device includes: an image acquisition module for acquiring an image to be restored and inputting the image to be restored into an image restoration network; a restoration module for removing degradation in the image to be restored during the restoration stage to obtain a restored image; and a generation module for supplementing image details of the restored image during the generation stage to obtain a target image.

[0012] In an exemplary embodiment, the restoration stage utilizes a degradation model and a restoration model. The restoration module includes a degradation unit configured to synthesize a low-quality image using the degradation model and train the restoration model using the low-quality image. A restoration unit configured to perform image restoration on the image to be restored using the trained restoration model to obtain a restored image.

[0013] In an exemplary embodiment, the restoration module includes: a feature extraction unit, used to extract shallow features and deep features of the downsampled image to be restored; a feature fusion unit, used to fuse the extracted shallow features and deep features to obtain fused features of the image to be restored; and a scale restoration unit, used to restore the image scale of the image to be restored based on the fused features of the image to be restored to obtain the restored image.

[0014] In an exemplary embodiment, the generation stage calls a diffusion model that introduces a control module; the control module is used to control the image detail supplementation process of the diffusion model in the generation stage; the diffusion model is a trained machine learning model that has the ability to supplement image details of the repaired image.

[0015] In an exemplary embodiment, the control module includes: a mapping unit configured to map the inpainted image to a latent space using an encoder in the diffusion model to obtain encoding information corresponding to the inpainted image; a control information acquisition unit configured to input the encoding information and the current noisy image into the control module to obtain control information corresponding to the current noisy image; and a denoising unit configured to perform inverse denoising on the current noisy image using the diffusion model under the guidance of the control information corresponding to the current noisy image, and to use the denoised image as the current noisy image, repeating the denoising until all noise is removed to obtain the target image.

[0016] In an exemplary embodiment, the denoising unit includes: a feature generation unit for generating multiple intermediate features based on the inverse denoising process performed on the current noisy image; and a control unit for modifying at least one intermediate feature according to control information corresponding to the current noisy image to control the inverse denoising process corresponding to the current noisy image.

[0017] According to one aspect of the present application, an image restoration device is provided, characterized in that the device includes: a training module, configured to train the initial control model using a plurality of training images to obtain the trained control model;

[0018] In an exemplary embodiment, the training module includes: an image acquisition unit, used to acquire a plurality of the training images and initialize the parameters of the control module; during the training process of the control module, the parameters of the diffusion model remain unchanged; a control unit, used to control the diffusion model, under the guidance of the control module, to supplement the image details of the current training image to obtain an image restoration training result; a condition determination unit, used to determine whether the control module meets the training completion condition based on the difference between the image restoration training result and the current training image; if not, updating the control module parameters until the image restoration control module meets the training completion condition, thereby obtaining the control module that has completed training.

[0019] According to one aspect of the present application, an electronic device includes: at least one processor, at least one memory, and at least one communication bus, wherein a computer program is stored in the memory, and the processor reads the computer program in the memory through the communication bus; when the computer program is executed by the processor, the image restoration method described above is implemented.

[0020] According to one aspect of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor, the image restoration method as described above is implemented.

[0021] According to one aspect of the present application, a computer program product includes a computer program, the computer program is stored in a storage medium, a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that the computer device implements the image restoration method described above when executing the computer program.

[0022] The beneficial effects of the technical solution provided by this application are:

[0023] The above-mentioned technical solution utilizes a two-stage image restoration network. In the restoration stage, a low-quality image is synthesized through multiple layers of diverse degradation to train the restoration model. The trained restoration model is then used to remove the degradation in the image to be restored, resulting in the restored image. In the generation stage, a control module is trained to control the inverse denoising process of the diffusion model to supplement the restored image with image details, resulting in the target image. This enables image restoration on any image containing arbitrary degradation, effectively resolving the issues of related art, such as the inability to simultaneously perform image restoration on any image containing arbitrary degradation, and the lack of generalization and robustness of image restoration methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0025] FIG1 is a schematic diagram of an implementation environment according to the present application;

[0026] FIG2 is a flow chart of an image restoration method according to an exemplary embodiment;

[0027] FIG3 is a flow chart of step 230 in one embodiment of the embodiment corresponding to FIG2 ;

[0028] FIG4 is a schematic diagram showing an image restoration process according to an exemplary embodiment;

[0029] FIG5 is a schematic diagram showing a process of supplementing image details according to an exemplary embodiment;

[0030] FIG6 is a flow chart of step 570 in one embodiment of the embodiment corresponding to FIG5 ;

[0031] FIG7 is a schematic diagram showing a control model training process according to an exemplary embodiment;

[0032] FIG8 is a schematic diagram of a specific implementation of an image restoration method in an application scenario;

[0033] FIG9 is a structural block diagram of an image restoration apparatus according to an exemplary embodiment;

[0034] FIG10 is a structural block diagram of a server according to an exemplary embodiment;

[0035] Fig. 11 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0037] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0038] As mentioned above, existing image restoration technologies cannot simultaneously perform image restoration on any image containing any degradation. Image restoration methods lack generalization and have poor robustness.

[0039] To this end, the image restoration method provided in this application can effectively improve the generalization and robustness of image restoration. Accordingly, the image restoration method is suitable for an image restoration device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device can be a desktop computer, a laptop computer, a server, etc.

[0040] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0041] FIG1 is a schematic diagram of an implementation environment involved in an image restoration method, wherein the implementation environment includes an image acquisition device 130 and a backend device 150 .

[0042] The image acquisition device 130 is used to capture and acquire image data, and may include an image sensor and a communication module. Specifically, the image acquisition device 130 may be a video camera, a still camera, or other electronic devices with image acquisition capabilities, such as a smartphone or tablet computer, and is not limited here.

[0043] The communication module is used to transmit image data with the backend device 150. Specifically, the communication module establishes a communication connection with the backend device 150 in advance through a wired or wireless manner, and realizes data transmission between the image acquisition device 130 and the backend device 150 through the communication connection.

[0044] The backend device 150 can be a desktop computer, laptop computer, server, or other device with image processing capabilities, and is not limited here. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. For example, in this implementation environment, the server can provide backend services related to low-light image restoration.

[0045] With the interaction between the image acquisition device 130 and the back-end device 150, the image sensor collects image data from the outside world and transmits the image data to the image acquisition device 130. Based on the transmission channel established between the image acquisition device and the back-end device 150, the image data is transmitted from the communication module of the image acquisition device 130 to the back-end device 150, so that the back-end device 150 inputs the image restoration network according to the image data to perform image restoration, removes the degradation in the image to be restored in the restoration stage, and obtains the restored image. In the generation stage, the image details of the restored image are supplemented to obtain the target image.

[0046] Please refer to FIG. 2 . An embodiment of the present application provides an image restoration method. The method is applicable to an electronic device, which may be the backend device 150 in the implementation environment shown in FIG. 1 .

[0047] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0048] As shown in FIG2 , an image restoration method is characterized in that the method may include the following steps: Step 210 , obtaining an image to be restored, and inputting the image to be restored into an image restoration network;

[0049] First of all, it should be noted that the image to be restored can be any image with any degradation, for example, old photos, old films and other images with various unknown degradation in the real world.

[0050] The image to be repaired is input into the image repair network for image repair to obtain a target image with better quality. The image repair process includes two stages: the repair stage and the generation stage.

[0051] Step 230, in the restoration stage, removing degradation from the image to be restored to obtain a restored image;

[0052] Among them, in the restoration stage, the image restoration network repairs the image to be restored through the restoration module and outputs a smooth restored image.

[0053] It should be noted that the restoration module is trained independently and can be replaced as needed in the image restoration network. For example, when the input image to be restored contains degradation beyond the generalization capability of the restoration module, the restoration module can be retrained separately and the degradation can be added during the training process to obtain a restoration module that can remove the degradation. The other parts of the image restoration network do not need to be retrained.

[0054] By independently training the restoration module, the independence and flexibility of the restoration module can be expanded, so that the image restoration network can quickly adapt to various real-world scenarios and improve generalization.

[0055] In one possible implementation, SwinIR is used as the restoration module in the image restoration network.

[0056] Step 250: In the generation phase, image details are supplemented on the repaired image to obtain a target image.

[0057] It should be noted that image detail supplementation is to generate more details in the repaired image through the inverse denoising process of the diffusion model, so as to obtain a target image with the same semantics as the repaired image but rich in details, thereby achieving the effect of realizing real image restoration.

[0058] Among them, the diffusion model can use a pre-trained stable diffusion model, or a diffusion model trained for the image type to be repaired. For example, to meet the needs of generating high-definition images, a large high-definition data set can be used to train the diffusion model, and then the trained diffusion model can be used to supplement the image details of the repaired image.

[0059] In the above process, various degradations in the image to be repaired can be removed through the repair module, and the image to be repaired can be repaired through the inverse denoising process of the diffusion model, thereby realizing image repair on any image containing any degradation.

[0060] Referring to FIG. 3 , in an exemplary embodiment, step 230 may include the following steps:

[0061] Step 310: synthesize a low-quality image using the degradation model, and train a restoration model using the low-quality image.

[0062] The degradation model simulates real-world degradation processes through three degradation processes: blurring, noise, and scaling. This includes the three most important factors: blur, noise, and scaling. The model applies a wide range of high-order degradation to high-definition images in the training set, producing degraded images that more closely resemble real-world degradation. For example, in the blurring process, the image is degraded by simulating isotropic and anisotropic Gaussian blurs. In the scaling process, the image is degraded using various algorithms, including bicubic interpolation, regional interpolation, and bilinear interpolation. For the noise process, Gaussian noise, Poisson noise, and JPEG compression noise are added to the image to be restored.

[0063] It should be noted that when synthesizing low-quality images, a two-stage degradation process and a random reordering strategy are used to simulate the random and unpredictable degradation processes in the real world. Each degradation stage includes blurring, noise generation, and scaling. The combination of multiple stages and degradation processes allows for a more comprehensive simulation and coverage of various degradation scenarios, increasing the diversity of degradation models and enabling a better fit to real-world degradation.

[0064] Step 330: Perform image restoration on the image to be restored using the trained restoration model to obtain a restored image.

[0065] In an exemplary embodiment, as shown in FIG4 , the image restoration process may include the following steps:

[0066] Step 410 , extracting shallow features and deep features from the downsampled image to be restored;

[0067] It can be understood that by using feature extraction tools at different levels, shallow features and deep features can be obtained.

[0068] Among them, shallow features are used to represent low-level information in degraded images, and deep features are used to represent high-level information in degraded images.

[0069] In one possible implementation, shallow features are obtained through 3×3 convolution operations.

[0070] In one possible implementation, deep features are obtained by extracting features from degraded images using Residual Swin Transformer Block (RSTB).

[0071] Step 430: Fusing the extracted shallow features and deep features to obtain fused features of the image to be repaired.

[0072] It is understood that the shallow features and deep features are image features that are used to uniquely represent the image to be processed. That is, the deep features and shallow features accurately describe the degraded image in a digital manner. It should be understood that different degraded images will result in different shallow features and deep features.

[0073] In one possible implementation, shallow features and deep features can be fused by adding them together to obtain fused features. The fused features of the degraded image include information expressed by both deep and shallow features, and can more comprehensively reflect the image information of the degraded image.

[0074] Step 450 : Perform image scale restoration on the image to be restored based on the fusion features of the image to be restored to obtain a restored image.

[0075] It can be understood that the image to be inpainted undergoes a downsampling process during the image feature extraction process. After obtaining the fused features, restoring the scale of the inpainted image effectively removes image degradation while preserving the valid information. Therefore, the inpainted image obtained after image restoration using the fused features is a smoothed image with degradation removed.

[0076] In a possible implementation, multiple nearest neighbor upsampling and convolution operations may be performed to restore the image from the downsampled image space to the downsampled image space, while simultaneously filtering out image degradation.

[0077] Through the above process, through the mutual cooperation of the degradation model and the restoration model, it is possible to remove the degradation in the image under realistic degradation conditions without generating erroneous details, thereby improving the image restoration quality and fidelity of the image restoration network and enhancing the generalization and robustness of image restoration.

[0078] In one embodiment, the generation phase calls a diffusion model introduced into a control module, wherein the control module is used to control the diffusion model to supplement image details during the generation phase.

[0079] Referring to FIG. 5 , in an exemplary embodiment, the image detail supplementation process may include the following steps:

[0080] Step 510 , mapping the repaired image to the latent space through the encoder in the diffusion model to obtain encoding information corresponding to the repaired image;

[0081] The diffusion model is a trained machine learning model capable of adding detail to inpainted images. The diffusion model's encoder can capture encoded information in the latent space that maximizes the preservation of inpainted image information, ensuring the quality of the inpainted image.

[0082] By mapping the inpainted image into the latent space, the encoded information can guide the inverse denoising process to fill in the image details. In this way, through the guidance of the control module, the authenticity and fidelity of the target image can be guaranteed without retraining the diffusion model.

[0083] Step 530 : Acquire a first noisy image as a current noisy image, input the coding information and the current noisy image into a control module, and obtain control information corresponding to the current noisy image. The first noisy image is a pure noise image.

[0084] The concatenation of the coding information and the noisy image is input into a control module. The control module generates control information that instructs the diffusion model to denoise the noisy image based on the guidance information contained in the input coding, thereby improving the accuracy of the diffusion model in supplementing image details. The targeted control information of the diffusion model can control the diffusion model to add rich details to the repaired image, thereby improving the robustness and generalization of image restoration.

[0085] Step 550 , under the guidance of the control information corresponding to the current noisy image, the current noisy image is inversely denoised using the diffusion model, and the denoised image is used as the current noisy image. The denoising is repeated until all noise is removed to obtain the target image.

[0086] The inverse denoising process of the diffusion model is to start from the pure noise image of the standard normal distribution under the guidance of the repaired image, and gradually remove the noise in the noisy image until all the noise is removed to obtain the target image.

[0087] In an exemplary embodiment, as shown in FIG6 , step 550 may include the following steps:

[0088] Step 610 , generating a plurality of intermediate features based on the inverse denoising process performed on the current noisy image;

[0089] Among them, the intermediate features contain the information of the noisy image generated by the inverse denoising process of the diffusion model.

[0090] Step 630: modify at least one intermediate feature according to the control information corresponding to the current noisy image to control the image detail supplementation process of the current noisy image in the inverse denoising process.

[0091] In the above process, the noisy image output from each diffusion model reverse denoising process can be combined with the encoding information and input into the control module to generate control information, thereby guiding the next diffusion model reverse denoising process. The encoding information and the control module can thus be used to adjust the diffusion model's reverse denoising process. The noisy image generated during the reverse denoising process can then be used as input to the control module to adjust the generation of control information, thus forming a closed control loop. Each reverse denoising process can result in a noisy image with richer details and better image restoration results.

[0092] In an exemplary embodiment, the control model is obtained through training, and the initial control model is trained using multiple training images to obtain a completed trained control model. It should be noted that since the image restoration network is divided into a restoration stage and a generation stage, the number of parameters that need to be adjusted during training is small, which can reduce the training cost and improve the stability of the training.

[0093] Referring to Figure 7, the control model training process may include the following steps:

[0094] Step 710: Acquire multiple training images and initialize the parameters of the control module; during the training process of the control module, the parameters of the diffusion model remain unchanged;

[0095] That is to say, when training the control module, the parameters of the diffusion model are kept unchanged, and by continuously updating the control parameters, the control module is trained in the direction of better controlling the diffusion model to supplement image details.

[0096] This method helps to keep the diffusion model's ability to supplement image details consistent during the training process, avoiding fluctuations in the diffusion model's quality during training and affecting the training of the control module.

[0097] Step 730: Under the guidance of the control module, the diffusion model is controlled to supplement the image details of the current training image to obtain an image restoration training result;

[0098] It can be understood that the better the image quality of the training result, the better the control ability of the control model.

[0099] Step 750 , based on the difference between the image restoration training result and the current training image, determine whether the control module meets the training completion condition; if not, execute step 751 , and if so, execute step 753 .

[0100] Step 751, update the control module parameters and return to step 730;

[0101] Step 753: Obtain a control module that has completed training.

[0102] It's worth noting that the control module parameters are gradually adjusted based on the differences between the image inpainting training results and the input training images. For a given training run, the differences between the image inpainting training results and the input training images are analyzed to determine the optimal direction for adjusting the control module parameters. This adjustment then improves the inpainting model's performance.

[0103] FIG8 is a schematic diagram of a specific implementation of an image restoration method in an application scenario.

[0104] In this application scenario, in the restoration stage (Stage 1), the two-stage degradation model is first used to synthesize the low-quality image I lq Specifically, in the first and second stages, the order of blur process (Blur), scaling process (Resize), and noise process (Noise) is determined by shuffling process (Shuffle) respectively, and then the high-definition image I hq Perform two degradations in two stages to obtain a low-quality image I lq . Use high-definition images I hq With low quality image I lq Train the restoration module SwinIR. Specifically, a simple L2 regression loss is selected as the loss function, and the pixel unshuffle operation is used to restore the degraded image I lq Perform 8x downsampling, use 3x3 convolution to extract shallow features, and use a series of Residual Swin Transformer Blocks (RSTB) to extract deep features. Each RSTB is composed of several Swin Transformer Layers (STL). After completing the feature extraction of shallow features and deep features in the image space after 8x downsampling, the shallow features and deep features are fused by addition. Finally, the image is restored to the original image space through 3 times nearest neighbor upsampling and convolution. The output of the repair module SwinIR is recorded as the repaired image I reg .

[0105] In the generation stage (Stage 2), the sampling strategy adopted is the Spaced sampling proposed by IDDPM, with a step number of 50 steps, and the pre-trained Stable Diffsuion encoder ε is used to restore the image I reg Mapped to the latent space, the corresponding encoding information ε(I reg ), the inverse denoising process (Denoiser) of the diffusion model is used to repair the image I reg Processing is performed by converting the coded information ε(I reg ) and the intermediate state (Z t , Zt-1 ...Z0) input control module (Parallel Module) to obtain control information to control the reverse denoising process of each diffusion model, and finally obtain the target image I diff .

[0106] This application can achieve better image restoration effects, realize image restoration on any image containing any degradation, and improve the generalization and robustness of image restoration.

[0107] The following are embodiments of the apparatus of the present application, which can be used to perform the image restoration method involved in the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the image restoration method involved in the present application.

[0108] Please refer to FIG. 9 . An embodiment of the present application provides an image restoration device 900 , including but not limited to: an image acquisition module 910 , a restoration module 930 , and a generation module 950 .

[0109] The image acquisition module is used to acquire the image to be repaired and input the image to be repaired into the image repair network;

[0110] The restoration module is used to remove degradation in the image to be restored during the restoration phase to obtain a restored image;

[0111] The generation module is used to supplement the image details of the repaired image in the generation stage to obtain the target image.

[0112] In one exemplary embodiment, the restoration phase utilizes both a degradation model and a restoration model. The restoration module includes a degradation unit configured to synthesize a low-quality image using the degradation model and train the restoration model using the low-quality image. A restoration unit configured to perform image restoration on the image to be restored using the trained restoration model to obtain a restored image.

[0113] In an exemplary embodiment, the restoration module includes: a feature extraction unit for extracting shallow features and deep features of the downsampled image to be restored; a feature fusion unit for fusing the extracted shallow features and deep features to obtain fused features of the image to be restored; and a scale restoration unit for restoring the image scale of the image to be restored based on the fused features of the image to be restored to obtain a restored image.

[0114] In an exemplary embodiment, the generation phase calls a diffusion model that introduces a control module; the control module is used to control the image detail supplementation process of the diffusion model in the generation phase; the diffusion model is a trained machine learning model that has the ability to supplement image details of the repaired image.

[0115] In an exemplary embodiment, the control module includes: a mapping unit, configured to map the repaired image to a latent space through an encoder in a diffusion model to obtain encoding information corresponding to the repaired image; and a control information acquisition unit, configured to input the encoding information and the current noisy image into the control module to obtain control information corresponding to the current noisy image.

[0116] The denoising unit is used to perform inverse denoising on the current noisy image through a diffusion model under the guidance of control information corresponding to the current noisy image, and use the denoised image as the current noisy image, repeatedly denoising until all noise is removed to obtain the target image.

[0117] In an exemplary embodiment, the denoising unit includes: a feature generation unit for generating multiple intermediate features based on an inverse denoising process performed on a current noisy image; and a control unit for modifying at least one intermediate feature according to control information corresponding to the current noisy image to control the inverse denoising process corresponding to the current noisy image.

[0118] According to one aspect of the present application, an image restoration device is characterized in that the device includes: a training module for training an initial control model using a plurality of training images to obtain a trained control model;

[0119] In an exemplary embodiment, the training module includes: an image acquisition unit, configured to acquire a plurality of training images and initialize the parameters of the control module; during the training process of the control module, the parameters of the diffusion model remain unchanged; a control unit, configured to control the diffusion model, under the guidance of the control module, to supplement the image details of a current training image to obtain an image restoration training result; a condition determination unit, configured to determine whether the control module meets the training completion condition based on the difference between the image restoration training result and the current training image; if not, updating the control module parameters until the image restoration control module meets the training completion condition, thereby obtaining a control module that has completed training.

[0120] It should be noted that the image restoration device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing image restoration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the image restoration device will be divided into different functional modules to complete all or part of the functions described above.

[0121] In addition, the image restoration device and the image restoration method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0122] Fig. 10 is a schematic diagram showing the structure of a server according to an exemplary embodiment. The server is applicable to the backend device 150 in the implementation environment shown in Fig. 1 .

[0123] It should be noted that the server is only an example adapted for the present application and should not be considered to provide any limitation on the scope of use of the present application. The server should not be interpreted as needing to rely on or necessarily having one or more components in the exemplary server 2000 shown in Figure 10.

[0124] The hardware structure of the server 2000 may vary greatly due to different configurations or performances. As shown in FIG10 , the server 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0125] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the server 2000 .

[0126] The interface 230 includes at least one wired or wireless network interface for interacting with external devices, such as the interaction between the terminal 100 and the server 200 in the implementation environment shown in FIG1 .

[0127] Of course, in other examples adapted by this application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, as shown in FIG10 , which is not specifically limited here.

[0128] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0129] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 200 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0130] The application 253 is a program instruction or code that performs at least one specific task based on the operating system 251. It may include at least one module (not shown in FIG10 ), each of which may include program instructions or code for the server 2000. For example, the information recommendation device may be considered as the application 253 deployed on the server 2000.

[0131] The data 255 may be photos, pictures, etc. stored in a disk, or may be recommendation information, etc. stored in the memory 250 .

[0132] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read program instructions or code stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 in the memory 250. For example, the information recommendation method may be implemented by the central processing unit 270 reading a series of program instructions or code stored in the memory 250.

[0133] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0134] Please refer to FIG. 11 . An electronic device 4000 is provided in an embodiment of the present application. The electronic device 4000 may include a desktop computer, a laptop computer, a server, etc.

[0135] In FIG. 11 , the electronic device 4000 includes at least one processor 4001 , at least one communication bus 4002 , and at least one memory 4003 .

[0136] The processor 4001 and the memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0137] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0138] Communication bus 4002 may include a path for transmitting information between the aforementioned components. Communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Communication bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG10 shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0139] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0140] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002 .

[0141] When the computer program is executed by the processor 4001 , the image restoration method in the above-mentioned embodiments is implemented.

[0142] In addition, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the image restoration method in the above embodiments is implemented.

[0143] The present application provides a computer program product including a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the image restoration method described in the above embodiments.

[0144] Compared with related technologies, the present application realizes image restoration on any image containing any degradation, thereby improving the generalization and robustness of image restoration.

[0145] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0146] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image restoration method, characterized in that: The method comprises: Acquire an image to be repaired, and input the image to be repaired into an image repair network; the image repair network includes a repair phase and a generation phase; In the restoration stage, the degradation in the image to be restored is removed to obtain a restored image; In the generation stage, the repaired image is supplemented with image details to obtain a target image.

2. The method according to claim 1, characterized in that The restoration stage calls a degradation model and a restoration model; removing the degradation in the image to be restored to obtain a restored image includes: synthesizing a low-quality image through the degradation model, and training a restoration model through the low-quality image; The image to be repaired is repaired using the trained repair model to obtain a repaired image.

3. The method according to claim 1, characterized in that The removing of the degradation in the image to be repaired to obtain the repaired image comprises: Extracting shallow features and deep features of the downsampled image to be restored respectively; Performing feature fusion on the extracted shallow features and deep features to obtain fused features of the image to be repaired; The image to be repaired is scaled based on the fusion features of the image to be repaired to obtain the repaired image.

4. The method according to claim 1, characterized in that The generation phase calls the diffusion model introduced into the control module; the control module is used to control the diffusion model to supplement the image details in the generation phase; the diffusion model is a trained machine learning model that has the ability to supplement the image details of the repaired image.

5. The method according to claim 4, characterized in that The controlling the diffusion model to supplement image details in the generating stage includes: Mapping the repaired image to a latent space through an encoder in the diffusion model to obtain encoding information corresponding to the repaired image; The first noisy image is obtained as the current noisy image, the encoding information and the current noisy image are input into the control module to obtain control information corresponding to the current noisy image, and the first noisy image is used as the current noisy image. A noisy image is a pure noise image; Under the guidance of the control information corresponding to the current noisy image, the current noisy image is reversely denoised through the diffusion model, and the denoised image is used as the current noisy image. The denoising is repeated until all noise is removed to obtain the target image.

6. The method according to claim 5, characterized in that The step of denoising the current noisy image by using the diffusion model under the guidance of the control information corresponding to the current noisy image includes: Generating a plurality of intermediate features based on a reverse denoising process performed on the current noisy image; At least one intermediate feature is modified according to control information corresponding to the current noisy image to control an inverse denoising process corresponding to the current noisy image.

7. The method according to any one of claims 4 to 6, characterized in that The image restoration method further comprises: Using a plurality of training images to train the initial control model to obtain the trained control model; The using of a plurality of training images to train the initial control model to obtain the trained control model comprises: Acquire a plurality of the training images and initialize the parameters of the control module; during the training process of the control module, the parameters of the diffusion model remain unchanged; Under the guidance of the control module, the diffusion model is controlled to supplement the image details of the current training image to obtain an image restoration training result; Determining whether the control module meets a training completion condition according to a difference between the image restoration training result and a current training image; If not, update the control module parameters until the image restoration control module meets the training completion condition, thereby obtaining the control module that has completed the training.

8. An image restoration device, characterized in that: include: An image acquisition module is used to acquire an image to be repaired and input the image to be repaired into an image repair network; A restoration module, used for removing degradation in the image to be restored during the restoration stage to obtain a restored image; The generation module is used to supplement the image details of the repaired image in the generation stage to obtain the target image.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, and at least one communication bus, wherein: The memory stores a computer program, and the processor reads the computer program in the memory through the communication bus; When the computer program is executed by the processor, the image restoration method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 7 is implemented.

Images