Image restoration method and device based on diffusion model, equipment and medium

Abstract

The invention relates to an image restoration method and device based on a diffusion model, equipment and a medium. Obtaining a to-be-restored image, and performing semantic segmentation on the to-be-restored image to obtain at least one image region corresponding to different semantic categories; then, based on the semantic category corresponding to the at least one image region, fidelity weight values are determined for pixels contained in the at least one image region, respectively. Thirdly, based on the fidelity weight value, performing image fusion on the to-be-restored image and a randomly generated noise image to obtain an initialized noise image; and finally, converting a prompt text generated based on the semantic category corresponding to the at least one image region and the initialized noise image into embedded vectors, inputting the embedded vectors into a pre-trained diffusion model, and performing reasoning calculation on the initialized noise image by the diffusion model based on the prompt text so as to complete image restoration of the to-be-restored image. And image restoration is carried out on the basis of initializing the noise image, so that the fidelity of the restored image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for image restoration based on a diffusion model. Background Art

[0002] During camera imaging, image processing, image transmission, and image storage, due to software and hardware defects in multiple links, images are prone to distortion due to factors such as image noise, image blur, image distortion, and insufficient image contrast. To address this issue, image restoration algorithms are often used to recover the original image from degraded images to improve image quality.

[0003] The current image restoration methods are mainly divided into two types: image restoration algorithms based on convolutional neural networks and image restoration algorithms based on stable diffusion models.

[0004] Image restoration algorithms based on convolutional neural networks can remove noise from images to improve image quality, but their generation capabilities are limited. For example, when processing severely degraded pet images, they are unable to reproduce the details of individual hair strands.

[0005] Image restoration algorithms based on the stable diffusion model can generate more image details, but the fidelity of the generated images is poor. For example, watermarks can be created on the sky, or Chinese characters in an image can be replaced with Arabic- or English-like symbols. Summary of the Invention

[0006] The present disclosure provides an image restoration method, apparatus, user equipment, and storage medium based on a diffusion model to address deficiencies in related technologies.

[0007] According to a first aspect of an embodiment of the present disclosure, a diffusion model-based image restoration method is proposed, comprising:

[0008] Acquiring an image to be restored, and performing semantic segmentation on the image to be restored to obtain at least one image region corresponding to different semantic categories;

[0009] Determining a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region; wherein the fidelity weight value determined for each image region is used to indicate a fidelity level of image restoration performed for the image region; and the fidelity weight value determined for each image region is positively correlated with a fidelity level of image content contained in the image region;

[0010] performing a weighted operation on pixel values ​​contained in the image to be restored and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to perform image fusion on the image to be restored and the randomly generated noise image to obtain an initialized noise image;

[0011] A prompt text is generated based on the semantic category corresponding to the at least one image area, and the prompt text and the initialized noise image are converted into embedding vectors and input into a pre-trained diffusion model. The diffusion model performs inference calculations on the initialized noise image based on the prompt text to complete image restoration for the image to be restored.

[0012] Optionally, the image to be restored includes a degraded image with at least part of its image content distorted.

[0013] Optionally, determining a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region includes:

[0014] Based on the semantic categories corresponding to the at least one image area, the preset first mapping relationships are queried respectively, and the fidelity weight values ​​of the semantic categories corresponding to the at least one image area that have a mapping relationship are determined as the fidelity weight values ​​corresponding to the pixels contained in each image area; wherein the first mapping relationship is a mapping relationship between semantic categories and fidelity weight values.

[0015] Optionally, determining a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region includes:

[0016] querying a preset first mapping relationship based on the semantic category corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the semantic category corresponding to the at least one image region as a first fidelity weight value corresponding to the pixels contained in each image region; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value;

[0017] determining a second fidelity weight value for each pixel contained in each of the at least one image region based on at least one auxiliary evaluation parameter;

[0018] Calculating a weighted average of the first fidelity weight value and the second fidelity weight value as a fidelity weight value corresponding to pixels included in the at least one image region;

[0019] Wherein, based on at least one auxiliary evaluation parameter, determining a second fidelity weight value for each pixel contained in each image region of the at least one image region includes at least one of the following:

[0020] Based on the scene category of the image to be restored, querying a preset second mapping relationship, and determining the fidelity weight value having a mapping relationship with the scene category of the image to be restored as the second fidelity weight value corresponding to the pixels contained in each image region; wherein the second mapping relationship is a mapping relationship between the scene category and the fidelity weight value;

[0021] querying a preset third mapping relationship based on the area size corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the area size corresponding to the at least one image region as a second fidelity weight value corresponding to the pixels contained in each image region; wherein the third mapping relationship is a mapping relationship between area size and fidelity weight value;

[0022] Based on the pixels included in the at least one image region, the fidelity weight value of the contour pixels in the at least one image region is determined as a second fidelity weight value corresponding to the contour pixels included in the respective image regions.

[0023] Optionally, the diffusion model includes a stable diffusion model.

[0024] Optionally, the image region is a mask image represented in the form of an image mask; pixel values ​​of pixels contained in the mask image are preset mask values ​​for representing semantic categories;

[0025] Performing a weighted operation on pixel values ​​contained in the at least one image region and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, including:

[0026] replacing pixel values ​​of pixels contained in the mask image corresponding to the at least one image region with fidelity weight values ​​corresponding to the pixels by the mask values;

[0027] The pixel value corresponding to each pixel contained in the mask image is used as a weight value, and a weighted operation is performed on the pixel value at the corresponding position contained in the randomly generated noise image and the pixel value at the corresponding position contained in the encoded image obtained by image encoding the image to be restored.

[0028] Optionally, generating prompt text based on the semantic category corresponding to the at least one image region includes:

[0029] generating a prompt text based on the semantic category corresponding to the at least one image region and the auxiliary information for generating the prompt text;

[0030] The auxiliary information includes at least one of the following:

[0031] At least one extracted shooting parameter corresponding to the image to be restored;

[0032] A scene label is obtained by performing scene recognition on the image to be restored.

[0033] Optionally, the diffusion model includes a first diffusion model for performing image restoration on the image to be restored, and a second diffusion model for extracting image features from the image to be restored; wherein the model parameters in the first diffusion model are fixed; and the second diffusion model is a pre-trained diffusion model that only includes an encoder;

[0034] Converting the prompt text and the initialized noise image into an embedding vector and inputting the embedded vector into a pre-trained diffusion model, so that the diffusion model performs image restoration on the image to be restored based on the prompt text, including:

[0035] Converting the prompt text and the initialization noise image into embedding vectors and inputting them into the first diffusion model, and obtaining first image features extracted from the initialization noise image by the first diffusion model;

[0036] Converting a coded image obtained by performing image coding on the image to be restored into an embedded vector and inputting the embedded vector into the second diffusion model, and obtaining a second image feature extracted from the coded image by the second diffusion model;

[0037] Image feature fusion is performed on the first image feature and the second image feature, and the fused image features are re-input into the first diffusion model. The diffusion model performs inference calculation based on the prompt text and the fused image features to restore the image to be restored.

[0038] Optionally, the method further includes:

[0039] Inputting the image to be restored into an auxiliary model to obtain an auxiliary restored image, wherein the fidelity of the auxiliary image is greater than the fidelity of the restored image;

[0040] performing anomaly detection on the restored image to detect abnormal areas in the restored image;

[0041] determining an image fusion weight value for image fusion of the image to be restored and the auxiliary restored image according to the abnormal area;

[0042] Based on the image fusion weight value, a weighted operation is performed on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the auxiliary restored image to fuse the restored image and the auxiliary restored image, and the restored image is optimized.

[0043] Optionally, performing abnormality detection on the restored image to detect abnormal areas in the restored image includes one or more combinations of the following:

[0044] performing error detection on sensitive areas in the restored image to detect error areas in the restored image;

[0045] performing abnormal texture detection on the image texture in the restored image to detect an abnormal texture area in the restored image;

[0046] Performing an image comparison between the restored image and the image to be restored to determine a difference area between the restored image and the image to be restored.

[0047] According to a second aspect of an embodiment of the present disclosure, a diffusion model-based image restoration device is provided, the device comprising:

[0048] an acquisition module, configured to acquire an image to be restored and perform semantic segmentation on the image to be restored to obtain at least one image region corresponding to different semantic categories;

[0049] a determination module configured to determine, based on a semantic category corresponding to the at least one image region, a fidelity weight value for each pixel contained in the at least one image region; wherein the fidelity weight value determined for each image region is used to indicate a fidelity level of image restoration performed for the image region; and the fidelity weight value determined for each image region is positively correlated with a fidelity level of image content contained in the image region;

[0050] a fusion module, configured to perform a weighted operation on pixel values ​​contained in the image to be restored and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to fuse the image to be restored with the randomly generated noise image to obtain an initialized noise image;

[0051] A restoration module is configured to generate a prompt text based on the semantic category corresponding to the at least one image region, and convert the prompt text and the initialized noise image into an embedding vector that is input into a pre-trained diffusion model. The diffusion model then performs inference calculations on the initialized noise image based on the prompt text to complete image restoration of the image to be restored.

[0052] Optionally, the determining module includes:

[0053] A query module is used to query a preset first mapping relationship based on the semantic category corresponding to the at least one image area, and determine the fidelity weight value of the semantic category corresponding to the at least one image area that has a mapping relationship as the fidelity weight value corresponding to the pixels contained in each image area; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value.

[0054] Optionally, the determining module includes:

[0055] a query module configured to query a preset first mapping relationship based on the semantic category corresponding to the at least one image region, and determine the fidelity weight value having a mapping relationship with the semantic category corresponding to the at least one image region as a first fidelity weight value corresponding to the pixels contained in each image region; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value;

[0056] a parameter module, configured to determine a second fidelity weight value for each pixel included in each image region of the at least one image region based on at least one auxiliary evaluation parameter;

[0057] a calculation module, configured to calculate a weighted average of the first fidelity weight value and the second fidelity weight value as a fidelity weight value corresponding to pixels contained in the at least one image region;

[0058] Wherein, based on at least one auxiliary evaluation parameter, determining a second fidelity weight value for each pixel contained in each image region of the at least one image region includes at least one of the following:

[0059] Based on the scene category of the image to be restored, querying a preset second mapping relationship, and determining the fidelity weight value having a mapping relationship with the scene category of the image to be restored as the second fidelity weight value corresponding to the pixels contained in each image region; wherein the second mapping relationship is a mapping relationship between the scene category and the fidelity weight value;

[0060] querying a preset third mapping relationship based on the area size corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the area size corresponding to the at least one image region as a third fidelity weight value corresponding to the pixels contained in each image region; wherein the third mapping relationship is a mapping relationship between area size and fidelity weight value;

[0061] Based on the pixels included in the at least one image area, the fidelity weight value of the contour pixels in the at least one image area is determined as a fourth fidelity weight value corresponding to the contour pixels included in each image area. 。

[0062] Optionally, the image region is a mask image represented in the form of an image mask; pixel values ​​of pixels contained in the mask image are preset mask values ​​for representing semantic categories; and the fusion module includes:

[0063] a replacement module, configured to replace a pixel value of a pixel contained in the mask image corresponding to the at least one image region from the mask value to a fidelity weight value corresponding to the pixel;

[0064] The operation module is used to use the pixel value corresponding to each pixel contained in the mask image as a weight value, and perform weighted operations on the pixel value at the corresponding position contained in the randomly generated noise image and the pixel value at the corresponding position contained in the encoded image obtained by image encoding the image to be restored.

[0065] Optionally, the restoration module includes:

[0066] a generating module, configured to generate a prompt text based on a semantic category corresponding to the at least one image region and auxiliary information for generating the prompt text;

[0067] The auxiliary information includes at least one of the following:

[0068] At least one extracted shooting parameter corresponding to the image to be restored;

[0069] A scene label is obtained by performing scene recognition on the image to be restored.

[0070] Optionally, the diffusion model includes a first diffusion model for restoring the image to be restored, and a second diffusion model for extracting image features from the image to be restored; wherein the model parameters in the first diffusion model are fixed; the second diffusion model is a pre-trained diffusion model that only includes an encoder; and the restoration module includes:

[0071] a first extraction module, configured to convert the prompt text and the initialization noise image into an embedding vector and input the embedded vector into the first diffusion model, and obtain a first image feature extracted from the initialization noise image by the first diffusion model;

[0072] a second extraction module, configured to convert a coded image obtained by performing image coding on the image to be restored into an embedding vector, input the embedded vector into the second diffusion model, and obtain second image features extracted from the coded image by the second diffusion model;

[0073] An inference module is used to perform image feature fusion on the first image feature and the second image feature, and re-input the fused image features into the first diffusion model, and the diffusion model performs inference calculation based on the prompt text and the fused image features to restore the image to be restored.

[0074] Optionally, the device further comprises:

[0075] An input module, configured to input the image to be restored into an auxiliary model to obtain an auxiliary restored image, wherein the fidelity of the auxiliary image is greater than that of the restored image;

[0076] a detection module, configured to perform anomaly detection on the restored image and detect an abnormal area in the restored image;

[0077] A weight module, configured to determine an image fusion weight value for image fusion of the image to be restored and the auxiliary restored image according to the abnormal area;

[0078] The optimization module is used to perform a weighted operation on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the auxiliary restored image based on the image fusion weight value, so as to fuse the restored image and the auxiliary restored image and optimize the restored image.

[0079] According to a third aspect of an embodiment of the present disclosure, a user equipment is provided, including:

[0080] processor;

[0081] a memory for storing processor-executable instructions;

[0082] Wherein, the processor is used to implement the above-mentioned image restoration method based on the diffusion model.

[0083] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the above-mentioned diffusion model-based image restoration method.

[0084] The technical solutions provided by the embodiments of the present disclosure may at least have the following beneficial effects:

[0085] According to an embodiment of the present disclosure, an image to be restored is obtained, and semantic segmentation is performed on the image to be restored to obtain at least one image region corresponding to different semantic categories. Then, based on the semantic category corresponding to the at least one image region, a fidelity weight value is determined for each pixel contained in the at least one image region. Next, based on the fidelity weight value, the image to be restored is fused with a randomly generated noise image to obtain an initialized noise image. Finally, the prompt text generated based on the semantic category corresponding to the at least one image region and the initialized noise image are converted into an embedding vector and input into a pre-trained diffusion model. The diffusion model performs inference calculation on the initialized noise image based on the prompt text to complete image restoration for the image to be restored. In the present disclosure, the image content of the image to be restored is added to the noise image to obtain an initialized noise image. Performing image restoration based on the initialized noise image can reduce the randomness of the diffusion model in restoring the image to be restored, thereby avoiding distortion of the image content of the generated restored image and improving the fidelity of the restored image. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0087] Figure 1 This is a schematic flowchart of an image restoration method based on a diffusion model according to an embodiment of the present disclosure.

[0088] Figure 2 is a schematic diagram of an image fusion method according to an embodiment of the present disclosure.

[0089] Figure 3 is a schematic diagram showing an image restoration method according to an embodiment of the present disclosure.

[0090] Figure 4 is based on Figure 1 A schematic diagram of another image restoration method based on a diffusion model is shown on the basis of the illustrated embodiment.

[0091] Figure 5 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the illustrated embodiment.

[0092] Figure 6 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the illustrated embodiment.

[0093] Figure 7 is based on Figure 1 A schematic diagram of another image restoration method based on a diffusion model is shown on the basis of the illustrated embodiment.

[0094] Figure 8 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the illustrated embodiment.

[0095] Figure 9 is a schematic diagram showing another image restoration method according to an embodiment of the present disclosure.

[0096] Figure 10 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the illustrated embodiment.

[0097] Figure 11 is a schematic diagram showing another image restoration method according to an embodiment of the present disclosure.

[0098] Figure 12 This is a schematic block diagram of an image restoration device based on a diffusion model according to an embodiment of the present disclosure.

[0099] Figure 13 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the illustrated embodiment.

[0100] Figure 14 is based on Figure 13 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the illustrated embodiment.

[0101] Figure 15 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the illustrated embodiment.

[0102] Figure 16 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the illustrated embodiment.

[0103] Figure 17 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the illustrated embodiment.

[0104] Figure 18 is based on Figure 12A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the illustrated embodiment.

[0105] Figure 19 This is a schematic block diagram of an image restoration device based on a diffusion model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0106] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0107] The current image restoration methods are mainly divided into two types: image restoration algorithms based on convolutional neural networks and image restoration algorithms based on stable diffusion models.

[0108] Image restoration algorithms based on convolutional neural networks can remove noise from images to improve image quality, but their generation capabilities are limited. For example, when processing severely degraded pet images, they are unable to reproduce the details of individual hair strands.

[0109] Image restoration algorithms based on the stable diffusion model can generate more image details, but the fidelity of the generated images is poor. For example, watermarks can be created on the sky, or Chinese characters in an image can be replaced with Arabic- or English-like symbols.

[0110] See Figure 1 , Figure 1 FIG. 1 is a schematic flow chart of an image restoration method based on a diffusion model according to an embodiment of the present disclosure. Figure 1 As shown, the image restoration method based on the diffusion model may include the following steps:

[0111] In step 102, an image to be restored is acquired, and semantic segmentation is performed on the image to be restored to obtain at least one image region corresponding to different semantic categories.

[0112] In one possible implementation, an image to be restored is obtained and semantically segmented to obtain at least one image region corresponding to different semantic categories. The image to be restored herein includes a degraded image with at least some distorted image content. Semantic categories may include portraits, plants, buildings, and so on.

[0113] For example, semantic segmentation is performed on the image to be restored to obtain a mask image including an image region with a semantic category of grass, an image region with a semantic category of little boy, and an image region with a semantic category of building.

[0114] In step 104, based on the semantic category corresponding to the at least one image region, a fidelity weight value is determined for each pixel contained in the at least one image region; wherein the fidelity weight value determined for any image region is used to indicate the fidelity level of image restoration for the image region; and the fidelity weight value determined for any image region is positively correlated with the fidelity level of the image content contained in the image region.

[0115] In real-world applications, users have varying fidelity requirements for different image contents. For example, users are insensitive to deformation of plant textures on grass and have lower fidelity requirements for image regions containing grass. However, users are extremely sensitive to deformation of text, faces, and hands and have higher fidelity requirements for these areas.

[0116] Based on this, the present disclosure can determine the fidelity weight values ​​of different image regions according to the semantic categories corresponding to different image regions.

[0117] In one possible implementation, a fidelity weight value is determined for each pixel in the at least one image region based on the semantic category corresponding to the at least one image region. The fidelity weight value determined for each image region indicates the fidelity of image restoration performed for that image region. The fidelity weight value determined for each image region is positively correlated with the fidelity of the image content contained in that image region.

[0118] For example, the higher the fidelity weight value determined for an image region, the higher the fidelity of image restoration for the image region; the lower the fidelity weight value determined for an image region, the lower the fidelity of image restoration for the image region.

[0119] In step 106, a weighted operation is performed on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to perform image fusion on the image to be restored and the randomly generated noise image to obtain an initialized noise image.

[0120] In practical applications, to improve the fidelity of the restored image, the entire image content of the image to be restored is often added to the noise image. This reduces the randomness of the diffusion model's restoration of the image to be restored, helping the diffusion model better preserve image details during the restoration process. However, this method cannot adjust the fidelity of certain areas of the restored image.

[0121] Based on this, the present disclosure can perform image fusion on the image to be restored and the randomly generated noise image based on the fidelity weight value to adjust the fidelity of a part of the area in the restored image.

[0122] In one possible implementation, a weighted operation is performed on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to perform image fusion on the image to be restored and the randomly generated noise image to obtain an initialized noise image.

[0123] For example, an image region containing grass has a low fidelity weight. During the image fusion process, a small portion of the image content in the image region containing grass is fused with the noise image, resulting in a low fidelity image for the image region containing grass in the initial noise image. An image region containing text has a high fidelity weight. During the image fusion process, a large portion of the image content in the image region containing text is fused with the noise image, resulting in a high fidelity image for the image region containing text in the initial noise image.

[0124] See Figure 2 , Figure 2 is a schematic diagram of an image fusion method according to an embodiment of the present disclosure.

[0125] exist Figure 2 In the , based on the fidelity weight value, the restored image and the noise image are fused to obtain the initialized noise image.

[0126] In step 108, a prompt text is generated based on the semantic category corresponding to the at least one image area, and the prompt text and the initialized noise image are converted into embedding vectors and input into a pre-trained diffusion model. The diffusion model performs inference calculations on the initialized noise image based on the prompt text to complete image restoration for the image to be restored.

[0127] In one possible implementation, a prompt text is generated based on the semantic category corresponding to at least one image region. The prompt text and the initial noise image are converted into an embedding vector and input into a pre-trained diffusion model. The diffusion model then performs inference calculations on the initial noise image based on the prompt text to complete image restoration for the image to be restored. The diffusion model includes a stable diffusion model.

[0128] It should be noted that the diffusion model performs image restoration based on the initialized noise image. Image regions with higher fidelity weights have lower randomness in image restoration and higher fidelity. Conversely, image regions with lower fidelity weights have higher randomness in image restoration and lower fidelity.

[0129] See Figure 3 , Figure 3 is a schematic diagram showing an image restoration method according to an embodiment of the present disclosure.

[0130] exist Figure 3 In the embodiment, semantic segmentation is performed on the image to be restored to obtain a mask image, wherein the mask image contains at least one image region corresponding to different semantic categories.

[0131] Then, based on the semantic category corresponding to the at least one image region, a fidelity weight value is determined for each pixel contained in the at least one image region.

[0132] Then, based on the fidelity weight value, the restored image and the randomly generated noise image are fused to obtain the initialized noise image.

[0133] Finally, the prompt text and the initialized noise image are converted into embedding vectors and input into the pre-trained diffusion model. The diffusion model performs inference calculations on the initialized noise image based on the prompt text to complete image restoration for the image to be restored.

[0134] It can be seen from the above embodiments that performing image restoration based on the initialized noise image can reduce the randomness of the diffusion model in restoring the image areas with higher fidelity weight values ​​in the restored image, thereby avoiding distortion of the image content in the image areas with higher fidelity weight values ​​in the generated restored image and improving the fidelity of the restored image.

[0135] Furthermore, the randomness of the diffusion model in restoring the image region with a low fidelity weight value in the image to be restored can be improved, thereby improving the clarity of the image region with a low fidelity weight value in the restored image.

[0136] See Figure 4 , Figure 4 is based on Figure 1 FIG. 1 is a schematic diagram of another image restoration method based on a diffusion model based on the embodiment shown. Figure 4 As shown, the method may further include:

[0137] In step 402, a preset first mapping relationship is queried based on the semantic category corresponding to the at least one image area, and the fidelity weight value of the semantic category corresponding to the at least one image area that has a mapping relationship is determined as the fidelity weight value corresponding to the pixels contained in each image area; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value.

[0138] In one possible implementation, a preset first mapping relationship is queried based on the semantic category corresponding to at least one image region, and the fidelity weight value corresponding to the queried semantic category corresponding to the at least one image region and having a mapping relationship is determined as the fidelity weight value corresponding to the pixels contained in each image region. The first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value.

[0139] See Figure 5 , Figure 5 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the embodiment shown. Figure 5 As shown, the method may further include:

[0140] In step 502, a preset first mapping relationship is queried based on the semantic category corresponding to the at least one image area, and the fidelity weight value of the semantic category corresponding to the at least one image area that has a mapping relationship is determined as the first fidelity weight value corresponding to the pixels contained in each image area; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value.

[0141] In step 504 , a second fidelity weight value is determined for each pixel included in each image region of the at least one image region based on at least one auxiliary evaluation parameter.

[0142] In step 506, a weighted average of the first fidelity weight value and the second fidelity weight value is calculated as a fidelity weight value corresponding to the pixels included in the at least one image region;

[0143] Wherein, based on at least one auxiliary evaluation parameter, determining a second fidelity weight value for each pixel contained in each image region of the at least one image region includes at least one of the following:

[0144] Based on the scene category of the image to be restored, querying a preset second mapping relationship, and determining the fidelity weight value having a mapping relationship with the scene category of the image to be restored as the second fidelity weight value corresponding to the pixels contained in each image region; wherein the second mapping relationship is a mapping relationship between the scene category and the fidelity weight value;

[0145] querying a preset third mapping relationship based on the area size corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the area size corresponding to the at least one image region as a second fidelity weight value corresponding to the pixels contained in each image region; wherein the third mapping relationship is a mapping relationship between area size and fidelity weight value;

[0146] Based on the pixels included in the at least one image region, the fidelity weight value of the contour pixels in the at least one image region is determined as a second fidelity weight value corresponding to the contour pixels included in the respective image regions.

[0147] In practical applications, subregions within an image region belonging to the same semantic category may have different fidelity requirements. For example, in facial images, users are more sensitive to the area where the eyes are located and therefore require higher fidelity. However, methods that determine the fidelity weight corresponding to image regions based solely on semantic category fail to capture the differences between the image region where the eyes are located and other image regions, thereby reducing the fidelity of the restored image.

[0148] In one possible implementation, a preset first mapping relationship is queried based on the semantic category corresponding to at least one image region, and the fidelity weight value for each semantic category corresponding to the at least one image region that has a mapping relationship is determined as the first fidelity weight value corresponding to each pixel in each image region. The first mapping relationship is a mapping relationship between a semantic category and a fidelity weight value.

[0149] Then, based on the at least one auxiliary evaluation parameter, a second fidelity weight value is determined for each pixel contained in each image region of the at least one image region.

[0150] Finally, a weighted average of the first fidelity weight value and the second fidelity weight value is calculated as a fidelity weight value corresponding to the pixels included in the at least one image region.

[0151] Wherein, based on at least one auxiliary evaluation parameter, determining a second fidelity weight value for each pixel contained in each image region of the at least one image region includes at least one of the following:

[0152] Based on the scene category of the image to be restored, a preset second mapping relationship is queried, and the fidelity weight value that has a mapping relationship with the scene category of the image to be restored is determined as the second fidelity weight value corresponding to the pixels contained in each image region. The second mapping relationship is a mapping relationship between the scene category and the fidelity weight value.

[0153] Based on the area size corresponding to the at least one image region, a preset third mapping relationship is queried, and the fidelity weight value for which the queried area size corresponding to the at least one image region has a mapping relationship is determined as the second fidelity weight value corresponding to the pixels contained in each image region. The third mapping relationship is a mapping relationship between area size and fidelity weight value.

[0154] Based on the pixels included in the at least one image region, the fidelity weight value of the contour pixels in the at least one image region is determined as a second fidelity weight value corresponding to the contour pixels included in each image region.

[0155] For example, the fidelity weight value of an image region whose image content is grass and whose area is larger is lower than the fidelity weight value of an image region whose image content is grass and whose area is smaller. For another example, the fidelity weight value of an image whose image content is a face and whose pixels are not contour pixels is lower than the fidelity weight value of an image whose image content is a face and whose pixels are contour pixels. For another example, the fidelity weight value of an image to be restored whose scene category is landscape is lower than the fidelity weight value of an image to be restored whose scene category is portrait. It should be noted that the present disclosure does not limit the size of the fidelity weight value corresponding to the auxiliary evaluation parameter.

[0156] It can be seen from the above embodiments that the fidelity weight value of the image area is determined based on the semantic category corresponding to the image area, the scene category of the image to be restored, the area size corresponding to the image area, and whether the pixel is a contour pixel. The fidelity weight value corresponding to the pixels contained in each image area can be further adjusted, thereby further improving the fidelity of the restored image and meeting the user's requirements for the fidelity of different image areas.

[0157] It should be noted that the present disclosure can also use traditional image processing methods to count the local texture complexity, texture degree, color, skin color, edge contour, etc. of the image to determine the fidelity weight value corresponding to the pixels contained in each image area.

[0158] See Figure 6 , Figure 6 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the embodiment shown. Figure 6 As shown, the method may further include:

[0159] In step 602, pixel values ​​of pixels contained in the mask image corresponding to the at least one image region are replaced by fidelity weight values ​​corresponding to the pixels from the mask values.

[0160] In step 604, the pixel values ​​corresponding to the pixels contained in the mask image are used as weight values, and weighted operations are performed on the pixel values ​​at the corresponding positions contained in the randomly generated noise image and the pixel values ​​at the corresponding positions contained in the encoded image obtained by image encoding the image to be restored.

[0161] In a possible implementation, the image region is a mask image represented in the form of an image mask, and pixel values ​​of pixels contained in the mask image are preset mask values ​​for representing semantic categories.

[0162] The pixel values ​​of pixels contained in the mask image corresponding to the at least one image region are replaced by the mask values ​​and the fidelity weight values ​​corresponding to the pixels.

[0163] Then, the pixel values ​​corresponding to the pixels in the mask image are used as weight values, and weighted operations are performed on the pixel values ​​at the corresponding positions in the randomly generated noise image and the pixel values ​​at the corresponding positions in the encoded image obtained by encoding the image to be restored.

[0164] In a possible implementation, a mask image containing pixel values ​​corresponding to respective pixels as weight values ​​is generated based on fidelity weight values ​​corresponding to pixels contained in a mask image corresponding to at least one image region.

[0165] See Figure 7 , Figure 7 is based on Figure 1 FIG. 1 is a schematic diagram of another image restoration method based on a diffusion model based on the embodiment shown. Figure 7 As shown, the method may further include:

[0166] In step 702, a prompt text is generated based on the semantic category corresponding to the at least one image region and the auxiliary information used to generate the prompt text.

[0167] The auxiliary information includes at least one of the following:

[0168] At least one shooting parameter corresponding to the image to be restored is extracted.

[0169] A scene label is obtained by performing scene recognition on the image to be restored.

[0170] In a possible implementation, the prompt text is generated based on a semantic category corresponding to at least one image region and auxiliary information used to generate the prompt text.

[0171] The auxiliary information includes at least one of the following:

[0172] At least one shooting parameter corresponding to the image to be restored is extracted. The shooting parameters mentioned here are obtained from the header of the Exchangeable Image File (EXIF) file of the image to be restored. Shooting parameters include: aperture, white balance, ISO sensitivity (ISO), exposure time, magnification, focal length, date and time, etc.

[0173] The scene label obtained by performing scene recognition on the image to be restored. The scene label mentioned here can be obtained by performing scene recognition through a classification model. Scene labels include: landscape, outdoor, people, etc.

[0174] In one possible implementation, a prompt text is generated based on the semantic categories, shooting parameters, and scene labels corresponding to at least one image region. For example, if the semantic categories are "a little boy, on the grass, plants, people, and buildings," the shooting parameters are "ISO 100, telephoto 30x, exposure 1 / 1000, blur, daytime, sunny," and the scene label is "outdoors," the generated prompt text is "a little boy, on the grass, plants, people, buildings, ISO 100, telephoto 30x, exposure 1 / 1000, blur, daytime, sunny, and outdoor."

[0175] Furthermore, there are multiple methods for converting prompt text into an embedding vector. For example, the prompt text is input into a CLIP model (Contrastive Language-Image Pre-training) to obtain an embedding vector corresponding to the prompt text. For another example, a diffusion model maintains a mapping relationship between text and embedding vectors, and an embedding vector corresponding to the prompt text is determined based on the mapping relationship between the prompt text query text and the embedding vector. This disclosure does not limit the method for converting prompt text into an embedding vector.

[0176] It can be seen from the above embodiments that maintaining the mapping relationship between text and embedding vectors in the diffusion model can reduce the storage cost and inference time overhead of the CLIP model.

[0177] See Figure 8 , Figure 8 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the embodiment shown. Figure 8 As shown, the method may further include:

[0178] In step 802, the prompt text and the initialization noise image are converted into embedding vectors and input into the first diffusion model, and the first image features extracted from the initialization noise image by the first diffusion model are obtained.

[0179] In step 804, the coded image obtained by performing image coding on the image to be restored is converted into an embedding vector and input into the second diffusion model, and the second image feature extracted from the coded image by the second diffusion model is obtained.

[0180] In step 806, image feature fusion is performed on the first image feature and the second image feature, and the fused image features are re-input into the first diffusion model. The diffusion model performs inference calculation based on the prompt text and the fused image features to restore the image to be restored.

[0181] In one possible implementation, the diffusion model includes a first diffusion model for restoring the image to be restored and a second diffusion model for extracting image features from the image to be restored. The model parameters of the first diffusion model are fixed. The second diffusion model is a pre-trained diffusion model that only includes an encoder.

[0182] First, the prompt text and the initialization noise image are converted into embedding vectors and input into the first diffusion model, and the first image features extracted from the initialization noise image by the first diffusion model are obtained.

[0183] Then, the coded image obtained by performing image coding on the image to be restored is converted into an embedding vector and input into the second diffusion model, and the second image feature extracted from the coded image by the second diffusion model is obtained.

[0184] Finally, image feature fusion is performed on the first image feature and the second image feature, and the fused image feature is re-input into the first diffusion model. The diffusion model performs inference calculation based on the prompt text and the fused image feature to restore the image to be restored.

[0185] See Figure 9 , Figure 9 is a schematic diagram showing another image restoration method according to an embodiment of the present disclosure.

[0186] exist Figure 9 In the embodiment, the first diffusion model and the second diffusion model both include a U-Net neural network. The U-Net neural network includes several network sub-layers.

[0187] First, the prompt text and the initialized noise image are converted into embedding vectors and input into the first network sublayer in the first diffusion model, and the first image features corresponding to the first network sublayer in the first diffusion model are obtained, which are extracted from the initialized noise image by the first network sublayer in the first diffusion model.

[0188] Secondly, the encoded image obtained by image encoding the image to be restored is converted into an embedding vector and input into the first network sublayer in the second diffusion model, and the second image features corresponding to the first network sublayer in the second diffusion model are extracted from the encoded image by the first network sublayer in the second diffusion model.

[0189] Then, the first image features corresponding to the first network sublayer in the first diffusion model and the second image features corresponding to the first network sublayer in the second diffusion model are input into the feature fusion layer, and the first image features corresponding to the first network sublayer in the first diffusion model and the second image features corresponding to the first network sublayer in the second diffusion model are fused, and the fused image features of the first network sublayer are re-input into the second network sublayer in the first diffusion model to obtain the first image features corresponding to the second network sublayer in the first diffusion model.

[0190] Then, the second image features corresponding to the first network sublayer in the second diffusion model are input into the second network sublayer in the second diffusion model, and the second image features corresponding to the second network sublayer in the second diffusion model extracted from the encoded image by the second network sublayer in the second diffusion model are obtained.

[0191] Finally, the second image features corresponding to the second network sublayer in the second diffusion model and the second image features corresponding to the second network sublayer in the first diffusion model are input into the feature fusion layer, and the second image features corresponding to the second network sublayer in the second diffusion model and the second image features corresponding to the second network sublayer in the first diffusion model are fused, and the fused image features of the second network sublayer are re-input into the third network sublayer in the first diffusion model, and so on, until the last network sublayer, to perform image restoration on the image to be restored.

[0192] It should be noted that the network sublayers in the U-Net neural network include encoding and decoding layers. Each encoding layer can perform a downsampling operation to halve the resolution of the feature map and extract shallow feature information. Each decoding layer can perform an upsampling operation and is connected to a convolutional layer to double the resolution of the feature map and extract deep feature information. Then, through skip connections, the encoder and decoder at the same level are connected, combining shallow and deep information to produce a better target image.

[0193] In one possible implementation, the first image features, the second image features, and the mask image are input into a feature fusion layer, where they are fused. During the image feature fusion, the mask image is used to further enhance the first diffusion model's perception of image semantics.

[0194] It can be seen from the above embodiments that in the feature fusion layer, the image features of the image to be restored and the image of the mask image are fused, which further ensures that the diffusion model is not distorted during the image restoration process, thereby improving the fidelity of the restored image.

[0195] Before applying the diffusion model to restore the image to be restored, the diffusion model needs to be trained.

[0196] In one possible implementation, image training samples are obtained, where the image training samples have a higher quality than the image to be restored. Noise is gradually added to the image training samples to obtain a number of noisy image training samples and a number of steps corresponding to the number of noisy image training samples.

[0197] Next, a noisy image training sample is randomly selected from the plurality of noisy image training samples as a target image training sample. Semantic segmentation is performed on the target image training sample to obtain at least one image region corresponding to a different semantic category. Based on the semantic category corresponding to the at least one image region, a fidelity weight value is determined for each pixel in the at least one image region.

[0198] Then, based on the fidelity weight value, a weighted operation is performed on the pixel values ​​contained in the target image training sample and the pixel values ​​at the corresponding positions in the randomly generated noise image to perform image fusion on the target image training sample and the randomly generated noise image to obtain an initialized noise image.

[0199] Then, the prompt text, the initialization noise image, and the number of steps corresponding to the target image training sample are converted into an embedding vector and input into the first diffusion model, and the first image feature extracted from the initialization noise image by the first diffusion model is obtained.

[0200] Next, the prompt text, the encoded image obtained by image encoding the image to be restored, and the number of steps corresponding to the target image training sample are converted into embedding vectors and input into the second diffusion model, and the second image features extracted from the encoded image by the second diffusion model are obtained.

[0201] Finally, the first image feature and the second image feature are fused, and the fused image features are re-input into the first diffusion model. The diffusion model performs inference calculations based on the prompt text and the fused image features, and takes minimizing the error between the calculated predicted noise and the actual noise of the initialized noise image as the optimization goal to train the image restoration model.

[0202] In one possible implementation, during the training of the diffusion model, image feature fusion is performed on the first image feature, the second image feature, the mask image, and the number of steps corresponding to the target image training sample, and the fused image features are re-input into the first diffusion model. The diffusion model performs inference calculations based on the prompt text and the fused image features, and trains the image restoration model with the optimization goal of minimizing the error between the calculated predicted noise and the actual noise of the initialized noise image.

[0203] See Figure 10 , Figure 10 is based on Figure 1 A schematic flow chart of another image restoration method based on a diffusion model is shown on the basis of the embodiment shown. Figure 10 As shown, the method may further include:

[0204] In step 1002, the image to be restored is input into an auxiliary model to obtain an auxiliary restored image, where the fidelity of the auxiliary image is greater than that of the restored image.

[0205] In step 1004, anomaly detection is performed on the restored image to detect abnormal areas in the restored image.

[0206] In step 1006, an image fusion weight value for image fusion of the image to be restored and the auxiliary restoration image is determined according to the abnormal region.

[0207] In step 1008, a weighted operation is performed on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the auxiliary restored image based on the image fusion weight value, so as to fuse the restored image and the auxiliary restored image and optimize the restored image.

[0208] In practical applications, even if image restoration is performed based on an initialized noisy image, the fidelity of the restored image is improved. However, the restored image may still have generation errors.

[0209] In one possible implementation, the image to be restored is input into an auxiliary model to obtain an auxiliary restored image, where the fidelity of the auxiliary image is greater than that of the restored image. The auxiliary model can be implemented based on a convolutional neural network or a stable diffusion model. This disclosure does not limit the specific algorithm of the auxiliary model.

[0210] Then, anomaly detection is performed on the restored image to detect abnormal areas in the restored image.

[0211] Next, an image fusion weight value for image fusion between the image to be restored and the auxiliary restored image is determined according to the abnormal area.

[0212] Finally, based on the image fusion weight value, a weighted operation is performed on the pixel values ​​contained in the image to be restored and the pixel values ​​at the corresponding positions in the auxiliary restored image to fuse the restored image and the auxiliary restored image, and the restored image is optimized.

[0213] It should be noted that there are many methods for detecting anomalies in restored images, which may include one or a combination of the following:

[0214] Error detection is performed on sensitive areas in the restored image to detect error areas in the restored image. The sensitive areas mentioned here can be determined based on expert experience, such as areas where human body parts are located, areas where animals are located, etc.

[0215] Abnormal texture detection is performed on the image texture in the restored image to detect abnormal texture areas in the restored image. Image texture here can refer to the texture of the sky, the texture of the water surface, etc. Abnormal texture can refer to whether irregular watermarks, text, or other symbols appear in areas such as the sky and the water surface.

[0216] The restored image is compared with the image to be restored to determine the difference between the restored image and the image to be restored. If there is a significant difference between the restored image and the image to be restored, it can be determined that the image region in the restored image has been generated incorrectly.

[0217] See Figure 11 , Figure 11 is a schematic diagram showing another image restoration method according to an embodiment of the present disclosure.

[0218] exist Figure 11In the process, error detection, abnormal texture detection, and image comparison are performed on sensitive areas in the restored image. Based on the detected error areas, abnormal texture areas, and difference areas in the restored image, the image fusion weights for fusing the image to be restored with the auxiliary restored image are determined. Then, based on the image fusion weights, a weighted calculation is performed on the pixel values ​​in the image to be restored and the pixel values ​​at corresponding positions in the auxiliary restored image to fuse the restored image and the auxiliary restored image, and the restored image is optimized.

[0219] As can be seen from the above embodiments, fusing the restored image and the auxiliary restored image allows the restored image output by the diffusion model to be retained in areas without abnormalities, while the auxiliary restored image output by the auxiliary model is retained in areas with abnormalities, thereby balancing clarity and fidelity. By optimizing the restored image, a clear, non-degraded, and artifact-free restored image is obtained.

[0220] It should be noted that the present disclosure may also fuse the image features of the image to be restored with the image features of the auxiliary restored image to obtain fused image features, and then decode the fused image features through a decoder to obtain an optimized restored image. The present disclosure does not limit the method for fusing the restored image and the auxiliary restored image.

[0221] Corresponding to the aforementioned embodiment of the method for generating a code, the present disclosure further provides an embodiment of an image restoration device based on a diffusion model.

[0222] See Figure 12 , Figure 12 FIG. 1 is a schematic block diagram of an image restoration device based on a diffusion model according to an embodiment of the present disclosure. Figure 12 As shown, the device may include:

[0223] An acquisition module 1202 is configured to acquire an image to be restored and perform semantic segmentation on the image to be restored to obtain at least one image region corresponding to different semantic categories;

[0224] Determining module 1204 is configured to determine a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region; wherein the fidelity weight value determined for each image region is used to indicate a fidelity level of image restoration performed for the image region; and the fidelity weight value determined for each image region is positively correlated with a fidelity level of image content contained in the image region;

[0225] a fusion module 1206 configured to perform a weighted operation on pixel values ​​contained in the image to be restored and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to fuse the image to be restored with the randomly generated noise image to obtain an initialized noise image;

[0226] The restoration module 1208 is used to generate a prompt text based on the semantic category corresponding to the at least one image area, and convert the prompt text and the initialized noise image into an embedding vector and input it into a pre-trained diffusion model. The diffusion model performs inference calculations on the initialized noise image based on the prompt text to complete image restoration for the image to be restored.

[0227] See Figure 13 , Figure 13 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the embodiment shown. Figure 13 As shown, the determination module includes:

[0228] The query module 1302 is used to query the preset first mapping relationship based on the semantic category corresponding to the at least one image area, and determine the fidelity weight value of the semantic category corresponding to the at least one image area that has a mapping relationship as the fidelity weight value corresponding to the pixels contained in each image area; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value.

[0229] See Figure 14 , Figure 14 is based on Figure 13 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the embodiment shown. Figure 14 As shown, the determination module includes:

[0230] A query module 1402 is configured to query a preset first mapping relationship based on the semantic category corresponding to the at least one image region, and determine the fidelity weight value having a mapping relationship with the semantic category corresponding to the at least one image region as a first fidelity weight value corresponding to the pixels contained in each image region; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value;

[0231] A parameter module 1404 is configured to determine a second fidelity weight value for each pixel included in each of the at least one image region based on at least one auxiliary evaluation parameter;

[0232] A calculation module 1406 is configured to calculate a weighted average of the first fidelity weight value and the second fidelity weight value as a fidelity weight value corresponding to pixels included in the at least one image region;

[0233] Wherein, based on at least one auxiliary evaluation parameter, determining a second fidelity weight value for each pixel contained in each image region of the at least one image region includes at least one of the following:

[0234] Based on the scene category of the image to be restored, querying a preset second mapping relationship, and determining the fidelity weight value having a mapping relationship with the scene category of the image to be restored as the second fidelity weight value corresponding to the pixels contained in each image region; wherein the second mapping relationship is a mapping relationship between the scene category and the fidelity weight value;

[0235] querying a preset third mapping relationship based on the area size corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the area size corresponding to the at least one image region as a third fidelity weight value corresponding to the pixels contained in each image region; wherein the third mapping relationship is a mapping relationship between area size and fidelity weight value;

[0236] Based on the pixels included in the at least one image region, the fidelity weight value of the contour pixels in the at least one image region is determined as a fourth fidelity weight value corresponding to the contour pixels included in the respective image regions.

[0237] See Figure 15 , Figure 15 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the embodiment shown. Figure 15 As shown, the fusion module includes:

[0238] a replacement module 1502, configured to replace pixel values ​​of pixels contained in the mask image corresponding to the at least one image region with fidelity weight values ​​corresponding to the pixels from the mask values;

[0239] The operation module 1504 is used to use the pixel value corresponding to each pixel contained in the mask image as a weight value, and perform weighted operations on the pixel value at the corresponding position contained in the randomly generated noise image and the pixel value at the corresponding position contained in the encoded image obtained by image encoding the image to be restored.

[0240] See Figure 16 , Figure 16 is based on Figure 12A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the embodiment shown. Figure 16 As shown, the restoration module includes:

[0241] A generating module 1602, configured to generate a prompt text based on the semantic category corresponding to the at least one image region and the auxiliary information for generating the prompt text;

[0242] The auxiliary information includes at least one of the following:

[0243] At least one extracted shooting parameter corresponding to the image to be restored;

[0244] A scene label is obtained by performing scene recognition on the image to be restored.

[0245] See Figure 17 , Figure 17 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the embodiment shown. Figure 17 As shown, the restoration module includes:

[0246] A first extraction module 1702 is configured to convert the prompt text and the initialization noise image into an embedding vector and input the embedded vector into the first diffusion model, and obtain a first image feature extracted from the initialization noise image by the first diffusion model;

[0247] A second extraction module 1704 is configured to convert a coded image obtained by performing image coding on the image to be restored into an embedding vector, input the embedded vector into the second diffusion model, and obtain second image features extracted from the coded image by the second diffusion model;

[0248] The inference module 1706 is used to perform image feature fusion on the first image feature and the second image feature, and re-input the fused image features into the first diffusion model, and the diffusion model performs inference calculation based on the prompt text and the fused image features to restore the image to be restored.

[0249] See Figure 18 , Figure 18 is based on Figure 12 A schematic block diagram of another image restoration device based on a diffusion model is shown on the basis of the embodiment shown. Figure 18 As shown, the device also includes:

[0250] An input module 1802 is configured to input the image to be restored into an auxiliary model to obtain an auxiliary restored image, wherein the fidelity of the auxiliary image is greater than that of the restored image;

[0251] A detection module 1804 is configured to perform anomaly detection on the restored image to detect abnormal areas in the restored image;

[0252] A weight module 1806 is configured to determine an image fusion weight value for image fusion of the image to be restored and the auxiliary restored image according to the abnormal region;

[0253] The optimization module 1808 is used to perform a weighted operation on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the auxiliary restored image based on the image fusion weight value, so as to fuse the restored image and the auxiliary restored image and optimize the restored image.

[0254] Regarding the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the relevant methods and will not be elaborated on here.

[0255] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0256] Accordingly, the present disclosure further provides a user equipment, comprising: a processor;

[0257] a memory for storing processor-executable instructions;

[0258] Wherein, the processor is used to implement the above-mentioned image restoration method based on the diffusion model.

[0259] Accordingly, the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the image restoration method based on the diffusion model is implemented.

[0260] like Figure 19 As shown, Figure 19 FIG1 is a schematic block diagram of a diffusion model-based image restoration apparatus 1900 according to an embodiment of the present disclosure. For example, apparatus 1900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0261] Reference Figure 19 , device 1900 may include one or more of the following components: a processing component 1902 , a memory 1904 , a power component 1906 , a multimedia component 1908 , an audio component 1910 , an input / output (I / O) interface 1912 , a sensor component 1914 , and a communication component 1916 .

[0262] Processing component 1902 generally controls the overall operation of device 1900, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. Processing component 1902 may include one or more processors 1920 to execute instructions to perform all or part of the steps of the aforementioned methods. Furthermore, processing component 1902 may include one or more modules to facilitate interaction between processing component 1902 and other components. For example, processing component 1902 may include a multimedia module to facilitate interaction between multimedia component 1908 and processing component 1902.

[0263] The memory 1904 is configured to store various types of data to support the operation of the device 1900. Examples of such data include instructions for any application or method operating on the device 1900, contact data, phone book data, messages, pictures, videos, etc. The memory 1904 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0264] The power supply component 1906 provides power to the various components of the device 1900. The power supply component 1906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1900.

[0265] The multimedia component 1908 includes a screen that provides an output interface between the device 1900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1908 includes a front camera and / or a rear camera. When the device 1900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0266] The audio component 1910 is configured to output and / or input audio signals. For example, the audio component 1910 includes a microphone (MIC) that is configured to receive external audio signals when the device 1900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 1904 or transmitted via the communication component 1916. In some embodiments, the audio component 1910 further includes a speaker for outputting audio signals.

[0267] I / O interface 1912 provides an interface between processing component 1902 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0268] Sensor assembly 1914 includes one or more sensors for providing various aspects of the status assessment of device 1900. For example, sensor assembly 1914 can detect the open / closed state of device 1900, the relative positioning of components, such as the display and keypad of device 1900. Sensor assembly 1914 can also detect changes in the position of device 1900 or a component of device 1900, the presence or absence of user contact with device 1900, the orientation or acceleration / deceleration of device 1900, and changes in the temperature of device 1900. Sensor assembly 1914 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1914 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1914 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0269] The communication component 1916 is configured to facilitate wired or wireless communication between the device 1900 and other devices. The device 1900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR or a combination thereof. In an exemplary embodiment, the communication component 1916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0270] In an exemplary embodiment, the apparatus 1900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the method described in any of the above embodiments.

[0271] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1904 including instructions, and the instructions can be executed by the processor 1920 of the apparatus 1900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0272] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0273] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0274] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0275] The above is a detailed introduction to the methods and devices provided in the embodiments of the present disclosure. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the methods and core ideas of the present disclosure. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there may be changes in the specific implementation methods and application scopes. In summary, the content of the present disclosure should not be understood as a limitation of the present disclosure.

Claims

1. An image restoration method based on a diffusion model, characterized in that: include: Acquiring an image to be restored, and performing semantic segmentation on the image to be restored to obtain at least one image region corresponding to different semantic categories; Determining a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region; wherein the fidelity weight value determined for each image region is used to indicate a fidelity level of image restoration performed for the image region; and the fidelity weight value determined for each image region is positively correlated with a fidelity level of image content contained in the image region; performing a weighted operation on pixel values ​​contained in the image to be restored and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to perform image fusion on the image to be restored and the randomly generated noise image to obtain an initialized noise image; A prompt text is generated based on the semantic category corresponding to the at least one image area, and the prompt text and the initialized noise image are converted into embedding vectors and input into a pre-trained diffusion model. The diffusion model performs inference calculations on the initialized noise image based on the prompt text to complete image restoration for the image to be restored.

2. The method according to claim 1, characterized in that The image to be restored includes a degraded image with at least a portion of image content distorted.

3. The method according to claim 1, characterized in that Determining a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region includes: Based on the semantic categories corresponding to the at least one image area, the preset first mapping relationships are queried respectively, and the fidelity weight values ​​of the semantic categories corresponding to the at least one image area that have a mapping relationship are determined as the fidelity weight values ​​corresponding to the pixels contained in each image area; wherein the first mapping relationship is a mapping relationship between semantic categories and fidelity weight values.

4. The method according to claim 3, characterized in that Determining a fidelity weight value for each pixel contained in the at least one image region based on a semantic category corresponding to the at least one image region includes: querying a preset first mapping relationship based on the semantic category corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the semantic category corresponding to the at least one image region as a first fidelity weight value corresponding to the pixels contained in each image region; wherein the first mapping relationship is a mapping relationship between the semantic category and the fidelity weight value; determining a second fidelity weight value for each pixel contained in each of the at least one image region based on at least one auxiliary evaluation parameter; Calculating a weighted average of the first fidelity weight value and the second fidelity weight value as a fidelity weight value corresponding to pixels included in the at least one image region; Wherein, based on at least one auxiliary evaluation parameter, determining a second fidelity weight value for each pixel contained in each image region of the at least one image region includes at least one of the following: Based on the scene category of the image to be restored, querying a preset second mapping relationship, and determining the fidelity weight value having a mapping relationship with the scene category of the image to be restored as the second fidelity weight value corresponding to the pixels contained in each image region; wherein the second mapping relationship is a mapping relationship between the scene category and the fidelity weight value; querying a preset third mapping relationship based on the area size corresponding to the at least one image region, and determining the fidelity weight value having a mapping relationship with the area size corresponding to the at least one image region as a second fidelity weight value corresponding to the pixels contained in each image region; wherein the third mapping relationship is a mapping relationship between area size and fidelity weight value; Based on the pixels included in the at least one image region, the fidelity weight value of the contour pixels in the at least one image region is determined as a second fidelity weight value corresponding to the contour pixels included in the respective image regions.

5. The method according to claim 1, wherein The diffusion model includes a stable diffusion model.

6. The method according to claim 1, characterized in that The image region is a mask image represented in the form of an image mask; pixel values ​​of pixels contained in the mask image are preset mask values ​​for representing semantic categories; Performing a weighted operation on pixel values ​​contained in the at least one image region and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, including: replacing pixel values ​​of pixels contained in the mask image corresponding to the at least one image region with fidelity weight values ​​corresponding to the pixels by the mask values; The pixel value corresponding to each pixel contained in the mask image is used as a weight value, and a weighted operation is performed on the pixel value at the corresponding position contained in the randomly generated noise image and the pixel value at the corresponding position contained in the encoded image obtained by image encoding the image to be restored.

7. The method according to claim 1, characterized in that Generating prompt text based on the semantic category corresponding to the at least one image region includes: generating a prompt text based on the semantic category corresponding to the at least one image region and the auxiliary information for generating the prompt text; The auxiliary information includes at least one of the following: At least one extracted shooting parameter corresponding to the image to be restored; A scene label is obtained by performing scene recognition on the image to be restored.

8. The method according to claim 1, characterized in that The diffusion model includes a first diffusion model for restoring the image to be restored, and a second diffusion model for extracting image features from the image to be restored; wherein the model parameters in the first diffusion model are fixed; and the second diffusion model is a pre-trained diffusion model that only includes an encoder; Converting the prompt text and the initialized noise image into an embedding vector and inputting the embedded vector into a pre-trained diffusion model, so that the diffusion model performs image restoration on the image to be restored based on the prompt text, including: Converting the prompt text and the initialization noise image into embedding vectors and inputting them into the first diffusion model, and obtaining first image features extracted from the initialization noise image by the first diffusion model; Converting a coded image obtained by performing image coding on the image to be restored into an embedded vector and inputting the embedded vector into the second diffusion model, and obtaining a second image feature extracted from the coded image by the second diffusion model; Image feature fusion is performed on the first image feature and the second image feature, and the fused image features are re-input into the first diffusion model. The diffusion model performs inference calculation based on the prompt text and the fused image features to restore the image to be restored.

9. The method according to claim 1, characterized in that The method further comprises: Inputting the image to be restored into an auxiliary model to obtain an auxiliary restored image, wherein the fidelity of the auxiliary image is greater than the fidelity of the restored image; performing anomaly detection on the restored image to detect abnormal areas in the restored image; determining an image fusion weight value for image fusion of the image to be restored and the auxiliary restored image according to the abnormal area; Based on the image fusion weight value, a weighted operation is performed on the pixel values ​​contained in the image to be restored and the pixel values ​​at corresponding positions in the auxiliary restored image to fuse the restored image and the auxiliary restored image, and the restored image is optimized.

10. The method according to claim 9, characterized in that Performing abnormality detection on the restored image to detect abnormal areas in the restored image includes one or more of the following combinations: performing error detection on sensitive areas in the restored image to detect error areas in the restored image; performing abnormal texture detection on the image texture in the restored image to detect an abnormal texture area in the restored image; Performing an image comparison between the restored image and the image to be restored to determine a difference area between the restored image and the image to be restored.

11. An image restoration device based on a diffusion model, characterized in that: The device comprises: an acquisition module, configured to acquire an image to be restored and perform semantic segmentation on the image to be restored to obtain at least one image region corresponding to different semantic categories; a determination module configured to determine, based on a semantic category corresponding to the at least one image region, a fidelity weight value for each pixel contained in the at least one image region; wherein the fidelity weight value determined for each image region is used to indicate a fidelity level of image restoration performed for the image region; and the fidelity weight value determined for each image region is positively correlated with a fidelity level of image content contained in the image region; a fusion module, configured to perform a weighted operation on pixel values ​​contained in the image to be restored and pixel values ​​at corresponding positions in the randomly generated noise image based on the fidelity weight value, so as to fuse the image to be restored with the randomly generated noise image to obtain an initialized noise image; A restoration module is configured to generate a prompt text based on the semantic category corresponding to the at least one image region, and convert the prompt text and the initialized noise image into an embedding vector that is input into a pre-trained diffusion model. The diffusion model then performs inference calculations on the initialized noise image based on the prompt text to complete image restoration of the image to be restored.

12. A user equipment, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images