Photo restoration method and related device
By obtaining the degree of damage of the photo and using the noise prediction model and diffusion model to perform noise addition and denoising, the problems of poor photo restoration effect and slow speed in the existing technology are solved, and high-quality and efficient photo restoration is achieved.
Patent Information
- Application Number
- CN202510770098.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-10
- Publication Date
- 2025-09-19
AI Technical Summary
Existing photo restoration technology is prone to problems such as blurring and distortion when processing severely damaged photos or photos with complex textures, and the restoration speed is slow.
By obtaining the damage degree value of the photo, the noise data is generated using the noise estimation model, and the noise addition and denoising are performed through the diffusion model to generate the repaired photo.
It improves the photo restoration effect, reduces the calculation amount and time of the restoration process, avoids information loss, and improves the restoration speed and quality.
Smart Images
Figure CN120672623A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a photo restoration method and related devices. Background Art
[0002] As an important record of life and history, photographs hold irreplaceable value. However, over time, some old photos may become damaged, such as torn, creased, or stained. To preserve the photo's aesthetic appeal and important information, photo restoration can be performed.
[0003] In related technologies, photo restoration solutions typically fall into two categories: one analyzes the pixels surrounding the area to be restored and fills these areas using an interpolation algorithm. However, this approach is prone to blurring and distortion in severely damaged photos or those with complex textures, resulting in poor restoration results. Another approach involves searching an image library for a similar image and using that image as a reference for restoration. However, this approach is computationally intensive and slow. Summary of the Invention
[0004] In view of this, an embodiment of the present application provides a photo restoration method and related devices, the purpose of which is to improve the photo restoration effect while increasing the photo restoration speed.
[0005] In a first aspect, an embodiment of the present application provides a method for restoring a photograph, the method comprising:
[0006] Obtaining a damage degree value of the photo to be restored; the damage degree value is used to represent the damage degree of the photo to be restored under the target damage type;
[0007] generating noise data corresponding to the photo to be restored using a noise estimation model based on the photo to be restored and the damage degree value of the photo to be restored; wherein the noise estimation model is trained based on training photos to be restored, the damage degree values of the training photos to be restored, and the training noise data, and the noise estimation model is used to correct the noise estimation of the photo to be restored based on the damage degree value of the photo to be restored;
[0008] performing noise processing on the photo to be restored based on the noise data to obtain a noisy photo to be restored;
[0009] The noise-added photo to be restored is denoised using a diffusion model to generate a restored photo.
[0010] In a possible implementation, performing denoising on the noisy photo to be restored by using a diffusion model to generate a restored photo includes:
[0011] The diffusion model is used to perform denoising on the noisy photo to be restored based on preset time step information to generate a restored photo; the preset time step information is used to indicate the number of times the diffusion model performs denoising on the noisy photo to be restored.
[0012] In one possible implementation, the noise estimation model is trained in the following manner:
[0013] Obtaining the training photo to be restored, the damage degree value of the training photo to be restored, and the training noise data;
[0014] generating predicted noise data based on the training photo to be restored and the damage degree value of the training photo to be restored by the first to-be-trained model;
[0015] Training the first to-be-trained model according to the training noise data and the predicted noise data;
[0016] When the first training cutoff condition is met, the training is terminated to obtain the noise estimation model.
[0017] In a possible implementation, obtaining the damage degree value of the photo to be restored includes:
[0018] A damage degree estimation model is used to generate a damage degree value of the photo to be restored based on the photo to be restored. The damage degree estimation model is trained based on training photos to be restored and training damage degree values of the training photos to be restored, and the training damage degree values are obtained by detecting the training photos to be restored.
[0019] In one possible implementation, the damage degree estimation model is trained in the following manner:
[0020] Obtaining the training photo to be restored and a training damage degree value of the training photo to be restored;
[0021] Generate a predicted damage degree value based on the training photos to be repaired by a second to-be-trained model;
[0022] training the second to-be-trained model according to the training damage degree value and the predicted damage degree value;
[0023] When the second training cutoff condition is met, the training is terminated to obtain the damage degree estimation model.
[0024] In a possible implementation, the target damage type includes at least one of the following damage types: blur level, noise level, color saturation, and hue distribution;
[0025] Among them, the damage degree value corresponding to the blur degree includes the blur sum value and the number of blur iterations; the damage degree value corresponding to the noise level includes the noise mean or the noise variance; the damage degree value corresponding to the color saturation includes the color concentration value; the damage degree value corresponding to the hue distribution includes the hue distribution statistical value.
[0026] In a possible implementation, the method further includes:
[0027] Based on the target features of the photo to be restored, post-restoration processing is performed on the restored photo to obtain a restored photo that retains the target features.
[0028] In a second aspect, an embodiment of the present application provides a photo restoration device, the device comprising:
[0029] A damage degree value acquisition module is used to obtain a damage degree value of the photo to be restored; the damage degree value is used to represent the damage degree of the photo to be restored under the target damage type;
[0030] a noise data generation module, configured to generate noise data corresponding to the photo to be restored using a noise estimation model based on the photo to be restored and the damage degree value of the photo to be restored; wherein the noise estimation model is trained based on training photos to be restored, the damage degree values of the training photos to be restored, and training noise data, and the noise estimation model is configured to correct a noise estimation of the photo to be restored based on the damage degree value of the photo to be restored;
[0031] a noise processing module, configured to perform noise processing on the photo to be restored based on the noise data to obtain a noisy photo to be restored;
[0032] The denoising processing module is used to perform denoising on the noisy photo to be restored through a diffusion model to generate a restored photo.
[0033] In a third aspect, an embodiment of the present application provides a photo restoration device, the device comprising a memory and a processor:
[0034] The memory is used to store a computer program and transmit the computer program to the processor;
[0035] The processor is used to execute the computer program so that the device performs the photo restoration method described in the first aspect.
[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, the device executing the computer program implements the photo restoration method described in the first aspect.
[0037] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0038] An embodiment of the present application provides a photo restoration method and a related device. In this method, a damage degree value of the photo to be restored is first obtained, and the damage degree value is used to characterize the damage degree of the photo to be restored under the target damage type; then, through a noise prediction model, based on the photo to be restored and the damage degree value of the photo to be restored, noise data corresponding to the photo to be restored is generated, wherein the noise prediction model is trained based on training photos to be restored, the damage degree values of the training photos to be restored, and the training noise data, and the noise estimation model is used to correct the noise estimation of the photo to be restored based on the damage degree value of the photo to be restored; then, the photo to be restored is denoised based on the noise data to obtain the noisy photo to be restored; and the noisy photo to be restored is denoised through a diffusion model to generate a restored photo.
[0039] In this way, the damage level value can specifically reflect the degree of damage to the photo to be restored under the target damage type. The damage level value can also serve as prior information, allowing the noise estimation model to correct the noise estimate of the photo to be restored, obtaining more accurate and targeted noise data. Subsequently, based on this noise data and the photo to be restored, data that can be input into the diffusion model can be constructed, allowing the diffusion model to use its own photo denoising capabilities to repair the photo to be restored with superimposed noise data, resulting in a photo with better restoration results. At the same time, using the trained noise estimation model and diffusion model, the photo to be restored can be restored directly through an end-to-end process from model input to model output, greatly shortening the restoration process and increasing the speed of photo restoration.
[0040] In addition, the diffusion model can be used to repair damage to the photo in one go, which can avoid the problem of information loss and help further improve the restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the following briefly introduces the drawings required for use in the embodiment or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0042] Figure 1 This is an application scenario of a photo restoration method provided in an embodiment of the present application;
[0043] Figure 2 A flowchart of a photo restoration method provided in an embodiment of the present application;
[0044] Figure 3 A schematic structural diagram of a photo restoration device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0046] Currently, there are two common approaches to photo restoration. One involves analyzing the pixels surrounding the area to be restored and filling it with interpolation algorithms. However, this approach is not effective for severely damaged photos or those with complex textures, as it can easily cause blurring and distortion. Another approach involves searching for similar images in an image library and using them as a reference for restoration. However, this approach is computationally intensive and slow.
[0047] Based on this, in order to solve the above problems, an embodiment of the present application provides a photo restoration method and related devices, in which the damage degree value of the photo to be restored is first obtained, and the damage degree value is used to characterize the damage degree of the photo to be restored under the target damage type; then, through a noise prediction model, based on the photo to be restored and the damage degree value of the photo to be restored, noise data corresponding to the photo to be restored is generated, wherein the noise prediction model is trained based on the training photo to be restored, the damage degree value of the training photo to be restored and the training noise data, and the noise estimation model is used to correct the noise estimation of the photo to be restored based on the damage degree value of the photo to be restored; then, the photo to be restored is denoised based on the noise data to obtain the noisy photo to be restored; and the noisy photo to be restored is denoised through a diffusion model to generate a restored photo.
[0048] In this way, the damage level value can specifically reflect the degree of damage to the photo to be restored under the target damage type, facilitating subsequent targeted restoration based on the damage type. Furthermore, the damage level value can serve as prior information, enabling the noise estimation model to correct its noise estimate of the photo to be restored, resulting in more accurate and targeted noise data. Subsequently, based on this noise data and the photo to be restored, data that can be input into the diffusion model can be constructed. This allows the diffusion model to leverage its own photo denoising capabilities to restore the photo to which the noise data is superimposed, resulting in a more effectively restored photo. Furthermore, by using the trained noise estimation and diffusion models, the photo to be restored can be restored directly from model input to model output, significantly shortening the restoration process and increasing the speed of photo restoration.
[0049] In addition, the diffusion model can be used to repair damage to the photo in one go, which can avoid the problem of information loss and help further improve the restoration effect.
[0050] For example, the embodiments of the present application can be applied to Figure 1 In the scenario shown, the scenario includes a database 101 and a terminal device 102, where the terminal device 102 can be a smartphone, a computer, a tablet, etc. Database 101 includes photos to be restored. Terminal device 102 can obtain the photos to be restored from database 101 and restore the photos to be restored using the implementation methods provided in the embodiments of the present application.
[0051] First, in the above application scenario, although the actions of the implementation provided by the embodiment of the present application are described as being executed by the terminal device 102, the embodiment of the present application is not limited in terms of the execution subject, as long as the actions disclosed in the implementation provided by the embodiment of the present application are executed. For example, the execution subject of the embodiment of the present application can also be a server, which can be an independent server, a cluster server, or a cloud server, etc.
[0052] Secondly, the above scenario is only an example scenario provided by the embodiment of the present application, and the embodiment of the present application is not limited to this scenario.
[0053] The specific implementation of the photo restoration method and related devices in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0054] See also Figure 2 , which is a flow chart of a photo restoration method provided by an embodiment of the present application, combined with Figure 2 Specifically, it may include:
[0055] S201: Obtaining a damage degree value of the photo to be repaired.
[0056] The damage degree value is used to represent the damage degree of the photo to be restored under the target damage type.
[0057] It should be understood that a photo may have one or more types of damage, and the repairs performed for different damage types will also be different. Therefore, targeted analysis of the degree of damage of the photo to be repaired under the target damage type will help with subsequent targeted repairs.
[0058] In a possible implementation, the target damage type may include at least one of the following damage types: blur level, noise level, color saturation, and hue distribution.
[0059] It should be noted that the target damage type may include more content than the above examples, and this application does not limit this. For ease of understanding, the following embodiments are described as if the target damage type includes all the damage types in the above examples, but this does not limit this application.
[0060] The degree of damage to the restored image under blur can be expressed using the blur sum value and the number of blur iterations, reflecting the loss of clarity in the restored image. The blur sum quantifies the blur level of each image, facilitating objective comparison of blur levels across different images. The number of blur iterations simulates the degree of blur in the restored image. For example, the blur sum value could be 5, the number of blur iterations could be 13, and the damage value would be (5, 13).
[0061] The degree of damage to the image to be restored under noise conditions can be expressed using the noise mean or noise variance as the damage degree value. Alternatively, both the noise mean and noise variance can be used, though this application does not limit this. By analyzing the noise mean and noise variance, the noise type of the image to be restored can be analyzed.
[0062] For example, if the noise mean is close to 0 and the noise variance is medium, it can be determined that the noise type of the photo to be restored is Gaussian noise; if the noise mean is an extreme value and the noise variance is extremely high, it can be determined that the noise type of the photo to be restored is salt and pepper noise.
[0063] The degree of damage to the restored photo under color saturation can be represented by a color density value. For example, the color density value range can be [0, 1], or [0%, 100%]. A higher value indicates a higher color saturation.
[0064] The damage degree of the to-be-restored photo under the tonal distribution can be measured using a statistical value of the tonal distribution as the damage degree value, for example, one or more of the tonal standard deviation, tonal skewness, and tonal kurtosis.
[0065] Exemplarily, the damage degree value of the photo to be repaired may include a blur sum value and a blur iteration number (5, 13), a noise mean value and a noise variance (5, 15), a color density value of 0.6, and a hue distribution statistics value of 80.
[0066] In addition, in some embodiments, the degree of damage of the photo to be repaired under the blur level may also include the damage location, and the degree of damage of the photo to be repaired under the color saturation may also include the damage range, etc. The same applies to the training damage degree values mentioned later and will not be repeated here.
[0067] In a possible implementation of the present application, S201 may specifically be: generating a damage degree value of the photo to be restored based on the photo to be restored by using a damage degree estimation model.
[0068] The damage degree estimation model is obtained by training based on training photos to be restored and training damage degree values of the training photos to be restored, and the training damage degree values are obtained by detecting the training photos to be restored.
[0069] The damage degree values obtained from the photos to be restored are used as training labels for training, which enables the model to learn how to perform damage assessment on the photos to be restored.
[0070] In some embodiments, the damage degree values corresponding to multiple different damage types can be used as training damage degree values, so that the damage degree estimation model can estimate the damage degree values corresponding to the multiple damage types of the photo to be repaired.
[0071] In this way, the trained damage degree estimation model can accurately estimate the damage degree of the photo to be restored, including the damage location, damage range, and damage severity. Based on this, it helps to make the subsequent restoration of the photo to be restored more targeted, and the restored photos can have more complete details, higher texture and color restoration, and the overall visual effect is closer to the original intact state, which also helps to achieve high-quality photo restoration.
[0072] In one possible implementation of the present application, the damage degree estimation model introduced above can be trained through the following steps: first obtain the training photos to be restored and the training damage degree values of the training photos to be restored; then, through the second model to be trained, generate the predicted damage degree values based on the training photos to be restored; then, according to the training damage degree values and the predicted damage degree values, train the second model to be trained; finally, when the second training cutoff condition is met, end the training to obtain the damage degree estimation model.
[0073] When obtaining training damage values, edge detection algorithms can be used to detect the training photos to be restored, obtaining damage values corresponding to the blur level of the training photos to be restored. Noise statistics can be used to calculate the noise level of the training photos to be restored, obtaining damage values corresponding to the noise level of the training photos to be restored. Color analysis tools can be used to analyze the color of the photos to be restored, obtaining damage values corresponding to the color saturation and hue distribution of the photos to be restored. The damage values for each damage type can be found in the above description and will not be repeated here.
[0074] The second model to be trained can be a Vector Quantized Generative Adversarial Network (VQGUN), or other deep learning networks, which is not limited in this application.
[0075] Based on the loss function, the difference between the training damage degree value and the predicted damage degree value can be calculated, and the iterative training process is repeated to make the difference between the training damage degree value and the predicted damage degree value smaller and smaller.
[0076] For example, the second training cutoff condition may be that the number of training times for the second to-be-trained model reaches a preset threshold number of times; or the model performance of the second to-be-trained model meets preset requirements, such as the difference between the training damage degree value and the predicted damage degree value meets a preset difference condition. This application is not limited to this.
[0077] In this way, through the above training process, the trained damage degree estimation model can be enabled to perform damage assessment on the photos to be repaired, laying the foundation for subsequent targeted repairs.
[0078] It should be understood that the sources of photos to be restored are diverse, and they may be downloaded from the Internet, uploaded from a network disk, etc. The sizes and resolutions of different photos to be restored vary greatly. In order to improve the model processing effect, the photos to be restored can be uniformly cropped and scaled.
[0079] For example, the photo to be restored may be cropped and scaled, etc., and uniformly adjusted to a size of 512×512, which is not limited in this application.
[0080] In some embodiments, the photo to be restored may also be normalized to normalize the pixel value of each pixel in the photo to be restored to [0, 1].
[0081] In this way, by preprocessing the photos to be restored as described above, the model input can be made more uniform, which helps to reduce the difficulty of model processing and improve the model processing effect.
[0082] S202: Generate noise data corresponding to the photo to be restored based on the photo to be restored and the damage degree value of the photo to be restored using a noise estimation model.
[0083] The noise estimation model is trained based on training photos to be restored, damage degree values of the training photos to be restored, and training noise data. The noise estimation model is used to correct the noise estimation of the photos to be restored based on the damage degree values of the photos to be restored.
[0084] In one possible implementation of the present application, the noise prediction model introduced above can be trained through the following steps: first, obtain the training photos to be restored, the damage degree values of the training photos to be restored, and the training noise data; then, through the first model to be trained, generate predicted noise data based on the training photos to be restored and the damage degree values of the training photos to be restored; then, train the first model to be trained based on the training noise data and the predicted noise data; when the first training cutoff condition is met, end the training to obtain the noise estimation model.
[0085] When obtaining the damage degree value of the training photo to be restored, the training photo to be restored can be input into the damage degree estimation model introduced above, and the damage degree estimation model can output the damage degree value of the training photo to be restored.
[0086] When obtaining training noise data, it can be obtained through the formula of the Denoising Diffusion Probabilistic Model (DDPM) (i.e., the diffusion model used later). For example, random noise can be sampled from a standard Gaussian distribution and then calculated based on the random noise and the cumulative noise scheduling parameter, which is not limited in this application.
[0087] The first model to be trained may also be the VQGUN introduced above, or other deep learning networks, which is not limited in this application.
[0088] For other implementations of the training process of the first model to be trained, please refer to the specific implementation of the second model to be trained introduced above, which will not be repeated here.
[0089] In this way, through the above training process, the damage assessment value can be used as prior information. When the noise estimation model performs noise estimation on the photo to be restored, it can correct the estimated result based on this prior information, thereby obtaining noise data that is more suitable for the photo to be restored.
[0090] S203: Noise the photo to be restored based on the noise data to obtain a noisy photo to be restored.
[0091] Noising the restored photo can be understood as constructing an intermediate state of the restored photo in the diffusion model. The difference is that while the diffusion model may require multiple noise additions to construct the intermediate states, this application only requires a single noise addition based on the noise data obtained.
[0092] In some embodiments, the photo to be restored may be subjected to noise processing based on the noise data using the following formula:
[0093]
[0094] in, is the photo to be restored after adding noise, y0 is the photo to be restored, f ω is the noise estimation model, f ω (y0,k M ) represents noise data, k M Indicates the Mth noise addition process, It is a parameter related to the time step, and is the parameter when the noise processing is performed on the photo to be restored for the Mth time. In this embodiment, it can be expressed as one time step, that is, one noise addition is performed.
[0095] For example, assuming that the diffusion model will add noise to the input data 20 times and then gradually denoise it 20 times to obtain the final output data, then the intermediate state constructed this time can be understood as the first denoising process on the photo to be restored to obtain the noisy photo to be restored, that is, directly constructing the data after denoising 19 times.
[0096] S204: De-noising the noisy photo to be restored using a diffusion model to generate a restored photo.
[0097] The denoising function carried by the diffusion model can be used to denoise the noisy photos to be restored. Because the noise data and the noise of the photos to be restored (that is, the data related to the degree of damage) will be integrated in the noisy photos to be restored, the diffusion model can process all the noise carried by the noisy photos to obtain the restored photos.
[0098] In a possible implementation of the present application, S204 may specifically include: performing denoising on the noisy photo to be restored based on preset time step information through a diffusion model to generate a restored photo.
[0099] The preset time step information is used to indicate the number of times the diffusion model performs denoising processing on the noisy photo to be restored.
[0100] For example, if the sampling step number of the diffusion model is 20, the preset time step information may be 19, indicating that the diffusion model has performed 19 denoising processes, and only one denoising process may be performed on the noisy photo to be restored.
[0101] As another example, in an example where the sampling step number of the diffusion model is 20 steps, the preset time step information may be 18 steps, indicating that the diffusion model has performed 18 denoising processes and the noisy photo to be restored may be subjected to two more denoising processes.
[0102] The preset time step information can be set based on the degree of damage to the photo to be restored. If the damage to the photo to be restored is severe, it can be subjected to two or more denoising processes. If the damage to the photo to be restored is relatively minor, it can be subjected to a single denoising process. This can be flexibly adjusted and is not limited in this application.
[0103] In some embodiments, the denoising process can be performed on the photo to be restored after adding noise using the following formula:
[0104]
[0105] Among them, g θ is a function determined by the noise estimation model, is the noise coefficient associated with the time step, is the noise coefficient when the i-th photo to be restored is denoised, y0 is the photo to be restored, k i-1 Indicates that after the i-1th denoising process, f ω is the noise estimation model. Assuming that two denoising processes are performed, two denoising processes need to be performed using the above formula. In the first denoising process, refers to the photo to be restored after adding noise. Refers to the photo to be restored after the first denoising process; the second denoising process requires the photo to be restored after the first denoising process. Refers to the photo to be restored after a denoising process. It refers to the photo to be restored after two denoising processes, and so on.
[0106] It should be understood that for some old photos (i.e., photos to be restored), some features on them may be what the user wants to keep, but when they are restored, they will be treated as damaged and repaired, causing the restored photos to lose these features.
[0107] In a possible implementation, the photo restoration method may further include: performing post-restoration processing on the restored photo based on target features of the photo to be restored, to obtain a restored photo that retains the target features.
[0108] The target feature may be a color coordination feature, a local detail feature, etc. of the photo to be restored. This application does not limit this and can be set according to user needs.
[0109] Taking the color saturation of the photo to be restored as an example, the photo to be restored is an old photo, and the low color saturation is a feature that the user wants to retain. However, after restoration, the color saturation of the photo is improved. At this time, the color concentration of the restored photo can be determined, and the color concentration of the restored photo can be adjusted based on the color concentration of the photo to be restored previously.
[0110] As can be seen from the above technical content, compared with the related art, this application has the following technical effects:
[0111] 1. The constructed damage degree estimation model can learn and analyze one or more damage types corresponding to the training damage degree values during training. It can accurately identify the damage degree of the target damage feature, including the severity, range, and location of the damage, and obtain a damage degree value that can accurately represent the damage degree of the photo to be restored.
[0112] 2. The constructed noise estimation model can use the damage degree value as prior information to estimate the noise of the photo to be restored. It can make corrections during the estimation process to obtain noise data that is more suitable for the photo to be restored.
[0113] 3. The diffusion model is based on the photo to be restored with superimposed noise data. The noise data is obtained based on the degree of damage. Therefore, it can accurately repair the photo based on the degree of damage to be restored, making the restored photo more complete in details, with a higher degree of texture and color restoration, and the overall visual effect closer to the original intact state, which helps to achieve high-quality photo restoration.
[0114] At the same time, it can be repaired in one step based on the preset time step information. On the one hand, it can effectively avoid the information loss caused by the independent repair operations in each stage of the traditional multi-stage repair method (for example, in the related technology, the damage of the photo to be repaired can be repaired separately based on the different types of damage). When dealing with complex textures and damaged areas, the image to be repaired can be reconstructed more efficiently. In other words, it simplifies the photo repair process. Through the various models introduced above, photo repair can be achieved from end to end, greatly reducing the amount of calculation and time consumption, and significantly improving the repair speed. This means that more photos to be repaired can be repaired in a shorter time, reducing time costs, and at the same time improving the overall repair quality, which helps to enhance the user experience.
[0115] On the other hand, the preset time step information can be flexibly adjusted, which can avoid the problems of insufficient restoration and excessive restoration for the photos to be restored, and help to further improve the restoration effect.
[0116] The above are some specific implementations of the photo restoration method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding photo restoration device. The photo restoration device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.
[0117] See also Figure 3 , which is a schematic diagram of the structure of a photo restoration device provided in an embodiment of the present application. The photo restoration device 300 may include:
[0118] The damage degree value acquisition module 310 is used to obtain the damage degree value of the photo to be restored; the damage degree value is used to represent the damage degree of the photo to be restored under the target damage type;
[0119] A noise data generation module 320 is configured to generate noise data corresponding to the photo to be restored using a noise estimation model based on the photo to be restored and the damage level value of the photo to be restored. The noise estimation model is trained based on the training photos to be restored, the damage level values of the training photos to be restored, and the training noise data. The noise estimation model is configured to correct the noise estimation of the photo to be restored based on the damage level value of the photo to be restored.
[0120] Noise processing module 330, configured to perform noise processing on the photo to be restored based on the noise data to obtain a noisy photo to be restored;
[0121] The denoising module 340 is used to perform denoising on the noisy photo to be restored by using a diffusion model to generate a restored photo.
[0122] As an implementation, the denoising processing module 340 may be specifically configured to:
[0123] The diffusion model is used to perform denoising on the noisy photo to be restored based on preset time step information to generate a restored photo. The preset time step information is used to indicate the number of denoising operations performed by the diffusion model on the noisy photo to be restored.
[0124] As an implementation, the noise estimation model is trained by the following units:
[0125] A first acquisition unit is used to acquire training photos to be restored, damage degree values of the training photos to be restored, and training noise data;
[0126] A first prediction unit is configured to generate predicted noise data based on the training photo to be restored and the damage degree value of the training photo to be restored by using the first to-be-trained model;
[0127] A first training unit is used to train a first to-be-trained model based on the training noise data and the predicted noise data;
[0128] The second acquisition unit is configured to terminate the training and obtain the noise estimation model when the first training cutoff condition is met.
[0129] As an implementation manner, the damage degree value obtaining module 310 may be specifically configured to:
[0130] A damage degree estimation model is used to generate a damage degree value of the photo to be restored based on the photo to be restored. The damage degree estimation model is trained based on the training photos to be restored and the training damage degree values of the training photos to be restored. The training damage degree values are obtained by detecting the training photos to be restored.
[0131] As an implementation method, the damage degree estimation model is trained by the following units:
[0132] a third obtaining unit, configured to obtain a training photo to be restored and a training damage degree value of the training photo to be restored;
[0133] A second prediction unit is configured to generate a predicted damage degree value based on the training photo to be restored using a second to-be-trained model;
[0134] A second training unit is used to train a second to-be-trained model according to the training damage degree value and the predicted damage degree value;
[0135] The fourth acquisition unit is configured to terminate the training and obtain a damage degree estimation model when the second training cutoff condition is met.
[0136] As an embodiment, the target damage type includes at least one of the following damage types: blur level, noise level, color saturation, and hue distribution;
[0137] Among them, the damage degree value corresponding to the blur degree includes the blur sum value and the number of blur iterations; the damage degree value corresponding to the noise level includes the noise mean or noise variance; the damage degree value corresponding to the color saturation includes the color concentration value; the damage degree value corresponding to the hue distribution includes the hue distribution statistical value.
[0138] As an embodiment, the photo restoration device 300 may further include:
[0139] The post-restoration processing module is used to perform post-restoration processing on the restored photo based on the target features of the photo to be restored, so as to obtain a restored photo that retains the target features.
[0140] The embodiments of the present application also provide corresponding photo restoration equipment and computer-readable storage media for implementing the solutions provided in the embodiments of the present application.
[0141] Among them, the photo restoration device includes a memory and a processor, the memory is used to store computer programs, and the processor is used to execute computer programs so that the device executes the photo restoration method of any embodiment of the present application.
[0142] A computer program is stored in a computer-readable storage medium. When the computer program is executed, the device executing the computer program implements the photo restoration method of any embodiment of the present application.
[0143] The "first" and "second" (if any) in the names mentioned in the embodiments of this application are only used as name identifiers and do not mean the first or second in order.
[0144] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a readable storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or certain parts of the embodiments of the present application.
[0145] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.
[0146] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A photo restoration method, characterized in that: The method comprises: Obtaining a damage degree value of the photo to be restored; the damage degree value is used to represent the damage degree of the photo to be restored under the target damage type; generating noise data corresponding to the photo to be restored using a noise estimation model based on the photo to be restored and the damage degree value of the photo to be restored; wherein the noise estimation model is trained based on training photos to be restored, the damage degree values of the training photos to be restored, and the training noise data, and the noise estimation model is used to correct the noise estimation of the photo to be restored based on the damage degree value of the photo to be restored; performing noise processing on the photo to be restored based on the noise data to obtain a noisy photo to be restored; The noise-added photo to be restored is denoised using a diffusion model to generate a restored photo.
2. The method according to claim 1, characterized in that The step of performing denoising on the noisy photo to be restored by using a diffusion model to generate a restored photo includes: The diffusion model is used to perform denoising on the noisy photo to be restored based on preset time step information to generate a restored photo; the preset time step information is used to indicate the number of times the diffusion model performs denoising on the noisy photo to be restored.
3. The method according to claim 1, characterized in that The noise estimation model is trained as follows: Obtaining the training photo to be restored, the damage degree value of the training photo to be restored, and the training noise data; generating predicted noise data based on the training photo to be restored and the damage degree value of the training photo to be restored by the first to-be-trained model; Training the first to-be-trained model according to the training noise data and the predicted noise data; When the first training cutoff condition is met, the training is terminated to obtain the noise estimation model.
4. The method according to claim 1, wherein The step of obtaining the damage degree value of the photo to be restored includes: A damage degree estimation model is used to generate a damage degree value of the photo to be restored based on the photo to be restored. The damage degree estimation model is trained based on training photos to be restored and training damage degree values of the training photos to be restored, and the training damage degree values are obtained by detecting the training photos to be restored.
5. The method according to claim 4, characterized in that The damage degree estimation model is trained as follows: Obtaining the training photo to be restored and a training damage degree value of the training photo to be restored; Generate a predicted damage degree value based on the training photos to be repaired by a second to-be-trained model; training the second to-be-trained model according to the training damage degree value and the predicted damage degree value; When the second training cutoff condition is met, the training is terminated to obtain the damage degree estimation model.
6. The method according to claim 1, characterized in that The target damage type includes at least one of the following damage types: blur level, noise level, color saturation, and hue distribution; Among them, the damage degree value corresponding to the blur degree includes the blur sum value and the number of blur iterations; the damage degree value corresponding to the noise level includes the noise mean or the noise variance; the damage degree value corresponding to the color saturation includes the color concentration value; the damage degree value corresponding to the hue distribution includes the hue distribution statistical value.
7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Based on the target features of the photo to be restored, post-restoration processing is performed on the restored photo to obtain a restored photo that retains the target features.
8. A photo restoration device, characterized in that: The device comprises: A damage degree value acquisition module is used to obtain a damage degree value of the photo to be restored; the damage degree value is used to represent the damage degree of the photo to be restored under the target damage type; a noise data generation module, configured to generate noise data corresponding to the photo to be restored using a noise estimation model based on the photo to be restored and the damage degree value of the photo to be restored; wherein the noise estimation model is trained based on training photos to be restored, the damage degree values of the training photos to be restored, and training noise data, and the noise estimation model is configured to correct a noise estimation of the photo to be restored based on the damage degree value of the photo to be restored; a noise processing module, configured to perform noise processing on the photo to be restored based on the noise data to obtain a noisy photo to be restored; The denoising processing module is used to perform denoising on the noisy photo to be restored through a diffusion model to generate a restored photo.
9. A photo restoration device, characterized in that: The device includes a memory and a processor: The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute the computer program to enable the device to perform the steps of the photo restoration method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program. When the computer program is executed, a device executing the computer program implements the steps of the photo restoration method according to any one of claims 1 to 7.