Image fidelity measurement method and device, electronic equipment and storage medium

Abstract

The invention provides an image fidelity measurement method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a first wavelet domain coefficient WHQ of an original image, a second wavelet domain coefficient WLQ of a degraded image and a third wavelet domain coefficient WT of a restored image; wherein the degraded image is an image obtained after the original image is degraded, and the restored image is an image obtained after the degraded image is restored; attenuating the first wavelet domain coefficient WHQ based on the second wavelet domain coefficient WLQ to obtain a first attenuated wavelet coefficient WLQA; attenuating the third wavelet domain coefficient WT based on the first attenuation wavelet coefficient WLQA to obtain a second attenuation wavelet coefficient WTA; and determining the fidelity of the restored image based on the first attenuation wavelet coefficient WLQA and the second attenuation wavelet coefficient WTA. According to the invention, the accuracy of image fidelity evaluation is improved, the improvement and optimization of an image restoration algorithm can be guided, and the usability and universality are high.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image fidelity measurement method and device, an electronic device, and a storage medium. Background Art

[0002] In practical applications, it is often necessary to restore degraded low-quality images to high-quality images to improve the quality of displayed images, such as image deblurring, image noise removal, image super-resolution, Joint Photographic Experts Group (JPEG) distortion removal, and repairing the combined degradation of multiple degradations.

[0003] At present, the traditional full reference image quality index (Full Reference) can be used to calculate the image quality index by comparing the difference between the repaired image and the original image. However, the full reference image quality index does not take into account the attenuation of the original image due to image degradation, so the fidelity evaluation score is low, especially the image quality index of the heavily degraded repaired image is seriously low. In addition, the original high-quality image needs to be known to evaluate the index, so it can only be used to simulate degradation scenes, not real degradation scenes.

[0004] In addition, the traditional no-reference image quality (No Reference) can be used to evaluate the image quality alone and obtain an image quality score similar to the human eye image quality evaluation, but this method cannot evaluate the fidelity of the restored image. Summary of the invention

[0005] In view of this, the present specification provides an image fidelity measurement method and device, an electronic device, and a storage medium.

[0006] According to a first aspect of the present specification, a method for measuring image fidelity is provided, the method being performed by an electronic device, the method comprising:

[0007] Determine the first wavelet domain coefficient W of the original image HQ , the second wavelet domain coefficient W of the degraded image LQ And the third wavelet domain coefficient W of the restored image T ; wherein the degraded image is an image obtained after the original image is degraded, and the repaired image is an image obtained after the degraded image is repaired;

[0008] Based on the second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A ;

[0009] Based on the first attenuated wavelet coefficient W LQ_AFor the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient W T_A ;

[0010] Based on the first attenuated wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A , determine the fidelity of the repaired image.

[0011] According to a second aspect of the present specification, there is provided an image fidelity measurement device, the device comprising:

[0012] The coefficient determination module is used to determine the first wavelet domain coefficient W of the original image. HQ , the second wavelet domain coefficient W of the degraded image LQ And the third wavelet domain coefficient W of the restored image T ; wherein the degraded image is an image obtained after the original image is degraded, and the repaired image is an image obtained after the degraded image is repaired;

[0013] The first attenuation module is used to attenuate the second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A ;

[0014] The second attenuation module is used to attenuate the wavelet coefficient W based on the first attenuation LQ_A For the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient W T_A ;

[0015] A measuring module is used for measuring the first attenuation wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A , determine the fidelity of the repaired image.

[0016] According to a third aspect of this specification, an electronic device is provided, the device comprising:

[0017] One or more processors; wherein the processor is used to execute the steps of any method described in the first aspect.

[0018] According to a fourth aspect of the present specification, there is provided a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any one of the first aspects.

[0019] According to a fifth aspect of the present specification, there is provided a computer program product, comprising a computer program / instruction, which implements the steps of the method as described in any one of the first aspects when executed by a processor.

[0020] The technical solution provided in this specification may have the following beneficial effects:

[0021] In this specification, the wavelet domain coefficients of the original image can be attenuated based on the wavelet domain coefficients of the degraded image to obtain the equivalent first attenuation coefficient, and the wavelet domain coefficients of the repaired image can be attenuated based on the first attenuation coefficient to obtain the second attenuation coefficient, and finally the image fidelity is evaluated based on the second attenuation coefficient and the first attenuation coefficient. The accuracy of image fidelity evaluation is improved, and the improvement and optimization of image repair algorithms can be guided, with high usability and versatility.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated in and constitute a part of this specification, illustrate aspects of the specification, and together with the description, serve to explain the principles of the specification.

[0024] Figure 1 A schematic diagram of image degradation and image restoration is provided.

[0025] Figure 2 One of the flow charts of an image fidelity measurement method provided is shown.

[0026] Figure 3 This is the second flowchart of an image fidelity measurement method provided.

[0027] Figure 4 This is the third flowchart of an image fidelity measurement method provided.

[0028] Figure 5 A schematic diagram of fidelity measurement in a simulated degradation scenario is provided.

[0029] Figure 6 It is a schematic diagram of the attenuation image extraction model training in a simulated degradation scenario.

[0030] Figure 7 It is a schematic diagram of fidelity measurement in a real degradation scenario.

[0031] Figure 8 The invention provides a structural schematic diagram of an image fidelity measurement device.

[0032] Fig. 9A structural schematic diagram of an electronic device is provided. DETAILED DESCRIPTION

[0033] The invention will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described below by way of example do not represent all embodiments consistent with this specification. Instead, they are merely examples of apparatus and methods consistent with some aspects of this specification as detailed in the appended claims.

[0034] Before introducing the solutions provided by the present disclosure, the nouns and / or terms involved in the present disclosure are first introduced.

[0035] Fidelity measurement: Measure and evaluate the consistency between the restored image and the original image. The higher the consistency, the higher the fidelity.

[0036] Image restoration: Use image processing algorithms to process degraded images and output restored high-quality images.

[0037] Image degradation modeling: Modeling the image degradation process makes it easier to repair degraded images.

[0038] Wavelet transform: An alternative to the Fourier transform for time and / or frequency analysis.

[0039] There are many algorithms and models in the field of image restoration that can repair single-type or combined degradation. The restored image must not only have high quality, but also be consistent with the original image in content and color. The consistency with the original image is usually called fidelity.

[0040] Taking the traditional full reference image quality index (Full Reference) as an example, it calculates image quality indicators by comparing the differences between the restored image and the original image, such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), Deep Image Similarity with Unsupervised Learning of Distances (DISTS), etc.

[0041] The traditional full-reference image quality index has two major flaws. First, it does not take into account the attenuation of the original image due to image degradation. For example, if 50% of an image has degraded to the point where it cannot be repaired, then if the fidelity is still determined based on the original image and the repaired image, the fidelity evaluation score will be low, especially the image quality index of the heavily degraded repaired image will be seriously low. Second, the original high-quality image must be known to evaluate the index, so it can only be used to simulate degraded scenes, while the original high-quality image may not be determined in real degraded scenes. Therefore, the full-reference image quality index is not suitable for real degraded scenes.

[0042] For traditional no-reference image quality (No Reference), it only supports the independent evaluation of image quality and obtains an image quality score similar to the image quality evaluation of the human eye, but it cannot evaluate the fidelity of the restored image.

[0043] In order to accurately evaluate the fidelity index in general scenarios, the present disclosure provides the following image fidelity measurement method and device, electronic device and storage medium.

[0044] In the present disclosure, a method for measuring the fidelity of image restoration based on wavelet domain image degradation modeling is proposed, which is applicable not only to simulated degradation scenarios but also to real degradation scenarios. For simulated degradation scenarios, the wavelet domain attenuation of the image during the degradation process is analyzed, and a more accurate fidelity index is obtained by calculating the consistency of the attenuated image; for real degradation scenarios, a deep learning model is trained to convert the degraded image into an image of the attenuated signal, and the fidelity index evaluation of the real degradation scenario is realized for the first time. The method provided by the present disclosure is highly versatile and applicable to different types of single or combined degradations. It can accurately evaluate the fidelity of the restored image, thereby guiding the improvement and optimization of the image restoration algorithm.

[0045] For example Figure 1 As shown, the original high-quality image is called the original image, or the high-quality image I HQ , the degraded image can also be called a low-quality image I LQ , the repaired image can be called the test image I T .

[0046] For example, assuming that the original high-quality image I HQ Some information in the image has been completely lost, for example, the information of one flower has been completely lost. T The petals of the flower can be completely closed, slightly open, or fully open, but no matter what T The restoration state of the upper petals should not affect the evaluation result of the image fidelity. The reason is that since the information of this part has been completely lost, when evaluating the fidelity, the other parts should be evaluated, such as the information of other flowers that have not been lost relative to the original high-quality image I.HQ The fidelity of the image should not ultimately affect the fidelity assessment result of the information in the non-lost part relative to the original high-quality image.

[0047] In this disclosure, the measurement process of image fidelity in simulated degradation scenarios is first introduced.

[0048] Figure 2 FIG. 1 is a flowchart of an image fidelity measurement method provided. Figure 2 As shown, the method may be executed by an electronic device, and the electronic device may be, for example but not limited to, a server, a terminal, other electronic devices, etc. The method includes the following steps:

[0049] Step S201, determine the first wavelet domain coefficient W of the original image HQ , the second wavelet domain coefficient W of the degraded image LQ And the third wavelet domain coefficient W of the restored image T .

[0050] For example, in the simulated degradation scenario, the original image I HQ For a known image, the electronic device can HQ Perform random degradation to obtain the degraded image I LQ .

[0051] In one example, the degradation type may include but is not limited to at least one of the following: Gaussian blur, downsampling, Gaussian noise, Poisson noise, JPEG compression distortion, etc. Of course, it may also be a combination of the above types, which is not limited in the present disclosure.

[0052] For example, by HQ By performing random simulated degradation, high-quality image I can be obtained. HQ , and the high quality image I HQ Paired (or matched) degraded images I LQ , so that the full reference image quality index can be evaluated later.

[0053] For example, the degraded image I LQ Perform repair and obtain the repaired image I T The restoration method may include but is not limited to reducing or even removing noise in the degraded image, adjusting image clarity, enhancing image quality, etc. The present disclosure does not limit the restoration method.

[0054] For example, the original image I HQ , degraded image I LQ , repair image I TPerform wavelet transforms, such as discrete wavelet transform (DWT), respectively, to determine the corresponding wavelet domain coefficients, namely the first wavelet domain coefficients W HQ , the second wavelet domain coefficient W LQ And the third wavelet domain coefficient W T .

[0055] In one example, wavelet transform has multiple roles in signal processing, mainly including the following aspects:

[0056] Multi-resolution analysis: Wavelet transform has the characteristics of multi-resolution (multi-scale), which can gradually refine the analysis of signals and observe the details of signals from coarse to fine. This characteristic enables wavelet transform to adapt to the needs of signal analysis of different frequencies and provide more detailed signal processing capabilities.

[0057] Time-frequency localized analysis: Compared with Fourier transform, wavelet transform has the ability of localized analysis in both time domain and frequency domain. It refines the signal at multiple scales through scaling and translation operations, and can achieve time subdivision at high frequencies and frequency subdivision at low frequencies, thus automatically adapting to the requirements of time-frequency signal analysis. This feature enables wavelet transform to better handle non-stationary signals.

[0058] Signal and Image Processing: Wavelet transform is widely used in signal and image processing. It can be used for image compression, denoising, feature extraction, etc. By decomposing the image into four components at different scales, wavelet transform can enhance or blur specific image features and improve the effect of image processing.

[0059] Detecting transient or singular points of signals: Wavelet transform can have good characterization capabilities in both time domain and frequency domain by selecting appropriate basic wavelets, which is conducive to detecting transient or singular points of signals. This feature makes wavelet transform important in the fields of fault diagnosis and signal analysis.

[0060] Compared with Fourier transform, wavelet transform can better handle non-stationary signals because it has the ability of localized analysis in both time domain and frequency domain.

[0061] For example, the original image I can be transformed into HQ , degraded image I LQ , repair image I T Wavelet transform is performed separately to obtain wavelet coefficients of multiple scales and different directions.

[0062] In one example, the wavelet basis function may be an orthogonal wavelet basis function, such as a SYM wavelet or a DB wavelet.

[0063] Among them, SYM wavelet is a symmetric wavelet, also known as Symlets wavelet. It is mainly used in signal processing and image processing. SYM wavelet has symmetry, which means that it can effectively avoid phase distortion in image processing because its corresponding filter has the characteristics of linear phase.

[0064] DB wavelet, also known as Daubechies wavelet, has high vanishing moment, which means they can make more wavelet coefficients zero or generate fewer non-zero wavelet coefficients, which is very beneficial for data compression and noise removal.

[0065] The above is only an exemplary description, and the present disclosure does not limit the wavelet basis function used in the wavelet transform.

[0066] Step S202: based on the second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A .

[0067] For example, the second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and obtain the equivalent attenuated signal, that is, the first attenuated wavelet coefficient W LQ_A , thereby achieving modeling of image degradation.

[0068] In one example, the attenuation process includes two stages of attenuation. First, the second wavelet domain coefficient W LQ Decomposed into the first attenuated signal W LQ_A1 (i), and the additive noise W N (i).

[0069] The first attenuation signal is a signal of the first wavelet domain coefficient W HQ The attenuated signal obtained by attenuation is W HQ attenuation signal.

[0070] Furthermore, the additive noise W N (i) Convert to obtain the second attenuated signal W LQ_A (i), that is, for the first wavelet domain coefficient W HQ The equivalent attenuated signal W of the two-stage attenuation LQ_A (i).

[0071] In one example, the second wavelet domain coefficient W LQ Decomposed into the first attenuated signal W LQ_A1 (i), and the additive noise W N The process of (i) includes:

[0072] First, for each first wavelet domain coefficient W HQ , the second wavelet domain coefficient W LQ Divide into N small blocks of size B×B, each small block is denoted by W HQ(i) , W LQ(i) , where i represents the small block number, i=1, 2, 3, ...N.

[0073] Exemplarily, each small block may correspond to a pixel on the image. The present disclosure does not limit the values ​​of B and N.

[0074] Furthermore, the image degradation process is modeled as signal attenuation and additive noise in the wavelet domain using Formula 1:

[0075] W LQ(i) = λ 1(i) × W HQ(i) + W N(i) Formula 1

[0076] Among them, λ 1(i) is the attenuation coefficient corresponding to the i-th small block. This disclosure mainly considers global image degradation such as Gaussian blur, downsampling, Gaussian noise, Poisson noise, and JPEG compression distortion. The degradation degree of each small block is the same, corresponding to the same attenuation coefficient, denoted as λ1. N(i) is the noise of the ith small block. λ1 and W N(i) It can be calculated by formula 2 and formula 3:

[0077] λ1=∑ i Cov(W LQ (i),W HQ (i)) / ∑ i Cov(W HQ (i),W HQ (i)) Formula 2

[0078] W N(i) = W LQ(i) -λ1×W HQ(i) Formula 3

[0079] In the above formula, Cov() is the covariance of two vectors, and the calculated one-stage attenuation signal is W LQ_A1 (i) For example, formula 4:

[0080] W LQ_A1(i) = λ1 × W HQ(i) Formula 4

[0081] In one example, the process of converting the additive noise WN(i) to obtain the second attenuated signal WLQ_A(i) is as follows:

[0082] After a stage of attenuation, the second wavelet domain coefficient W LQ It can be decomposed into: LQ(i) =W LQ_A1(i) +W N(i) , where W LQ_A1(i) and W N(i) It can be modeled as a multidimensional Gaussian distribution. N(i) This will lead to the introduction of uncertainty in the original signal. The introduced uncertainty can be equivalently converted into a two-stage signal attenuation, that is, the second attenuated signal W LQ_A (i), as shown in Formula 5:

[0083]

[0084] In the above formula, s(i) 2 Σ LQ_A1 is a Gaussian random variable W LQ_A1(i) The covariance of is the noise variance, I is the identity matrix, s(i) 2 ,Σ LQ_A1 and The calculations are shown in Equation 6 to Equation 8:

[0085]

[0086] Where M is W LQ_A (i) The number of elements in the vector, W LQ_A (i) T It is W LQ_A The transposed vector of (i).

[0087] In one example, based on the first attenuation signal W LQ_A1 (i) and the second attenuated signal W LQ_A (i) Determine the first attenuation coefficient W LQ_A , as shown in Formula 9:

[0088]

[0089] It can be understood that through the first stage attenuation of the above process, the degraded image I can be converted into LQ The image is decomposed into a blurred image (only blur) and a noise image (only noise). The noise image can be equivalent to a blurred image through two-stage attenuation. Finally, the image noise is removed through two-stage attenuation, and only the blur in the image is retained.

[0090] Step S203: based on the first attenuation wavelet coefficient W LQ_A For the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient WT_A .

[0091] For example, based on the first attenuated wavelet coefficient W LQ_A For the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient W T_A , so as to calculate the image fidelity later.

[0092] The electronic device may be based on the first attenuation wavelet coefficient W LQ_A and the third wavelet domain coefficient W T , determine the degradation factor λ T , further, based on the degradation factor λ T and the third wavelet domain coefficient W T , determine the second attenuated wavelet coefficient W T_A , where the second attenuated wavelet coefficient W T_A The calculation of can be shown in formula 10, for example, the degradation factor λ T The calculation of can be shown as formula 11:

[0093] WT_A(i) = λT * WT(i) Formula 10

[0094] λ T =∑ i Cov(W T (i),W LQ_A (i)) / ∑ i Cov(W T (i),W T (i)) Formula 11

[0095] Step S204: based on the first attenuation wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A , determine the fidelity of the repaired image.

[0096] For example, the electronic device may respectively calculate the first attenuation wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A Perform inverse wavelet transform to obtain the first attenuation image I LQ_A and the second attenuation image I T_A .

[0097] The inverse wavelet transform is the reverse process of the wavelet transform, and the image is obtained by filtering and transforming the wavelet coefficients. The inverse wavelet transform process can use the same wavelet basis function as the aforementioned DWT process. The inverse wavelet transform can be an inverse discrete wavelet transform (IDWT).

[0098] Furthermore, the electronic device may use the root mean square difference to evaluate the first attenuation image I LQ_A and the second attenuation image I T_A The difference between them is used as the repair image I T The fidelity index is as described in Formula 12:

[0099]

[0100] In the above formula, p represents the label of each image pixel, and P represents the total number of image pixels.

[0101] The present disclosure can improve the accuracy of image fidelity assessment in simulated degradation scenarios, and has high usability and versatility.

[0102] Next, we will introduce the process of measuring image fidelity in real degradation scenarios.

[0103] In real-world image degradation scenarios, only degraded low-quality images can be obtained. LQ (i.e. degraded image), the original high-quality image I cannot be obtained HQ Existing image quality indices can only evaluate the quality of the restored image, but cannot evaluate the fidelity of the restored image.

[0104] Among them, it is proposed to use an artificial intelligence (AI) model to replace the attenuation image extraction process. This model can be implemented by a variety of different types of deep learning models. LQ Convert to the first attenuation image I LQ_A , to avoid the original image I HQ rely.

[0105] The AI ​​model can also be called the “attenuated image extraction model”, which can be used to convert the original high-quality image I HQ Convert to attenuation image I LQ_A .

[0106] Attenuation Image I LQ_A is the degraded image I LQ Perform noise removal and blur operations. The attenuated image extraction model is recommended to use a deep learning model similar to noise removal, such as the U-shaped Network (UNet), the Shifted Window Transformer (SWIN) model, or its improved version.

[0107] The training process of the AI ​​model is as follows: Figure 3 As shown, the following steps are included:

[0108] Step S301, determining each degraded image I in a plurality of degraded images LQ The corresponding first attenuation image I LQ_A .

[0109] For example, the electronic device may process at least one original image I HQ Use different degradation methods to determine each original image I HQ Corresponding to at least one degraded image I LQ Furthermore, we can use each degraded image I LQ Repair and obtain the corresponding repaired image I T .

[0110] Furthermore, the electronic device may determine, based on the method of steps S2101 to S204, that each degraded image I LQ The corresponding first attenuation image I LQ_A The specific process will not be described here.

[0111] Step S302: using the multiple degraded images as input values ​​of the initial AI model, obtaining the output of the initial AI model and the corresponding image I of each degraded image. LQ The corresponding third attenuation image I LQ_A_p .

[0112] For example, the initial AI model can adopt a deep learning model that supports noise removal, such as UNet, SWIN, etc.

[0113] Step S303: based on each third attenuation image I LQ_A_p and each first attenuation image I LQ_A The difference between them determines the loss function.

[0114] For example, the loss function Loss may be L1 Loss, which may be the absolute difference between the predicted value and the actual value. In the present disclosure, it may be each third attenuated image I LQ_A_p and each first attenuation image I LQ_A The absolute difference between .

[0115] For example, Loss may be L2 Loss, which may be the square difference between the predicted value and the actual value, where each third attenuated image I LQ_A_p and each first attenuation image I LQ_A The squared difference between .

[0116] The present disclosure does not limit the method for determining the loss function.

[0117] Step S304, training the initial AI model based on the loss function, and obtaining a trained target AI model when the training stop condition is met.

[0118] For example, the training process of the initial AI model includes but is not limited to adjusting the parameters of each layer of the model and reducing the loss function.

[0119] For example, the training stop condition may include but is not limited to at least one of the following: the loss function is reduced to a tolerance range, a preset training cycle is reached, and the learning rate reaches a preset value.

[0120] For example, when the training stop condition is met, the training is stopped to obtain the target AI model, which is the attenuated image extraction model.

[0121] The present invention can obtain a target AI model by training a deep learning model to extract attenuated images, thereby improving the feasibility of image fidelity assessment in real degradation scenarios.

[0122] The measurement process of image fidelity in real degradation scenarios is as follows: Figure 4 As shown, the method may be executed by an electronic device, and the electronic device may be, for example but not limited to, a server, a terminal, other electronic devices, etc. The method includes the following steps:

[0123] Step S401: input a real degraded image into the target AI model to obtain a first estimated attenuation image output by the target AI model.

[0124] For example, the electronic device may input a real degradation image into the target AI model to obtain a first estimated attenuation image output by the target AI model.

[0125] Step S402: determining the fourth wavelet domain coefficient W of the first estimated attenuation image LQ_A_P , and the third wavelet domain coefficient W of the true restoration image T .

[0126] For example, DWT may be performed on the first estimated attenuation image and the true restoration image to determine the fourth wavelet domain coefficient W of the first estimated attenuation image. LQ_A_P , and the third wavelet domain coefficient W of the true restoration image T .

[0127] Step S403: based on the fourth attenuated wavelet coefficient W LQ_A_P The third wavelet domain coefficient W of the true restoration image T Attenuate and get the third attenuated wavelet coefficient W T_A '.

[0128] For example, the electronic device may be based on the fourth attenuated wavelet coefficient W LQ_A_P And the third wavelet domain coefficient W of the true restoration image T , determine the degradation factor λT ', and then based on the degradation factor λ T ' and the third wavelet domain coefficient W T , determine the third attenuated wavelet coefficient W T_A The determination process, such as the implementation process of step S203, will not be described in detail here.

[0129] Step S404: the third attenuated wavelet coefficient W T_A 'Perform inverse wavelet transform to obtain the third attenuated image.

[0130] For example, the electronic device may calculate the third attenuated wavelet coefficient W T_A 'Perform inverse wavelet transform to obtain the third attenuation image.

[0131] Step S405: determining the fidelity of the actual restored image based on the third attenuation image and the first estimated attenuation image.

[0132] For example, the electronic device may use the mean square error to evaluate the difference between two images based on Formula 12, thereby determining the fidelity of the true restoration image.

[0133] The present disclosure can improve the accuracy of image fidelity assessment in real degradation scenarios, and has high usability and versatility.

[0134] The above process is further illustrated below with examples.

[0135] Example 1, fidelity measurement of simulated degradation scenarios.

[0136] 1.1, System architecture.

[0137] In the simulated degradation scenario, the original high-quality image I HQ It is known that HQ Perform random degradation to obtain low-quality degraded image I LQ The image restoration task is to input the degraded image I LQ , use the image restoration algorithm to reconstruct the high-quality image and generate the restored image I T .

[0138] To evaluate the inpainted image I T The traditional full reference image quality index is compared with I T and I HQ The difference between the two is used to calculate the image quality indicators, such as PSNR, SSIM, IPLPS, DISTS, etc. The traditional no-reference image quality is evaluated separately. TThe image quality of the image is evaluated by the human eye, and the image quality score is obtained. The full reference image quality index can reflect the fidelity and quality of the restored image at the same time, but its fidelity evaluation is too strict and is not suitable for the evaluation of severely degraded restored images; the no reference image quality index only evaluates the image quality and cannot evaluate the fidelity. In order to more accurately evaluate the fidelity of restored images with different degradation levels, the I HQ ,I LQ and I T , and obtain a fidelity (FID) assessment that is compatible with severe, moderate, and mild degradation.

[0139] For example Figure 5 As shown, first, I HQ ,I LQ and I T Perform discrete wavelet transform (DWT) to obtain the corresponding wavelet domain coefficient W HQ , W LQ and W T Then according to W HQ , W LQ The degradation is modeled and analyzed to obtain the wavelet coefficient W of the equivalent attenuation image LQ_A ; Then to W T Perform corresponding attenuation and fit to match W LQ_A , and get the attenuated wavelet coefficient W T_A ; Finally, W LQ_A , W T_A Perform inverse wavelet transform to obtain image I LQ_A ,I T_A , and calculate the root mean square error to get the fidelity.

[0140] 1.2, random degradation.

[0141] Image degradation types include Gaussian blur, downsampling, Gaussian noise, Poisson noise, JPEG compression distortion, and other types, as well as the combination of multiple types of degradation. Randomly simulating the degradation of the original high-quality image can obtain pairs of high-quality and low-quality corresponding images, which is convenient for the evaluation of the full reference image quality index. This solution also uses random degradation to generate high-quality and low-quality corresponding images for calculating more accurate fidelity indicators.

[0142] 1.3, wavelet transform DWT.

[0143] Wavelet transform is to filter and transform the image I through wavelet basis functions to obtain multi-scale and different-direction wavelet subband coefficients W. This solution recommends using orthogonal wavelet basis, such as SYM and DB wavelets.

[0144] 1.4, Inverse wavelet transform IDWT.

[0145] Inverse wavelet transform is the reverse process of wavelet transform, which performs filtering transform on wavelet coefficients W to obtain image I. Inverse wavelet transform uses the same wavelet basis function as wavelet transform.

[0146] 1.5, attenuation image extraction.

[0147] According to W LQ To W HQ Attenuate and get the equivalent attenuated signal W LQ_A , to achieve modeling of image degradation. In the present disclosure, signal attenuation includes two stages of attenuation. First, W LQ Decompose into W HQ The attenuated signal and additive noise are obtained for W HQ The first-stage attenuation of W is obtained by analyzing the additive noise. HQ The second-stage equivalent decay of .

[0148] 1.5.1, Decomposition of attenuated signal and additive noise.

[0149] First, for each wavelet subband W HQ , W LQ The wavelet coefficients are divided into N small blocks of size B×B, each of which is denoted by W HQ(i) , W LQ(i) , where i represents the small block number, i=1, 2, 3…N.

[0150] Based on Formula 1, the image degradation process is modeled as signal attenuation and additive noise in the wavelet domain.

[0151] W LQ(i) = λ 1(i) × W HQ(i) + W N(i) Formula 1

[0152] Among them, λ 1(i) is the attenuation coefficient corresponding to the i-th small block, where global image degradation such as Gaussian blur, downsampling, Gaussian noise, Poisson noise, and JPEG compression distortion are mainly considered. The degradation degree of each small block is the same, corresponding to the same attenuation coefficient, denoted as λ1. N(i) is the noise of the ith small block. λ1 and W N(i) It can be calculated by formula 2 and formula 3:

[0153] λ1=∑ i Cov(W LQ (i),W HQ (i)) / ∑ i Cov(W HQ (i),W HQ (i)) Formula 2

[0154] W N(i) = W LQ(i) -λ1×W HQ(i) Formula 3

[0155] In the above formula, Cov() is the covariance of two vectors, and the calculated one-stage attenuation signal is W LQ_A1 (i) For example, formula 4:

[0156] W LQ_A1(i) = λ1 × W HQ(i) Formula 4

[0157] 1.5.2, Additive Noise Equivalent Attenuation.

[0158] After one stage of attenuation, it can be decomposed into: W LQ(i) =W LQ_A1(i) +W N(i) , where W LQ_A1(i) and W N(i) It can be modeled as a multidimensional Gaussian distribution, the noise signal W N(i) This will lead to the introduction of uncertainty in the original signal. The introduced uncertainty can be equivalently converted into a two-stage signal attenuation, as shown in Formula 5:

[0159]

[0160] In the above formula, s(i) 2 Σ LQ_A1 is the covariance of the Gaussian random variable WLQ_A1(i), is the noise variance, I is the identity matrix, s(i) 2 ,Σ LQ_A1 and The calculations are shown in Equation 6 to Equation 8:

[0161]

[0162] In the above formula, M is W LQ_A (i) The number of elements in the vector, W LQ_A (i) T It is W LQ_A The transposed vector of (i) is finally calculated to obtain the first attenuation coefficient W LQ_A , as shown in Formula 9:

[0163]

[0164] 1.6 Attenuation image fitting

[0165] In the above 1.5, the attenuation of wavelet domain signals caused by image degradation was analyzed and calculated. LQ_A (i) For W T(i) Perform corresponding attenuation to facilitate subsequent calculation of fidelity.

[0166] The second attenuated wavelet coefficient W T_A The calculation of can be shown in formula 10, for example, the degradation factor λ T The calculation of can be shown as formula 11:

[0167] WT_A(i) = λT * WT(i) Formula 10

[0168] λ T =∑ i Cov(W T (i),W LQ_A (i)) / ∑ i Cov(W T (i),W T (i)) Formula 11

[0169] 1.7 Root Mean Square Error (RMSE).

[0170] The present disclosure can use the root mean square difference to evaluate the difference between WLQ_A and WT_A as the fidelity indicator of the restored image:

[0171]

[0172] In the above formula, p represents the label of each image pixel, and P represents the total number of image pixels.

[0173] Example 2. Fidelity measurement of real degraded scenes.

[0174] 2.1 Attenuation image extraction model training.

[0175] In real-world image degradation scenarios, only degraded low-quality images can be obtained. LQ , it is impossible to get the original high-quality image I HQ The existing image quality index can only evaluate the quality of the restored image, but cannot evaluate the fidelity of the restored image. This scheme proposes to use an attenuation image extraction model to replace the attenuation image extraction process. The model can be implemented by a variety of different types of deep learning models. LQ Convert to I LQ_A , avoid the I HQ rely.

[0176] From the intuitive effect, the attenuation image I LQ_A is the degraded image I LQPerform noise removal and blur operations. The attenuation image extraction model is recommended to use a deep learning model similar to noise removal, such as UNet, SWIN, or its improved version. Deep learning models need to be trained to achieve good image conversion effects. This solution uses the system in 1.5 above to generate the corresponding I LQ and I LQ_A Training data, and use L1 loss function (Loss) or L2 Loss as the loss function to train the model.

[0177] Attenuation image extraction model training Figure 6 As shown, Figure 6 Random degradation, DWT, attenuated image extraction and IDWT modules in Figure 5 The modules in are the same.

[0178] 2.2 Realistic degradation scenarios: image inpainting fidelity measurement based on wavelet domain image degradation modeling.

[0179] After the attenuation image extraction model is trained, the model can be used to extract I LQ Convert to I LQ_A , and further calculate the fidelity index according to the figure below, and finally achieve I HQ Unknown fidelity assessment. For example Figure 7 As shown, Figure 7 DWT, attenuation image fitting, IDWT and RMSE modules in Figure 5 , Figure 6 The modules in are the same.

[0180] For example, this specification also provides an image fidelity measurement device, such as Figure 8 As shown, the device may include:

[0181] The coefficient determination module 801 is used to determine the first wavelet domain coefficient W of the original image. HQ , the second wavelet domain coefficient W of the degraded image LQ And the third wavelet domain coefficient W of the restored image T ; wherein the degraded image is an image obtained after the original image is degraded, and the repaired image is an image obtained after the degraded image is repaired;

[0182] The first attenuation module 802 is used to attenuate the LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A ;

[0183] The second attenuation module 803 is used to attenuate the wavelet coefficient W based on the first attenuation wavelet coefficient W. LQ_A For the third wavelet domain coefficient WT Attenuate and get the second attenuated wavelet coefficient W T_A ;

[0184] The measuring module 804 is configured to measure the first attenuation wavelet coefficient W based on the first attenuation wavelet coefficient W. LQ_A and the second attenuated wavelet coefficient W T_A , determine the fidelity of the repaired image.

[0185] This specification also provides an electronic device, which may include: one or more processors; wherein the processor is used to execute the steps of any of the methods described above.

[0186] Fig. 9 is a schematic diagram of an electronic device provided. Please refer to Fig. 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 909, a memory 908, and a non-volatile memory 910, and may also include hardware required for other functions. One or more examples of this specification can be implemented based on software. For example, a large model is deployed on the server, and the processor 902 reads the corresponding computer program from the non-volatile memory 910 to the memory 908 and then runs it. Of course, in addition to the software implementation, one or more examples of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0187] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of any of the above methods are implemented.

[0188] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0189] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification is to be regarded as exemplary only, and the true scope and spirit of the specification is indicated by the following claims.

[0190] The above description is only a preferred example of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for measuring image fidelity, characterized in that: The method is performed by an electronic device, and includes: Determine the first wavelet domain coefficient W of the original image HQ , the second wavelet domain coefficient W of the degraded image LQ And the third wavelet domain coefficient W of the restored image T ; wherein the degraded image is an image obtained after the original image is degraded, and the repaired image is an image obtained after the degraded image is repaired; Based on the second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A ; Based on the first attenuated wavelet coefficient W LQ_A For the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient W T_A ; Based on the first attenuated wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A , determine the fidelity of the repaired image.

2. The method according to claim 1, characterized in that The second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A ,include: For the second wavelet domain coefficient W LQ Decompose to get the first attenuated signal W LQ_A1 (i) and additive noise W N (i); wherein the first attenuation signal is the first wavelet domain coefficient W HQ an attenuated signal obtained by performing attenuation; For the additive noise W N (i) Convert the signal to obtain the second attenuated signal W LQ_A (i); Based on the first attenuation signal W LQ_A1 (i) and the second attenuated signal W LQ_A (i) Determine the first attenuation coefficient W LQ_A .

3. The method according to claim 1, characterized in that The first attenuated wavelet coefficient W LQ_A For the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient W T_A ,include: Based on the first attenuated wavelet coefficient W LQ_A and the third wavelet domain coefficient W T , determine the degradation factor λ T ; Based on the degradation factor λ T and the third wavelet domain coefficient W T , determine the second attenuated wavelet coefficient W T_A .

4. The method according to claim 1, characterized in that: The first attenuated wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A , determining the fidelity of the repaired image, comprising: The first attenuation wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A Performing inverse wavelet transform to obtain a first attenuation image and a second attenuation image; The fidelity of the restored image is determined based on a difference between the first attenuation image and the second attenuation image.

5. The method according to claim 4, characterized in that The method further comprises: Determining a first attenuated image corresponding to each of the plurality of degraded images; Using the multiple degraded images as input values ​​of an initial artificial intelligence AI model, and obtaining a third attenuated image corresponding to each degraded image output by the initial AI model; determining a loss function based on a difference between each third attenuation image and each first attenuation image; The initial AI model is trained based on the loss function, and when the training stop condition is met, a trained target AI model is obtained; wherein the target AI model is used to extract the attenuation image.

6. The method according to claim 5, characterized in that The method further comprises: Inputting the real degraded image into the target AI model to obtain a first estimated attenuation image output by the target AI model; Determine the fourth wavelet domain coefficient W of the first estimated attenuation image LQ_A_P , and the third wavelet domain coefficient W of the true restoration image T ; Based on the fourth attenuated wavelet coefficient W LQ_A_P The third wavelet domain coefficient W of the true restoration image T Attenuate and get the third attenuated wavelet coefficient W T_A '; For the third attenuated wavelet coefficient W T_A 'Perform inverse wavelet transform to obtain the third attenuation image; The fidelity of the actual restored image is determined based on the third attenuation image and the first estimated attenuation image.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The original image is randomly degraded to obtain the degraded image.

8. An image fidelity measurement device, characterized in that: The device comprises: The coefficient determination module is used to determine the first wavelet domain coefficient W of the original image. HQ , the second wavelet domain coefficient W of the degraded image LQ And the third wavelet domain coefficient W of the restored image T ; wherein the degraded image is an image obtained after the original image is degraded, and the repaired image is an image obtained after the degraded image is repaired; The first attenuation module is used to attenuate the second wavelet domain coefficient W LQ For the first wavelet domain coefficient W HQ Attenuate and get the first attenuated wavelet coefficient W LQ_A ; The second attenuation module is used to attenuate the wavelet coefficient W based on the first attenuation LQ_A For the third wavelet domain coefficient W T Attenuate and get the second attenuated wavelet coefficient W T_A ; A measuring module is used for measuring the first attenuation wavelet coefficient W LQ_A and the second attenuated wavelet coefficient W T_A , determine the fidelity of the repaired image.

9. An electronic device, characterized in that: The device comprises: One or more processors; wherein the processor is used to execute the steps of the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images