A method for evaluating the generalization ability of deep super-resolution networks

Abstract

The present invention discloses a method for evaluating the generalization ability of a deep super-resolution network. The method comprises: selecting a reference data set for a super-resolution model, the performance of the super-resolution model on the reference data set being known; inputting the reference data set and the target data set to be measured into the super-resolution model respectively to obtain corresponding reference depth statistical features and target depth statistical features; performing dimensionality reduction on the obtained reference depth statistical features and target depth statistical features; fitting the reduced dimensionality reference depth statistical features and target depth statistical features to a set distribution respectively to obtain distribution parameters reflecting the data distribution characteristics; calculating the distance between the corresponding distribution of the reference data set and the corresponding distribution of the target data set to evaluate the generalization of the super-resolution model. The present invention decouples image content and degradation through the deep statistical features within the network itself, and provides a non-parametric, highly interpretable and non-learning generalization evaluation index.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly, to a method for evaluating the generalization ability of a deep super-resolution network. Background Art

[0002] Super-resolution (SR) technology involves reconstructing a corresponding high-resolution (HR) image from an observed low-resolution (LR) image. It has important applications in image processing, computational photography, high-definition film and television, security monitoring, and other fields. The problem of super-resolution of images with unknown degradation is also known as blind super-resolution (blind SR) or real-world super-resolution (real-world SR). Existing work comparing the generalization capabilities and application scope of these blind super-resolution algorithms can only present some intuitive examples but does not provide any feasible evaluation strategies. Without a standard evaluation metric and dataset, different models cannot be fairly compared, which will greatly limit the progress of algorithm development in this field.

[0003] Generalization refers to a model's ability to make accurate predictions on new, unknown data after training. Generally, generalization error is used to measure generalization. This error is easily defined in high-level vision tasks, as it is calculated as the distance between the true label and the model's prediction. However, in low-level vision tasks such as image super-resolution and restoration, suitable strategies for evaluating generalization are lacking due to the lack of labeled data in most real-world scenarios and the complexity and difficulty of labeling.

[0004] Deep learning has achieved tremendous success in constrained environments, and researchers have gradually shifted their attention to its generalization capabilities. Generalization determines how well an algorithm can handle unseen data. However, due to its data-driven nature, deep learning can easily overfit the training data, leading to unpredictable generalization behavior. Traditional super-resolution models are typically trained with known degradation types and downsampling kernels, resulting in severe performance degradation in real images. Improving generalization is crucial for existing deep learning methods.

[0005] Deep learning-based blind super-resolution methods can be broadly categorized into four categories: degradation-targeted methods, kernel estimation methods, unsupervised methods, and internal statistical learning methods. Recent work using synthetic data has made significant progress. For example, BSRGAN and Real-ESRGAN demonstrate that, when sufficient degraded data can be synthesized, the model can be applied to a wide range of real-world scenarios. This phenomenon has also been observed in the blind face restoration task: GFP-GAN and GPEN use synthetic data to recreate old photos, and both demonstrate impressive results on a number of real-world images. While these algorithms can achieve strong performance on specific datasets, in some cases their results fall short, even falling short of traditional algorithms. This suggests that these algorithms have significant generalization limitations. Existing research fails to address the generalization capabilities and applicability of blind super-resolution algorithms, and research on the generalizability of the underlying visual models remains underdeveloped.

[0006] Currently, there are no specific evaluation metrics for generalization. In high-level vision tasks, such as classification, prediction accuracy on unseen datasets or categories is often used as a measure of generalization. However, in low-level vision tasks, such as image inpainting, no suitable evaluation strategy exists. Using image quality assessment (IQA) metrics, such as PSNR (peak signal-to-noise ratio) and NIQE (natural image quality evaluator), as a proxy for generalization has the following limitations. First, IQA is designed to assess image quality, but image quality does not equate to generalization. For example, traditional interpolation or filtering methods often produce lower IQA values ​​than deep learning models, yet they exhibit consistent performance (i.e., good generalization) across a wide range of data. Generalization should be a relative concept, dependent on the method itself, not just its output. Second, IQA is highly sensitive to image content and therefore can vary across different images. In image inpainting, a stable measure of unseen degradation is required, not dependent on a specific dataset. Third, IQA itself is not perfect. Full-reference based IQA metrics, such as PSNR and SSIM (structural similarity), cannot be used for real-world images without real labeled data, while non-reference IQA metrics, such as NIQE and PI, have poor stability and cannot accurately evaluate image quality.

[0007] In summary, performance and generalization are two important aspects for evaluating super-resolution networks. However, there is currently no research on the generalization ability of underlying visual network models. Summary of the Invention

[0008] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a method for evaluating the generalization ability of a deep super-resolution network, which comprises the following steps:

[0009] For the super-resolution model, a reference dataset is selected, on which the performance of the super-resolution model is known;

[0010] Inputting the reference data set and the target data set to be measured into the super-resolution model respectively to obtain corresponding reference depth statistical features and target depth statistical features;

[0011] Performing dimensionality reduction on the obtained reference depth statistical features and target depth statistical features;

[0012] The reference depth statistical features and target depth statistical features after dimensionality reduction are fitted to the set distribution respectively to obtain the distribution parameters reflecting the data distribution characteristics;

[0013] The distance between the corresponding distribution of the reference dataset and the corresponding distribution of the target dataset is calculated to evaluate the generalization of the super-resolution model.

[0014] Compared with the existing technology, the advantages of the present invention are that it proposes for the first time a method and index for evaluating the generalization ability of deep super-resolution networks, filling the gap in the generalization problem in the field of image super-resolution; the method provided for evaluating the generalization ability of deep super-resolution networks is based on the statistical characteristics of the model's internal features, rather than the output image; the evaluation method of the present invention is calculated on a test dataset but is insensitive to data selection; more importantly, the provided evaluation index is a non-parametric and non-learning index, which ensures that it has good generalization ability.

[0015] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0017] Figure 1 is a flowchart of a method for evaluating the generalization ability of a deep super-resolution network according to one embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of a process for evaluating the generalization ability of a deep super-resolution network according to an embodiment of the present invention;

[0019] Figure 3 3. It is a schematic diagram comparing the difference between the degradation type processing effect of the model according to one embodiment of the present invention on the test dataset and the processing effect on the reference dataset;

[0020] Figure 4FIG. 4 is a schematic diagram of obtaining depth statistical features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0022] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0023] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0024] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0025] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0026] This paper proposes a new generalization evaluation method for super-resolution networks, using the calculated metric SRGA to assess the degree of degradation of super-resolution models relative to different datasets. SRGA, based on statistical modeling of deep features in the super-resolution network, is a relative metric used to calculate the feature distribution distance between a reference dataset and a candidate test dataset. In this context, the reference dataset refers to a dataset with good model performance, typically co-distributed with the training dataset.

[0027] In short, the method for evaluating the generalization ability of super-resolution models provided by the present invention generally includes: first, obtaining the depth features corresponding to the model in the input data set. Then, compressing and projecting the features. Next, a predetermined distribution is used to perform probability distribution modeling on the projected feature set. Finally, the generalization error is measured by calculating the distance between the two probability distributions, thereby obtaining the generalization index SRGA. This index proposed by the present invention fills the gap in the generalization evaluation of the underlying visual model for the first time, and this index does not require the use of a large amount of data for learning, laying the foundation for subsequent research.

[0028] Specifically, combined Figure 1 and Figure 2 As shown, the provided method for evaluating the generalization ability of a deep super-resolution network comprises the following steps:

[0029] Step S110: Select a reference data set and a test data set.

[0030] Assume that the generalization ability of the super-resolution model G is tested. First, a reference dataset is selected Among them, S represents the data set, D ref Represents the degradation type of the dataset. The performance of model G on the reference dataset is known to be good, that is, model G can handle the degradation type D of the dataset well. ref Then, select other target data sets to be tested, or test data sets The SRGA index provided by the present invention can measure the degradation type D of the model on the test data set. test The difference between the treatment effect and the treatment effect on the reference dataset is shown in Figure 3 shown.

[0031] Step S120: Obtain depth statistical features of the super-resolution model on the reference dataset and the test dataset.

[0032] Input the reference dataset and test dataset into model G respectively to obtain the deep features of the model Where H, W, and C represent the height, width, and number of channels of the feature map, respectively. For example, the features of the last layer of the model can be selected as the depth features (i.e., the features of the layer before the last output layer of the model), such as Figure 4 In another embodiment, the deepest multi-layer features of the model can also be extracted to jointly fit the distribution.

[0033] Step S130: performing dimensionality reduction on the depth statistical features.

[0034] Specifically, the deep features of the reference dataset are obtained and deep features of the test dataset Use principal component analysis (PCA) for dimensionality reduction. PCA will find a projection matrix The dimension of the feature is reduced to D. The feature set obtained after PCA is recorded as and Principal component analysis can reduce the number of variables in a data set while retaining as much feature information as possible.

[0035] It should be noted that the dimension D of PCA dimensionality reduction can be appropriately set according to computational efficiency and subsequent evaluation accuracy, for example, D=300 or other values. In addition, other dimensionality reduction methods such as independent component analysis (ICA) can also be used.

[0036] Step S140: performing distribution fitting on the features after dimensionality reduction using a predetermined distribution.

[0037] The statistical characteristics of deep features will change due to the presence of degradation. Quantifying these changes can effectively measure the model's processing effect on different input degradations.

[0038] For example, the generalized Gaussian distribution is used to fit the probability distribution of the features after dimensionality reduction. Specifically, in order to model and analyze the deep features of the SR network, a suitable statistical distribution is selected to fit the empirical distribution p G,D Considering that the generalized Gaussian distribution (GGD) is universal in form, it covers many common distributions, such as Gaussian distribution, Laplace distribution and uniform distribution, and more importantly, a large amount of practice has proved that GGD can adapt to the statistical characteristics of natural images, and the deep features extracted by the SR network also inherit this property.

[0039] The probability function of the generalized Gaussian distribution with mean 0 is expressed as:

[0040]

[0041] in, Γ(.) is the gamma function, expressed as:

[0042] Γ(z)=∫0 ∞ t z-1 e -t dt(z>0) (2)

[0043] GGD has two parameters, one is the shape parameter α, and the other is the variance parameter σ. In one embodiment, the momentum matching method can be used to convert the depth features after dimensionality reduction into and Fit to the generalized Gaussian distribution and obtain the shape parameter and variance parameter of the fit.

[0044] It should be noted that, in view of the characteristics of the super-resolution network model dataset, it is preferred that the generalized Gaussian distribution can fit the distribution laws of various types of datasets, and an asymmetric generalized Gaussian distribution or other distributions may also be used.

[0045] Step S150 , measuring the distance between the corresponding distribution of the reference dataset and the corresponding distribution of the test dataset.

[0046] After estimating the GGD distribution parameters based on the deep features, the distribution changes can be captured by calculating the Kullback-Leibler divergence (KLD) between the distributions. and The KLD has a closed form solution:

[0047]

[0048] In order to evaluate the generalization ability of the model G, the feature distribution distance between the reference dataset and the test dataset is calculated as the generalization evaluation index, which is expressed as:

[0049]

[0050] in, and Denotes the GGD distribution parameter obtained by fitting the model G on the reference dataset and the test dataset, D kl is the KL distance. A smaller FDD means that the model G has similar processing effects on the two different types of input degradation. δ is introduced to avoid zero points in the logarithmic function.

[0051] It should be understood that when performing distance measurement on the fitted distribution, KL divergence or other indicators may be used to measure the distribution distance, for example, KL distance or Wasserstein distance.

[0052] The present invention can measure the degradation degree of the model G in one or more test data sets. If there are multiple test data sets with different degradation types, the average value of SRGA on different degradation data sets can be calculated, which is expressed as:

[0053]

[0054] In summary, for data-driven deep network models, their generalization ability is an important evaluation dimension, which is related to the stability and robustness of the model in practical applications. Measuring the generalization of the underlying visual network requires decoupling the image content and image degradation. Through a large amount of observation, analysis, experiments and tests, the inventors of the present invention discovered that the deep features of the underlying network can be fitted with a generalized Gaussian distribution, and this probability distribution can just characterize the model's processing effect on image degradation. Therefore, through the deep statistical features within the network itself, the image content and degradation are decoupled to a certain extent.

[0055] In summary, compared with the prior art, the present invention has the following technical effects:

[0056] Currently, there are no generalization metrics specifically designed for underlying visual depth models. Generalization is assessed manually or through inference using Image Quality Assessment (IQA). This paper proposes a generalization performance evaluation metric, SRGA, based on the internal statistical characteristics of deep super-resolution networks. It uses a generalized Gaussian distribution to model the probability distribution of deep features obtained by the model on a dataset, and derives a generalization metric by calculating the distance between these probability distributions.

[0057] 2) Currently, effective image quality evaluation index designs all require a large amount of manually labeled data for fitting and training, resulting in poor stability and prone to overfitting. The evaluation index of the present invention does not require complex parameter adjustment or learning from big data, avoiding the problem of overfitting and improving generalization and versatility.

[0058] 3) Current learning-based metrics have poor interpretability and lack clear physical meaning. This invention uses probabilistic modeling based on statistical features, which has clear physical meaning and strong interpretability. It fits the features of the deep network with a generalized Gaussian distribution and then calculates the distance between the distributions, which is more theoretically sound.

[0059] 4) Many current image quality evaluation metrics are based on learning, which is prone to overfitting and has limited scope of application. This invention does not rely on a training dataset and does not require training. Instead, it derives a physical model from in-depth analysis and statistical observation of the problem.

[0060] 5) The present invention is highly extensible. Although this indicator is currently designed for super-resolution tasks, it can still function normally when multiple degradation types, including noise and blur, are added to the super-resolution. This demonstrates that the indicator is highly extensible and can be extended to more image restoration tasks, rather than being limited to a specific restoration type. The indicator provided by this invention has a significant driving effect on the development of super-resolution algorithms, providing a direction for further algorithm optimization.

[0061] 6) Currently, many image-related metrics, such as PSNR and SSIM, require paired labeled images (ground-truth) as a reference, and then calculate the similarity between the test image and the labeled image. However, the metric provided by this invention does not require the collection of paired labeled data and can be evaluated on any real-world dataset.

[0062] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0063] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0065] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0066] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0068] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0069] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0070] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for evaluating the generalization ability of a deep super-resolution network, comprising the following steps: For the super-resolution model, a reference dataset is selected, on which the performance of the super-resolution model is known; Inputting the reference data set and the target data set to be measured into the super-resolution model respectively to obtain corresponding reference depth statistical features and target depth statistical features; Performing dimensionality reduction on the obtained reference depth statistical features and target depth statistical features; Fitting the reduced-dimensional reference depth statistical features and the target depth statistical features to a set distribution to obtain distribution parameters reflecting the data distribution characteristics; wherein the set distribution is a generalized Gaussian distribution, and the distribution parameters include a shape parameter and a variance parameter; The distance between the corresponding distribution of the reference dataset and the corresponding distribution of the target dataset is calculated to measure the generalization error, thereby obtaining a generalization index SRGA, and the generalization performance of the super-resolution model is evaluated using the generalization index SRGA.

2. The method according to claim 1, characterized in that The set distribution is a generalized Gaussian distribution with a mean of zero.

3. The method according to claim 2, characterized in that The following formula is used to evaluate the generalization of the super-resolution model: SRGA=log 10 (FDD+10 -δ )+δ in, and Represents the distribution parameters of the super-resolution model G fitted on the reference dataset and the target dataset, δ is a set non-zero constant, represents the reference dataset, represents the target dataset, D ref and D test Denote the degradation model of the corresponding data set, D kl It measures the distribution distance between the reference dataset and the target dataset, and SRGA is the evaluation indicator.

4. The method according to claim 3, characterized in that Distribution distance D kl Expressed as: Among them, α1 and α2 represent the shape parameters of the distribution corresponding to the reference dataset and the target dataset, respectively, σ1 and σ2 represent the variance parameters of the distribution corresponding to the reference dataset and the target dataset, respectively, and Γ(.) is the gamma function.

5. The method according to claim 1, wherein The reference depth statistical features and the target depth statistical features are features of the previous layer of the last output layer of the super-resolution model or are jointly represented by features of multiple layers, and are marked as Where H, W, and C represent the height, width, and number of channels of the feature map, respectively.

6. The method according to claim 3, characterized in that The distribution distance D kl is the KL distance.

7. The method according to claim 5, characterized in that The dimensionality reduction of the obtained reference depth statistical features and target depth statistical features includes: using principal component analysis to determine the projection matrix The projection matrix is ​​used to reduce the dimension of the reference depth statistical feature and the target depth statistical feature to D.

8. The method according to claim 3, characterized in that For multiple target datasets, the average value is used to measure the generalization of the super-resolution model, which is expressed as: Where N represents the number of target datasets and i represents the index of the target dataset.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images