Super-resolution method and device for mixed degenerate light field, terminal and storage medium

Abstract

This invention discloses a super-resolution method, apparatus, terminal, and storage medium for hybrid degraded light fields, comprising: decomposing downsampling degradation and noise degradation in a super-resolution task into a light field super-resolution task and a light field denoising task; training a first teacher network for super-resolution reconstruction to obtain first teacher features under the light field super-resolution task; training a second teacher network for light field denoising to obtain second teacher features under the light field denoising task; performing distillation learning on a student network based on the first and second teacher features, and training the student network based on fused features; performing hybrid degradation on the light field through the trained student network, aggregating the decoupled hybrid degradation features, and reconstructing a denoised high-resolution light field. This invention decouples the hybrid task into simple tasks, providing prior information through the simple tasks, enabling the student network to learn the reconstruction of hybrid degradation, thus achieving super-resolution of the light field degradation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a super-resolution method, apparatus, terminal, and storage medium for hybrid degraded light fields. Background Technology

[0002] Light field imaging is a novel image format that can capture rich scene geometry information, including spatial and angular details. It is widely used in many computer vision tasks, such as super-resolution, viewpoint synthesis, and quality assessment. Due to sensor resolution limitations, existing light field cameras struggle to obtain high-resolution light field images. Light field super-resolution, reconstructing high-resolution light fields from low-resolution ones, has significant application value. Early work proposed methods based on traditional models and optimizations, but these methods are susceptible to inaccurate depth estimation due to scene occlusion, resulting in noticeable artifacts in the reconstructed light field. In recent years, with the development of deep learning, light field super-resolution algorithms have demonstrated excellent performance. However, these methods typically only consider downsampling of the light field, making it difficult to handle multiple degradations in real-world scenes.

[0003] Image super-resolution is one of the important problems in computer vision. Deep learning methods have performed well in recent years for image super-resolution with multiple degradations. However, many methods have not studied the super-resolution of degraded light fields. Directly applying image super-resolution algorithms to light fields can lead to insufficient utilization of spatial and angular information; therefore, special design should be made for the scene geometry of the light field.

[0004] The purpose of knowledge distillation is to transfer learning capabilities from the teacher network to the student network, thereby reducing model parameters and optimizing model performance. Feature distillation is a type of knowledge distillation, where the student network optimizes model training by learning feature representations from the teacher network. Generalized knowledge distillation is not only used for model compression but also for privileged information learning, i.e., learning additional prior information from the teacher network and transferring it to the student network. Recent studies have included image super-resolution based on knowledge distillation; however, these methods only reconstruct downsampled degraded images and are not applicable to various degraded scenarios.

[0005] Therefore, existing technologies still need improvement. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a super-resolution method, device, terminal and storage medium for hybrid degraded light fields, in order to solve the technical problem that existing light field super-resolution algorithms lack robustness.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] In a first aspect, the present invention provides a super-resolution method for hybrid degraded light fields, comprising:

[0009] In the super-resolution task of mixed degraded light fields, downsampling degradation and noise degradation are decomposed into light field super-resolution task and light field denoising task.

[0010] The first teacher network is used to perform the light field super-resolution task, and the first teacher network is trained for super-resolution reconstruction to obtain the first teacher features under the light field super-resolution task.

[0011] The light field denoising task is performed by the second teacher network, and the second teacher network is trained for light field denoising to obtain the second teacher features under the light field denoising task.

[0012] The student network is distilled based on the first teacher features and the second teacher features to obtain fused features, and the student network is trained based on the fused features to obtain the trained student network.

[0013] The trained student network performs hybrid degradation on the light field, and the decoupled hybrid degradation features are aggregated to reconstruct a denoised high-resolution light field.

[0014] In one implementation, the step of performing the light field super-resolution task through a first teacher network and training the first teacher network for super-resolution reconstruction includes, prior to:

[0015] The low-resolution light field is bicubic upsampled to obtain the input light field image of the first teacher network;

[0016] The input light field image of the first teacher network is the same size as the input light field image of the second teacher network.

[0017] In one implementation, the step of performing the light field super-resolution task through a first teacher network and training the first teacher network for super-resolution reconstruction includes:

[0018] The encoder of the first teacher network generates the first latent feature from the input light field image of the first teacher network.

[0019] The first latent feature is reconstructed into a noise-free, high-resolution light field using the decoder of the first teacher network.

[0020] The first latent feature is used as the first teacher feature.

[0021] In one implementation, generating the first latent feature from the input light field image of the first teacher network via the encoder of the first teacher network includes:

[0022] The input light field image of the first teacher network is compressed into a set of feature maps through two convolutional layers, and the first hidden feature is output through two decoupled groups based on residual structures.

[0023] In one implementation, the step of performing the light field denoising task through a second teacher network, and training the second teacher network to obtain the second teacher features under the light field denoising task, includes:

[0024] The second latent feature is generated from the input light field image of the second teacher network through the encoder of the second teacher network;

[0025] The second latent feature is reconstructed into a noise-free, high-resolution light field through the decoder of the second teacher network;

[0026] The second latent feature is used as the second teacher feature.

[0027] In one implementation, generating the second latent feature from the input light field image of the second teacher network via the encoder of the second teacher network includes:

[0028] The input light field image of the second teacher network is compressed into a set of feature maps through two convolutional layers, and the second hidden feature is output through two decoupled groups based on residual structures.

[0029] In one implementation, the step of performing distillation learning on the student network based on the first teacher features and the second teacher features to obtain fused features includes, prior to:

[0030] The low-resolution light field is bicubic upsampled to obtain the input light field image of the student network;

[0031] The input light field image of the student network is the same size as the input light field image of the second teacher network.

[0032] In one implementation, the step of performing distillation learning on the student network based on the first teacher features and the second teacher features to obtain fused features, and then training the student network based on the fused features to obtain a trained student network, includes:

[0033] The encoder of the student network generates initial features from the input light field image of the student network.

[0034] The super-resolution learner and the denoising learner of the student network generate super-resolution features and denoising reconstructed features respectively through feature distillation learning.

[0035] The super-resolution features and the denoised reconstruction features are concatenated and input into the fusion network of the student network to fuse the single reconstruction features into hybrid reconstruction features;

[0036] The denoised high-resolution light field is reconstructed using the decoder of the student network.

[0037] In one implementation, the loss functions of both the first teacher network and the second teacher network are teacher reconstruction loss functions, which are the average absolute error between the predicted light field and the real label.

[0038] In one implementation, the loss function of the student network includes: super-resolution distillation loss, denoising distillation loss, and student reconstruction loss function.

[0039] In a second aspect, the present invention provides a super-resolution device for mixing degraded light fields, comprising:

[0040] The degradation decomposition module is used to decompose the downsampling degradation and noise degradation in the super-resolution task of mixed degraded light field into light field super-resolution task and light field denoising task;

[0041] The first teacher network module is used to perform the light field super-resolution task through the first teacher network, and to train the first teacher network for super-resolution reconstruction to obtain the first teacher features under the light field super-resolution task.

[0042] The second teacher network module is used to perform the light field denoising task through the second teacher network, train the second teacher network for light field denoising, and obtain the second teacher features under the light field denoising task.

[0043] The student network module is used to perform distillation learning on the student network based on the first teacher features and the second teacher features to obtain fused features, and to train the student network based on the fused features to obtain the trained student network.

[0044] The super-resolution module is used to perform hybrid degradation on the light field through the trained student network, and to aggregate the decoupled hybrid degradation features to reconstruct a denoised high-resolution light field.

[0045] Thirdly, the present invention provides a terminal comprising: a processor and a memory, the memory storing a super-resolution program for a hybrid degraded light field, the super-resolution program for a hybrid degraded light field being executed by the processor to implement the operation of the super-resolution method for a hybrid degraded light field as described in the first aspect.

[0046] Fourthly, the present invention also provides a storage medium, which is a computer-readable storage medium, storing a super-resolution program for a hybrid degraded light field, which, when executed by a processor, is used to implement the operation of the super-resolution method for a hybrid degraded light field as described in the first aspect.

[0047] The present invention, by employing the above technical solution, has the following effects:

[0048] This invention decouples hybrid tasks into simple tasks, using the simple tasks to provide prior information, enabling the decoupling and aggregation of hybrid features to be learned in a simple way. Two teacher networks learn the single distortion degradation reconstruction process, namely super-resolution and denoising, respectively, which can reduce the learning burden of the teacher networks and achieve better results. The student network learns the prior features of the single task generated by the two teacher networks through feature distillation, generating single restored features. The student network fuses these single restored features to reconstruct a clean, high-resolution light field, effectively achieving super-resolution of light field degradation and solving the technical problem of lack of robustness in existing light field super-resolution algorithms. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a super-resolution method for mixing degraded light fields in one implementation of the present invention.

[0051] Figure 2 This is a schematic diagram of the framework for super-resolution of hybrid degenerate light fields based on knowledge distillation in one implementation of the present invention.

[0052] Figure 3 This is a schematic diagram of the encoder and decoder in one implementation of the present invention.

[0053] Figure 4 This is a schematic diagram of the structure of the feature learner and the fusion network in one implementation of the present invention.

[0054] Figure 5 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0055] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] Exemplary methods

[0058] Existing light field super-resolution methods assume that low-resolution light fields are downsampled through interpolation to obtain degraded images. However, light fields in real-world scenes may contain other types of distortion, such as noise and blur, during generation, transmission, storage, and display. Traditional light field super-resolution algorithms are typically designed only for specific downsampling and struggle to effectively handle degraded light fields in real-world scenes. Therefore, existing light field super-resolution algorithms lack robustness and perform poorly in cases of mixed degradation. Although some methods have investigated mixed degradation in single-image super-resolution work, no such research has been conducted on light field images. Directly applying single-image super-resolution methods to light fields fails to fully utilize the rich geometric information of the scene.

[0059] For mixed-degraded light fields, direct super-resolution reconstruction may struggle to achieve ideal results due to the potential interference between different degradation components. Since both super-resolution and image restoration are low-level visual tasks, they share similarities in feature extraction of light field information. However, different degradation reconstruction tasks require the utilization of different characteristics. Therefore, simultaneous reconstruction of both tasks involves feature relationships and conflicts. In deep learning techniques, feature decoupling mechanisms can handle this situation by designing complex models, but these mechanisms aim to reduce learning difficulty and do not provide more prior information to aid learning.

[0060] Knowledge distillation, as a technique that can provide effective features and privileged information, has been used in image super-resolution. In many cases, the teacher network improves the performance of the student network by learning the same task as the student network. However, the teacher network can also assist the student network's learning by learning other related tasks. Most previous works have not considered mixed degradation cases and have only used a knowledge distillation framework with a teacher network and a student network to improve model performance. A key solution is to decompose complex degradation components and reduce the difficulty of reconstructing each degradation component to achieve better reconstruction results.

[0061] To address the aforementioned technical issues, this embodiment provides a super-resolution method for hybrid degraded light fields. In this embodiment, the hybrid task is decoupled into simple tasks, which provide prior information, allowing for easy learning of the decoupling and aggregation of hybrid features. Two teacher networks learn single-distortion degradation reconstruction processes, namely super-resolution and denoising, respectively, reducing the learning burden on the teacher networks and achieving better results. The student network learns the prior features of the single task generated by the two teacher networks through feature distillation, generating single restored features. The student network then fuses these single restored features to reconstruct a clean, high-resolution light field, effectively achieving super-resolution of the degraded light field and solving the technical problem of insufficient robustness in existing light field super-resolution algorithms.

[0062] like Figure 1 As shown, this embodiment of the invention provides a super-resolution method for hybrid degraded light fields, comprising the following steps:

[0063] Step S100: Decompose the downsampling degradation and noise degradation in the super-resolution task of mixed degraded light field into light field super-resolution task and light field denoising task.

[0064] In this embodiment, the super-resolution method for hybrid degraded light fields is applied to a terminal, which includes, but is not limited to, devices such as computers; the terminal is equipped with a super-resolution framework for hybrid degraded light fields based on knowledge distillation.

[0065] In this embodiment, the hybrid task is decoupled into simpler tasks, with the simpler tasks providing prior information. The decoupling and aggregation of hybrid features allows complex and difficult tasks to be learned in a simpler way. For example... Figure 2 As shown, this framework is applicable to reconstruction of any two types of degradation. For simplicity, this embodiment first assumes that the low-resolution light field is noisy, and then extends to include blurring.

[0066] Specifically, for denoising super-resolution, this embodiment proposes a unified framework that decomposes downsampling degradation and noise degradation into two single degradations. This framework consists of two teacher networks and one student network. The outputs of these three networks are all clean, high-resolution light fields. However, the input to the student network is a noisy, low-resolution light field, while the inputs to the teacher network are a clean, low-resolution light field and a noisy, high-resolution light field, respectively.

[0067] Two teacher networks learn single-distortion degradation reconstruction processes, namely super-resolution and denoising, respectively, which reduces the learning burden on the teacher networks and achieves better results. Each teacher network is based on an encoder-decoder structure. The encoder extracts latent features for a single task and provides them as prior information to the student network. The decoder reconstructs the latent features into a clean, high-resolution light field. On the other hand, the student network learns a hybrid degradation reconstruction. The student network learns the prior features for a single task generated by the two teacher networks through feature distillation, generating single restored features. Finally, the student network fuses these single restored features to reconstruct a clean, high-resolution light field.

[0068] This embodiment effectively achieves super-resolution of light field degradation within this framework. Experiments show (see below) that the framework proposed in this embodiment outperforms existing methods on multiple datasets and has significant application value for light field super-resolution.

[0069] like Figure 1 As shown, in one implementation of this invention, the super-resolution method for hybrid degraded light fields further includes the following steps:

[0070] Step S200: Perform the light field super-resolution task through the first teacher network, and train the first teacher network for super-resolution reconstruction to obtain the first teacher features under the light field super-resolution task.

[0071] The purpose of this embodiment is to provide a knowledge distillation-based super-resolution method for hybrid degraded light fields, which can solve the problem of super-resolution reconstruction of light fields under various degradation conditions.

[0072] Light fields are prone to various degradations in real-world scenes. If super-resolution is performed without eliminating distortion, the distortion effect will be amplified, leading to poor performance. For simplicity, this embodiment first discusses noisy low-resolution light fields, using bicubic downsampling and additive white Gaussian noise. The degradation process is as follows:

[0073] L noisy_lr =(L clean_hr )↓ s +n σ ,

[0074] Among them, L noisy_lr and L clean_hr These represent noisy low-resolution and clean high-resolution light fields, respectively. ↓ indicates bicubic downsampling, and n represents Gaussian noise. The light field image is bicubic downsampled with a scale factor of s, and Gaussian noise with a covariance (i.e., noise level) of σ. Changing the parameters n and s yields different degrees of light field degradation.

[0075] Given L noisy_lr Lclean_hr Reconstructed from the framework of this embodiment.

[0076] like Figure 2 As shown, the framework comprises two teacher networks and one student network. To reduce the learning burden and reconstruction difficulty, each teacher network learns only one task: light field super-resolution and light field denoising, respectively. Then, the student network learns the prior features provided by the teacher networks through feature distillation and performs hybrid degradation reconstruction. Finally, the student network aggregates the decoupled reconstruction features to reconstruct a clean, high-resolution light field.

[0077] Specifically, in one implementation of this embodiment, the following steps are included before step S200:

[0078] Step S201a: Perform bicubic upsampling on the low-resolution light field to obtain the input light field image of the first teacher network.

[0079] In this embodiment, the input light field image of the first teacher network is the same size as the input light field image of the second teacher network.

[0080] The two teacher networks proposed in this embodiment are a super-resolution teacher network (i.e., the first teacher network) and a denoising teacher network (i.e., the second teacher network). The input of the super-resolution teacher network is a clean low-resolution light field, while the input of the denoising teacher network is a noisy high-resolution light field. The output of both teacher networks is a clean high-resolution light field. The different input resolutions result in inconsistent feature map sizes, making feature distillation difficult. Therefore, for the super-resolution teacher network, before inputting the model, the low-resolution light field is first bicubic upsampled to obtain an input of the same size as the denoising teacher network. This process can be described as follows:

[0081]

[0082]

[0083] Where T SR and T Denoise The super-resolution teacher network and the denoising teacher network reconstruct the light field, respectively. and L clean_lr and L noisy_hr The degradation process is as follows, representing a clean low-resolution light field and a noisy high-resolution light field:

[0084] L clean_lr =(L clean_hr )↓ s ,

[0085] L noisy_hr =L clean_hr+n σ ,

[0086] In this embodiment, the two teacher networks have the same structure.

[0087] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0088] Step S201: Generate a first latent feature from the input light field image of the first teacher network through the encoder of the first teacher network;

[0089] Step S202: The first latent feature is reconstructed into a noise-free, high-resolution light field through the decoder of the first teacher network.

[0090] Step S203: Use the first hidden feature as the first teacher feature.

[0091] In this embodiment, to extract effective latent features, both teacher networks employ an encoder-decoder structure, such as... Figure 3 As shown. In the reconstruction process of the super-resolution teacher network, the encoder generates latent features from the degraded input light field image, which can represent important features for reconstruction and provide prior information for hybrid reconstruction. Then, the decoder reconstructs the latent features into a clean, high-resolution light field, as follows:

[0092]

[0093]

[0094] in, and It is a super-resolution encoder and decoder. Indicated by super-resolution encoder The generated latent features.

[0095] Specifically, in one implementation of this embodiment, step S201 includes the following steps:

[0096] Step S201a: The input light field image of the first teacher network is compressed into a set of feature maps through two convolutional layers, and the first hidden feature is output through two decoupled groups based on residual structure.

[0097] like Figure 3As shown in (a), in this embodiment, during the reconstruction of the super-resolution teacher network, the light field image is first compressed into a set of feature maps through two convolutional layers. Then, two decoupling groups based on residual structures are used, each containing a decoupling block. Each decoupling block contains a set of spatial convolutions, angular convolutions, and epipolar image convolutions. Finally, the output feature maps are used by the decoder to reconstruct latent features and for subsequent student network feature distillation. The decoder and encoder are constructed in a symmetrical structure, as shown in the specific structure below. Figure 3 As shown in (b).

[0098] like Figure 1 As shown, in one implementation of this invention, the super-resolution method for hybrid degraded light fields further includes the following steps:

[0099] Step S300: Perform the light field denoising task through the second teacher network, train the second teacher network for light field denoising, and obtain the second teacher features under the light field denoising task.

[0100] In this embodiment, the denoising teacher network has the same network structure as the super-resolution teacher network. Therefore, the denoising teacher network reconstruction process has the same processing steps as the super-resolution teacher network.

[0101] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0102] Step S301: Generate a second latent feature from the input light field image of the second teacher network through the encoder of the second teacher network;

[0103] Step S302: The second latent feature is reconstructed into a noise-free, high-resolution light field through the decoder of the second teacher network;

[0104] Step S303: Use the second latent feature as the second teacher feature.

[0105] In this embodiment, the process of the denoising teacher network is described as follows:

[0106]

[0107]

[0108] in and This represents the denoising decoder and the decoder.

[0109] Specifically, in one implementation of this embodiment, step S301 includes the following steps:

[0110] Step S301a: The input light field image of the second teacher network is compressed into a set of feature maps through two convolutional layers, and the second hidden feature is output through two decoupled groups based on residual structure.

[0111] In this embodiment, due to the excellent performance of DistgSSR in light field super-resolution, each teacher network built an encoder based on DistgSSR, such as... Figure 3 As shown in (a), during the reconstruction of the denoising teacher network, the light field image is first compressed into a set of feature maps through two convolutional layers. Then, two decoupling groups based on residual structures are used, each containing a decoupling block. Each decoupling block contains a set of spatial convolutions, angular convolutions, and epipolar image convolutions. Finally, the output feature maps are used by the decoder to reconstruct latent features and for subsequent student network feature distillation. The decoder and encoder are constructed in a symmetrical structure, as shown in Figure (a). Figure 3 As shown in (b).

[0112] like Figure 1 As shown, in one implementation of this invention, the super-resolution method for hybrid degraded light fields further includes the following steps:

[0113] Step S400: Distillation learning is performed on the student network based on the first teacher features and the second teacher features to obtain fused features, and the student network is trained based on the fused features to obtain the trained student network.

[0114] In this embodiment, students learn a hybrid reconstruction task online, where the input is a noisy, low-resolution light field and the output is a clean, high-resolution light field.

[0115] Specifically, in one implementation of this embodiment, the following steps are included before step S400:

[0116] Step S401a: Perform bicubic upsampling on the low-resolution light field to obtain the input light field image of the student network.

[0117] In this embodiment, the input light field image of the student network is the same size as the input light field image of the second teacher network.

[0118] In this embodiment, the input to the student network is also first bicubic upsampled, for the same reason as the super-resolution teacher network mentioned above. The process of the student network is described as follows:

[0119]

[0120] Where S represents the student network, and the reconstructed light field is represented as...

[0121] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0122] Step S401: Generate initial features from the input light field image of the student network through the encoder of the student network;

[0123] Step S402: Super-resolution features and denoised reconstruction features are generated respectively through the super-resolution learner and denoising learner of the student network using feature distillation learning.

[0124] Step S403: Connect the super-resolution features and the denoised reconstruction features and input them into the fusion network of the student network to fuse the single reconstruction features into hybrid reconstruction features;

[0125] Step S404: Reconstruct the denoised high-resolution light field using the decoder of the student network.

[0126] In this embodiment, the student network consists of four parts: an encoder, a feature learner, a fusion network, and a decoder, as follows: Figure 2 As shown in the diagram, for the student network, the encoder first generates initial features from the input. Then, the super-resolution learner and the denoising learner generate super-resolution and denoised reconstructed features respectively through feature distillation. The two sets of features are concatenated and input into the fusion network, which fuses the individual reconstructed features into hybrid reconstructed features. Finally, the decoder reconstructs a clean, high-resolution light field. This process is described as follows:

[0127] F_initial S =Encoder S (L noisy_lr ),

[0128]

[0129]

[0130] F_fuse S =Fusion S (Concatenate(F_SR S F_Denoise S )),

[0131]

[0132] Encoder S , Fusion S and Decoder S These represent the encoder, super-resolution learner, denoising learner, fusion network, and decoder, respectively. F_initialS It is the initial feature, F_SR S and F_Denoise S It refers to the generated super-resolution features and denoised features, F_fuse S These are the characteristics after fusion. It is a reconstructed, clean, high-resolution light field.

[0133] In this embodiment, the encoder and decoder of the student network have the same structure as the teacher network, such as... Figure 3 As shown, the super-resolution learner, denoising learner, and fusion network have similar structures, consisting of convolutional layers, residual blocks, and convolutional layers in sequence, as follows. Figure 4 As shown.

[0134] In this embodiment, during the training of the two teacher networks, the loss function for both networks is the reconstruction loss, which is the mean absolute error between the predicted light field and the true label.

[0135]

[0136]

[0137] Where L clean_hr It is the true label of the light field. and These are the predicted light fields generated by the super-resolution teacher network and the denoising teacher network, respectively.

[0138] This allows the teacher network to reconstruct a better light field, thus enabling the teacher's encoder network to provide more effective prior information.

[0139] The loss function of the student network first includes distillation loss. Variational information distillation provides a soft constraint, conveying more effective information by maximizing the mutual information between the feature maps of the teacher and student networks. This mutual information measures the distance between the student network's feature learner and the prior features provided by the teacher network. The mutual information is defined as follows:

[0140]

[0141] Where F T and F S These are the characteristics of teachers and students, respectively. H represents entropy, therefore H(F) T ) and H(F T |F S ) represents the entropy of the teacher network and the entropy of the teacher network given the student network. When maximizing mutual information I(F) T F S When ), it means minimizing the entropy H(F) of the teacher network given the student network. T |FS Student networks can better acquire knowledge from teacher networks, thus improving students' learning abilities. Because p(F) T |F S It is difficult to model, so a parametric model q(F) is used. T |F S Approaching the lower bound of mutual information, we obtain the lower bound of mutual information.

[0142] Because student network learning is independent of H(F) T ), maximize mutual information I(F) T F S This means maximizing the distribution of variables. Let q be a multivariate Laplace distribution, the super-resolution distillation loss can be expressed as:

[0143]

[0144] Where C, H, and W are the channels, height, and width of the feature map, respectively. μ SR It is a location parameter map, which uses the distance between the location parameter map and the feature map of the teacher network as the loss. SR It is a scale parameter graph that controls the distillation level by dynamically adjusting the degree to which the student network learns from the teacher network.

[0145] Specifically, this embodiment uses a small network with position and scale branches, which are based on the features F_SR generated by the super-resolution learner of the student network, respectively. S To estimate μ SR and b SR μ SR The estimation network consists of two 1×1 convolutional layers. SR The estimation network is then supplemented with a softplus function. Similarly, the denoising distillation loss can be described as:

[0146]

[0147] Where μ Denoise and b Denoise These are the position parameter map and scale parameter map generated by the denoising learner.

[0148] In addition, the training of student networks is also supervised by reconstruction loss, as described below:

[0149]

[0150] The total loss function of the student network is expressed as:

[0151]

[0152] The parameter λ is the parameter that balances the reconstruction loss and distillation loss; that is, the loss function of the student network includes: super-resolution distillation loss, denoising distillation loss, and student reconstruction loss function.

[0153] like Figure 1 As shown, in one implementation of this invention, the super-resolution method for hybrid degraded light fields further includes the following steps:

[0154] Step S500: The light field is hybridized and degraded using the trained student network. The decoupled hybridized features are aggregated and reconstructed to obtain a denoised high-resolution light field.

[0155] In this embodiment, the decoupling task can be performed through the trained student network, aggregating the decoupling of mixed degradation features to reconstruct a denoised, high-resolution light field. This embodiment is applicable to any degradation type including noise. For example, the invention can be extended to low-resolution blur kernels, and the degradation process can be represented as follows:

[0156]

[0157] Where k is the fuzzy kernel. It is a convolution operation, L blurred_lr This represents a blurry, low-resolution light field.

[0158] To verify the effectiveness of this invention, a dataset was used to train the proposed framework, and three categories—Occlusions, Reflective, and People—from the Stanford Lytro light field dataset were used for testing. The angular resolution of the light field images was 7×7, and during training, the spatial resolution was cut into 108×108 image patches. Random flipping and rotation were used as data augmentation strategies for the training data. The scale factor s was set to 2 and 4, the noise level σ was set to 5 and 10, and the blur kernel was an isotropic Gaussian blur kernel with kernel widths ranging from [0.2, 2] to [0.2, 4], corresponding to scale factors of 2 and 4, and a kernel size of 15×15. λ was set to 10. -6 Under these conditions, the model exhibits the best performance. The initial learning rate for both the teacher and student networks is set to 10. -4 And use cosine annealing to adjust the learning rate, where eta min =10 -5 Configure the Adam optimizer with β1 = 0.9 and β2 = 0.999.

[0159] First, in this embodiment, super-resolution and denoising of the light field are performed. The experimental results of this embodiment are first compared with a two-stage method consisting of existing light field super-resolution (resLF[5], LFSSR[6], LF-ATO[7], LF-Intemet[8] and DistgSSR[1]) and image denoising methods (VDN[9] and DANet

[10] ), and then compared with three super-resolution methods for degraded images (SRMD

[11] , IKC

[12] and DAN

[13] ). The experimental results are measured using peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), as shown in Table 1. The experimental results show that the method of this embodiment is superior to other methods, especially at a noise level of σ = 10. This embodiment is highly robust to noise in super-resolution, and therefore performs significantly better than other methods at high noise levels. This embodiment achieves excellent performance, demonstrating the effectiveness of super-resolution and denoising information learning under knowledge distillation.

[0160] Table 1 Comparison of performance in light field super-resolution and noise reduction

[0161]

[0162]

[0163] Then, this embodiment performs super-resolution and deblurring on the light field. This embodiment is compared with four super-resolution methods for degraded images (SRMD

[12] , SRMDNF

[12] , IKC

[13] , and DAN

[14] ). As shown in Table 2, this embodiment outperforms the other methods in super-resolution on blurred images, demonstrating the generalizability and scalability of this embodiment.

[0164] Table 2 Comparison of Light Field Super-Resolution and Deblurring Performance

[0165]

[0166]

[0167] In the technical solution of this embodiment, the degradation types are low resolution and noise, and low resolution and blur, but any two degradation types can be processed using the technical solution of this embodiment.

[0168] Furthermore, there are two types of degradation, but any number of degradation types can be processed using the technical solution of this embodiment. This is because, on the one hand, mixed degradation can be decomposed into two types of degradation, and the decomposed degradation can also be mixed degradation (only the number of degradation types is less than before decomposition); on the other hand, it can be decomposed into more than two types of degradation, and multiple teacher networks can be used to process multiple degradations together.

[0169] It is worth mentioning that, in addition to reconstructing light field images, this embodiment is also applicable to other forms of data, including two-dimensional images, videos, multi-view images, etc.

[0170] This embodiment achieves the following technical effects through the above technical solution:

[0171] This embodiment decouples the hybrid task into simple tasks, using the simple tasks to provide prior information, enabling the decoupling and aggregation of hybrid features to be learned in a simple way. The two teacher networks learn the single distortion degradation reconstruction process, namely super-resolution and denoising, respectively, which can reduce the learning burden of the teacher networks and achieve better results. The student network learns the prior features of the single task generated by the two teacher networks through feature distillation, generating single restored features. The student network fuses these single restored features to reconstruct a clean, high-resolution light field, effectively achieving super-resolution of light field degradation and solving the technical problem of lack of robustness in existing light field super-resolution algorithms.

[0172] Exemplary device

[0173] Based on the above embodiments, the present invention also provides a super-resolution device for hybrid degraded light fields, comprising:

[0174] The degradation decomposition module is used to decompose the downsampling degradation and noise degradation in the super-resolution task of mixed degraded light field into light field super-resolution task and light field denoising task;

[0175] The first teacher network module is used to perform the light field super-resolution task through the first teacher network, and to train the first teacher network for super-resolution reconstruction to obtain the first teacher features under the light field super-resolution task.

[0176] The second teacher network module is used to perform the light field denoising task through the second teacher network, train the second teacher network for light field denoising, and obtain the second teacher features under the light field denoising task.

[0177] The student network module is used to perform distillation learning on the student network based on the first teacher features and the second teacher features to obtain fused features, and to train the student network based on the fused features to obtain the trained student network.

[0178] The super-resolution module is used to perform hybrid degradation on the light field through the trained student network, and to aggregate the decoupled hybrid degradation features to reconstruct a denoised high-resolution light field.

[0179] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 5 As shown.

[0180] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a storage medium and internal memory; the storage medium stores the operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or mobile terminal.

[0181] When executed by a processor, this computer program is used to implement a super-resolution method for hybrid degraded light fields.

[0182] It will be understood by those skilled in the art that Figure 5 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a super-resolution program for a hybrid degraded light field, the super-resolution program for a hybrid degraded light field being executed by the processor to implement the operation of the super-resolution method for a hybrid degraded light field as described above.

[0184] In one embodiment, a storage medium is provided, wherein the storage medium stores a super-resolution program for a hybrid degraded light field, which, when executed by a processor, is used to implement the operation of the super-resolution method for a hybrid degraded light field as described above.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory.

[0186] In summary, this invention provides a super-resolution method, apparatus, terminal, and storage medium for hybrid degraded light fields. The method includes: decomposing downsampling degradation and noise degradation in a super-resolution task into a light field super-resolution task and a light field denoising task; training a first teacher network for super-resolution reconstruction to obtain first teacher features under the light field super-resolution task; training a second teacher network for light field denoising to obtain second teacher features under the light field denoising task; performing distillation learning on a student network based on the first and second teacher features, and training the student network based on fused features; performing hybrid degradation on the light field through the trained student network, aggregating the decoupled hybrid degradation features, and reconstructing a denoised high-resolution light field. This invention decouples the hybrid task into simple tasks, providing prior information through the simple tasks, enabling the student network to learn the reconstruction of hybrid degradation, thus achieving super-resolution of the light field degradation.

[0187] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A super-resolution method for hybrid degraded light fields, characterized in that, include: In the super-resolution task of mixed degraded light fields, downsampling degradation and noise degradation are decomposed into light field super-resolution task and light field denoising task. The light field super-resolution task is performed through a first teacher network. The first teacher network is trained for super-resolution reconstruction to obtain first teacher features under the light field super-resolution task, including: generating first latent features from the input light field image of the first teacher network using the encoder of the first teacher network. The first hidden feature is obtained through the decoder of the first teacher network. Reconstruct a noise-free, high-resolution light field; and use the first hidden feature... As the first teacher feature; the input of the first teacher network is a clean low-resolution light field; the output of the first teacher network is a clean high-resolution light field; The light field denoising task is performed through a second teacher network. The second teacher network is trained for light field denoising to obtain second teacher features under the light field denoising task, including: generating second latent features from the input light field image of the second teacher network using the encoder of the second teacher network. The second hidden feature is then processed by the decoder of the second teacher network. Reconstruct a noise-free, high-resolution light field; then use the second hidden feature. As the second teacher feature; the input of the second teacher network is a noisy high-resolution light field; the output of the second teacher network is a clean high-resolution light field; The low-resolution light field is bicubic upsampled to obtain the input light field image of the student network; wherein the input light field image of the student network is the same size as the input light field image of the second teacher network. The student network is subjected to distillation learning based on the first teacher features and the second teacher features to obtain fused features, and then trained based on the fused features to obtain a trained student network. This includes: generating initial features from the input light field image of the student network through the encoder of the student network; generating super-resolution features and denoising reconstruction features respectively through the super-resolution learner and denoising learner of the student network using feature distillation learning; concatenating the super-resolution features and the denoising reconstruction features and inputting them into the fusion network of the student network to fuse the single reconstruction features into a hybrid reconstruction feature; and reconstructing the denoised high-resolution light field through the decoder of the student network. The loss function of the student network includes: super-resolution distillation loss, denoising distillation loss, and student reconstruction loss function; The super-resolution distillation loss is: in, , and These represent the channels, height, and width of the feature map, respectively. For location parameter diagrams; This is a scale parameter diagram; The noise reduction distillation loss is: in, and The location parameter map and scale parameter map are generated by the denoising learner; The student reconstruction loss function is: in, For the true label of light field, The predicted labels for the student network; The total loss function of the student network is: Among them, parameters Parameters to balance reconstruction loss and distillation loss; The trained student network performs hybrid degradation on the light field, and the decoupled hybrid degradation features are aggregated to reconstruct a denoised, high-resolution light field. The hybrid degradation process is represented as follows: in, For fuzzy kernel, For convolution operations, It is a blurry, low-resolution light field.

2. The super-resolution method for hybrid degraded light fields according to claim 1, characterized in that, The step of performing the light field super-resolution task through the first teacher network and training the first teacher network for super-resolution reconstruction includes, prior to: The low-resolution light field is bicubic upsampled to obtain the input light field image of the first teacher network; The input light field image of the first teacher network is the same size as the input light field image of the second teacher network.

3. The super-resolution method for hybrid degraded light fields according to claim 1, characterized in that, The encoder of the first teacher network generates the first latent feature from the input light field image of the first teacher network. ,include: The input light field image of the first teacher network is compressed into a set of feature maps through two convolutional layers, and the first hidden features are output through two decoupled groups based on residual structures. .

4. The super-resolution method for hybrid degraded light fields according to claim 1, characterized in that, The encoder of the second teacher network generates the second latent feature from the input light field image of the second teacher network. ,include: The input light field image of the second teacher network is compressed into a set of feature maps through two convolutional layers, and the second hidden features are output through two decoupled groups based on residual structures. .

5. The super-resolution method for hybrid degraded light fields according to claim 1, characterized in that, The loss function of both the first teacher network and the second teacher network is the teacher reconstruction loss function, which is the average absolute error between the predicted light field and the real label.

6. A super-resolution apparatus for hybrid degraded light fields, used to implement the super-resolution method for hybrid degraded light fields as described in any one of claims 1-5, characterized in that, include: The degradation decomposition module is used to decompose the downsampling degradation and noise degradation in the super-resolution task of mixed degraded light field into light field super-resolution task and light field denoising task; The first teacher network module is used to perform the light field super-resolution task through the first teacher network, and to train the first teacher network for super-resolution reconstruction to obtain the first teacher features under the light field super-resolution task. The second teacher network module is used to perform the light field denoising task through the second teacher network, train the second teacher network for light field denoising, and obtain the second teacher features under the light field denoising task. The student network module is used to perform distillation learning on the student network based on the first teacher features and the second teacher features to obtain fused features, and to train the student network based on the fused features to obtain the trained student network. The super-resolution module is used to perform hybrid degradation on the light field through the trained student network, and to aggregate the decoupled hybrid degradation features to reconstruct a denoised high-resolution light field.

7. A terminal, characterized in that, include: The processor and memory, the memory storing a super-resolution program for a hybrid degraded light field, the super-resolution program for a hybrid degraded light field being executed by the processor to implement the operation of the super-resolution method for a hybrid degraded light field as described in any one of claims 1-5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium that stores a super-resolution program for a hybrid degraded light field. When executed by a processor, the super-resolution program for a hybrid degraded light field is used to implement the operation of the super-resolution method for a hybrid degraded light field as described in any one of claims 1-5.

Images