Lightweight image super-resolution method based on viT-cnn heterogeneous distillation

Abstract

The application belongs to the technical field of computer vision, and discloses a light-weight image super-resolution method based on ViT-CNN heterogeneous distillation, wherein a low-resolution image is input into a pre-trained teacher model and a student model to be trained, attention features of the teacher model and the student model are constructed and fused to realize spatial knowledge transfer and alignment across architectures, global features of the teacher model and the student model are aligned in a frequency domain to realize robust spectral knowledge transfer, and the student model is trained and optimized through loss optimization. The application improves the image super-resolution capability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a lightweight image super-resolution method based on ViT-CNN heterogeneous distillation. Background Technology

[0002] With the development of deep learning, image super-resolution (SR) has made significant progress in the field of computer vision. Convolutional Neural Networks (CNNs) have long been the mainstream method for super-resolution tasks. They can efficiently reconstruct image details through local receptive fields and convolutional operations, and have good optimization and hardware acceleration support in mobile and edge devices. However, CNNs are limited by their local inductive bias, which has certain shortcomings in global dependency modeling and complex structure recovery. They are also unable to fully capture the long-range spatial relationships of images, thus limiting their ability to reconstruct complex textures.

[0003] In recent years, Vision Transformer (ViT), as an emerging model architecture, has demonstrated outstanding performance in visual tasks such as image classification, detection, and segmentation thanks to its self-attention mechanism, which effectively captures global contextual information. When applied to image super-resolution tasks, ViT's global modeling capabilities can capture the low-frequency structure and high-frequency texture of images, providing rich global information for subsequent feature learning in convolutional networks, thereby improving the final reconstruction quality. However, ViT models typically require substantial computational resources and GPU memory, making direct deployment on mobile or embedded devices difficult. This makes achieving high-performance super-resolution on resource-constrained platforms a pressing issue.

[0004] Knowledge distillation (KD) offers a feasible approach to addressing the aforementioned problems. This method achieves a balance between accuracy and efficiency by transferring knowledge from a large model (teacher model) to a small model (student model). However, when the teacher model is a ViT and the student model is a CNN, traditional distillation methods struggle to effectively transfer knowledge. This is because ViT utilizes a self-attention mechanism to model global dependencies, while CNNs rely on local convolutional operations; the two differ fundamentally in their feature representation and structure. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a lightweight image super-resolution method based on ViT-CNN heterogeneous distillation. This method extracts spatial attention features and spectral features from the deep features of the Vision Transformer (ViT), guiding the student network to achieve stronger image reconstruction capabilities while maintaining computational efficiency. The technical problem this invention aims to solve is: in image super-resolution tasks, how to fully utilize the global spatial relationships and spectral information contained in the deep features of the teacher model through feature layer distillation to achieve cross-architecture knowledge transfer, enabling the lightweight CNN student network to acquire more complete structural awareness and global spectral capabilities, and ensuring efficient deployment on mobile and edge devices.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A lightweight image super-resolution method based on ViT-CNN heterogeneous distillation includes the following steps:

[0008] The data input steps are as follows: input low-resolution images into the pre-trained teacher model and the student model to be trained, where the teacher model is a ViT model based on the Transformer architecture and the student model is a lightweight model based on CNN.

[0009] The steps for spatial relationship alignment are as follows: extract deep features from the teacher model and generate teacher attention features, extract deep features from the student model and generate student attention features; fuse the two attention features to generate common attention features; calculate the spatial constraint distillation loss based on the teacher attention features, student attention features and common attention features to achieve cross-architecture spatial knowledge transfer and alignment.

[0010] The steps of global spectrum fusion are as follows: extract the global features of the deep features of the teacher model and transform them to the frequency domain to obtain the teacher's spectrum features; extract the global features of the deep features of the student model and transform them to the frequency domain to obtain the student's spectrum features; map the student's spectrum features to the same dimension as the teacher's spectrum features, and calculate the spectral distillation loss between the low-frequency components of the two to achieve frequency domain knowledge transfer.

[0011] The steps for loss calculation are as follows: calculate the image reconstruction loss based on the super-resolution reconstruction output of the student model and the real high-resolution image; use the weighted sum of spatial constraint distillation loss, spectral distillation loss and image reconstruction loss as the total loss, and train and optimize the student model.

[0012] Further steps in generating teacher attention characteristics include:

[0013] Flatten the deep feature map of the teacher model into a token sequence format;

[0014] Global average pooling is performed on the token sequence to obtain the global features of the teacher model;

[0015] The teacher model calculates the spatial relationship weights between the global features of the teacher model and the token sequence through a self-attention mechanism, and then normalizes them using Softmax to obtain the teacher attention features.

[0016] Further steps in generating student attention characteristics include:

[0017] Global average pooling is performed on the deep feature maps of the student model to obtain the global features of the student.

[0018] By using learnable projection transformations and nonlinear mappings, the student's global features are mapped to a high-dimensional attention space that is compatible with the teacher's attention features, thus obtaining the student's attention features.

[0019] Further, the steps for synthesizing public attention features are as follows:

[0020] Teacher attention characteristics and student attention characteristics are spliced ​​together along the channel dimension;

[0021] The concatenated features are then passed through a projection network containing linear layers, layer normalization, ReLU activation function, and Sigmoid activation function to synthesize common attention features.

[0022] Furthermore, the formula for calculating the space-constrained distillation loss is as follows:

[0023] ;

[0024] in, This indicates space-constrained distillation loss, and B represents the batch size. Indicates the sample number. This represents the attentional characteristics of the b-th student. Show the first A characteristic of public attention, Indicates the first Teacher attention characteristics represents the square of the L2 norm, and ⊙ represents element-wise multiplication.

[0025] Furthermore, the formula for calculating the spectral distillation loss is as follows:

[0026] ;

[0027] in, This indicates the spectral distillation loss, and B represents the batch size. Indicates the sample sequence number; This represents the spectral characteristics of the b-th aligned student. The b-th element represents the spectral characteristics of the teacher. This indicates the extraction of low-frequency components.

[0028] Furthermore, the formula for calculating the image reconstruction loss is as follows:

[0029] ;

[0030] in B represents the L1 image reconstruction loss for the task; B represents the batch size. Indicates the sample sequence number; This represents the b-th true high-resolution image; This represents the reconstructed image of the b-th student model;

[0031] The final overall optimization objective function is:

[0032] ;

[0033] in, This indicates space-constrained distillation loss; Indicates the loss during spectral distillation; The L1 image reconstruction loss represents the task. These are the weighting coefficients.

[0034] Furthermore, the method also includes a super-resolution reconstruction step: after the model training is completed, the low-resolution image to be processed is input, and the reconstructed high-resolution image is output.

[0035] Compared with the prior art, the advantages of the present invention are as follows:

[0036] (1) This invention proposes a lightweight cross-architecture image super-resolution method based on global feature guidance. By performing global average pooling on the deep features of the teacher model, spatial attention features and spectral features of the image are extracted and distilled into the student CNN as guiding information. This effectively transmits global contextual information and frequency domain features while maintaining the lightweight nature of the student model. Spatial attention distillation effectively bridges the differences between ViT and CNN in deep feature representation. The student model can learn the global dependencies contained in the teacher model under the local convolutional structure, providing richer structural and texture information for subsequent image reconstruction. This enables the student model to perform lightweight super-resolution reconstruction more efficiently on resource-constrained platforms and has the potential to outperform traditional methods in terms of structural integrity and texture detail.

[0037] (2) This invention guides the student network to learn the frequency pattern consistent with the teacher model in the feature representation by explicitly distilling the frequency information in the deep features of the teacher model in the feature layer, thereby providing richer global and local information for subsequent image reconstruction and helping to improve the ability to restore details.

[0038] (3) The present invention introduces global spatial relationship alignment and spectral distillation in the training phase, but no additional calculation is required in the inference phase, which ensures the lightweight characteristics of the student network and is suitable for efficient deployment on mobile and embedded devices.

[0039] (4) The global feature-guided distillation framework proposed in this invention takes into account both the global modeling capability and the local detail perception capability of the feature layer, providing more complete feature information for image reconstruction in various complex scenarios, which helps to improve the generalization and practicality of the model.

[0040] (5) The present invention can effectively reduce the computational load and storage requirements of the model, and is suitable for high-performance image super-resolution reconstruction tasks of edge computing and mobile devices. It can also be widely applied to various practical scenarios such as video surveillance, intelligent security, medical image processing, remote sensing image enhancement and industrial inspection.

[0041] In summary, the cross-architecture lightweight image super-resolution method based on global feature guidance proposed in this invention, while ensuring model lightweighting and deployment efficiency, fully leverages the complementary advantages of ViT and CNN through feature layer distillation, providing a more complete feature representation for super-resolution tasks, which helps improve reconstruction accuracy and robustness, and has high theoretical value and application prospects. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the system architecture of the present invention;

[0044] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0045] Figure 3 To visualize the comparison results. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1

[0048] This invention aims to solve several technical problems existing in current image super-resolution methods regarding knowledge transfer and lightweight deployment, specifically including the following aspects:

[0049] (1) Most existing knowledge distillation-based super-resolution methods are designed for homogeneous networks (such as CNN→CNN, ViT→ViT), and it is difficult to effectively align the feature representations of the two models when distilling across architectures (such as ViT→CNN). To address this issue, this invention proposes a lightweight cross-architecture image super-resolution method guided by global features. By extracting global features from the deep features of the teacher model and distilling them at the feature layer, the student CNN can learn the global dependency patterns and spatial relationships of the teacher ViT, thereby providing richer global information for subsequent reconstruction while maintaining the local inductive bias of the CNN.

[0050] (2) Existing methods generally neglect the frequency domain information contained in the deep features of the teacher model during the distillation process, relying only on spatial feature alignment, which leads to the student network's inadequacy in learning global frequency features. This invention introduces spectral distillation loss to explicitly transfer the low-frequency and high-frequency patterns of the global features of the teacher model to the student network at the feature layer, thereby improving the student model's ability to perceive global spectral information and providing support for subsequent image reconstruction.

[0051] (3) Existing lightweight super-resolution models generally face problems such as limited computing resources and constrained storage space when deployed in mobile and embedded devices. Existing cross-architecture distillation methods often require additional complex projection modules or large-scale computational overhead, which is difficult to meet the needs of resource-constrained scenarios. This invention introduces a global spatial-spectral distillation mechanism in the feature layer, enabling the student network to learn the global knowledge of the teacher model during the training phase, while maintaining the original lightweight CNN structure unchanged during the inference phase, thereby achieving efficient and lightweight deployment.

[0052] Specifically, this invention provides a lightweight image super-resolution method based on ViT-CNN heterogeneous distillation. The core idea of ​​this method is to utilize the global context information contained in the deep features of the teacher model and extract the global features of the image through global average pooling, and pass them as guiding information for cross-architecture distillation to the student convolutional neural network (CNN).

[0053] This invention includes the following steps:

[0054] The data input steps are as follows: input the low-resolution image into the pre-trained teacher model and the student model to be trained. In this embodiment, the teacher model is the SwinIR vision model based on the Transformer architecture (other vision models based on the Transformer architecture can also be implemented), and the student model is a lightweight model based on the convolutional neural network.

[0055] The steps for spatial relationship alignment are as follows: extract deep features from the teacher model and generate teacher attention features, extract deep features from the student model and generate student attention features; fuse the two attention features to generate common attention features; calculate the spatial constraint distillation loss based on the teacher attention features, student attention features and common attention features to achieve cross-architecture spatial knowledge transfer and alignment.

[0056] The steps of global spectrum fusion are as follows: extract the global features of the deep features of the teacher model and transform them to the frequency domain to obtain the teacher spectrum features; extract the global features of the deep features of the student model and transform them to the frequency domain to obtain the student spectrum features; map the student spectrum features to the same dimension as the teacher spectrum features, and calculate the spectral distillation loss between the low-frequency components of the two to achieve frequency domain knowledge transfer.

[0057] The steps for loss calculation are as follows: calculate the image reconstruction loss based on the super-resolution reconstruction output of the student model and the real high-resolution image; use the weighted sum of spatial constraint distillation loss, spectral distillation loss and image reconstruction loss as the total loss, and train and optimize the student model.

[0058] As one implementation method, combined with Figure 1 and Figure 2 Specifically, it includes the following steps:

[0059] Step 1: Data Acquisition and Preprocessing

[0060] In this embodiment, the dataset used for the image super-resolution task can be DIV2K, Set5, Set14, or other publicly available image super-resolution benchmark datasets. Alternatively, representative and diverse image data can be collected independently based on the actual application scenario. The images in the dataset are first used as a high-resolution image sample set, denoted as... ,in represents a high-resolution image, b represents the sample number, and B represents the batch size.

[0061] In order to obtain low-resolution images that correspond to the actual degradation process, this embodiment uses a high-resolution image sample set. A downsampling operation is performed. The downsampling method uses bicubic interpolation to simulate image degradation in real-world scenes. Through these operations, a corresponding low-resolution image sample set is generated. ,in, Represents high-resolution images The matching low-resolution image, where b represents the sample number and B represents the batch size.

[0062] In actual preprocessing, to enhance the robustness and generalization ability of the model, data augmentation operations can be further performed on the training images based on the above sampling and downsampling. Data augmentation methods include, but are not limited to, random cropping, horizontal or vertical flipping, rotation, brightness and contrast adjustment, etc., to ensure that the model can adapt to image degradation in various scenarios. In addition, to facilitate model input and training, all images need to be normalized to a uniform numerical range (e.g., [0,1] or [-1,1]) and converted into tensor form to meet the input requirements of deep learning frameworks.

[0063] Through the above data acquisition and preprocessing steps, pairs of high-resolution and low-resolution image samples can be obtained simultaneously, providing basic data support for the subsequent training of lightweight image super-resolution models across architectures.

[0064] Step 2: Spatial Relationship Alignment

[0065] This step is achieved through the spatial relationship alignment module. The specific steps for spatial relationship alignment include:

[0066] Extract deep features from the teacher model and generate teacher attention features (i.e., spatial attention features) based on these deep features.

[0067] Extract deep features from the student model and generate student attention features based on these deep features;

[0068] Based on teacher attention characteristics and student attention characteristics, a common attention characteristic is synthesized;

[0069] Based on the characteristics of teacher attention, student attention, and common attention, spatially constrained distillation loss is calculated to guide students to learn local convolutional features and align with the global dependencies implied by the teacher.

[0070] After completing the data acquisition and preprocessing in step 1, this embodiment will use the low-resolution image sample set. The inputs are fed into the pre-trained teacher model and the student model to be trained, respectively. The teacher model uses SwinIR, based on the Transformer architecture, which has strong global dependency modeling capabilities; the student model uses the classic EDSR based on convolutional neural networks, which has relatively few parameters, low computational complexity, and is more suitable for lightweight deployment in resource-constrained environments. This invention's proposed lightweight cross-architecture image super-resolution method, guided by global features, fully leverages the advantages of Transformer in global modeling and transfers its knowledge to lightweight convolutional networks, thus achieving a balance between performance and efficiency.

[0071] 2.1. Extraction of Teacher Attention Features

[0072] For the intermediate layer feature extraction representation of the teacher model, this embodiment selects to extract features in the last layer of the feature extraction stage of SwinIR.

[0073] Specifically, in the final layer of the SwinIR network's feature extraction, the network performs thorough contextual modeling and captures long-range dependencies of the input image; therefore, the features output by this layer contain rich global structural information. The super-resolution model architecture consists of a feature extraction module, an upsampling module, and an image reconstruction module.

[0074] For the teacher model, the features extracted by the feature extraction module are represented as follows: ;

[0075] in, B represents teacher characteristics, and B represents batch size. The number of channels representing teacher characteristics. , The height and width of the teacher feature represent the spatial dimensions of the feature map. To facilitate cross-architecture knowledge distillation with the student network and ensure effective alignment of information at corresponding spatial locations, the feature map is flattened into token form:

[0076] ;

[0077] This indicates the teacher's characteristics after being flattened. The number of tokens representing teacher characteristics; The feature dimension of the token represents the teacher's characteristics; to obtain global information, average pooling is performed on all spatial location vectors to obtain global features:

[0078] ;

[0079] in, Indicates the first A spatial location vector; This means averaging the features across all spatial locations to obtain a feature vector that aggregates global information; These are global features of the teacher model, used to guide the student network to focus on the structural and textural information of the entire image.

[0080] The teacher model computes global features using a self-attention mechanism. Characteristics of teachers after flattening Spatial relation weights:

[0081] ;

[0082] in, Indicating the characteristics of teacher attention, Represents the global features of the teacher model. This indicates the teacher's characteristics after being flattened. Indicates to The last two dimensions of the feature are transposed. The output attention map is then generated. The importance weights of each image region in the global reconstruction task were quantified.

[0083] 2.2 Extraction of Student Attention Features

[0084] For the student model EDSR, the convolutional features of the last layer of its feature extraction module are extracted and represented as follows:

[0085] ;

[0086] in, B represents student characteristics, and B represents batch size. The number of channels representing student characteristics. and This represents the height and width of the student features. Due to the limitations of the local receptive field in convolutional neural networks, their ability to capture global context is weak, while the teacher model (SwinIR) possesses a self-attention mechanism, enabling it to model global dependencies in the image. Therefore, to achieve cross-architecture knowledge distillation, it is necessary to map the convolutional features of the student network to a space compatible with the teacher's attention features, thereby achieving effective alignment.

[0087] This embodiment first performs global average pooling to compress the spatial features into a global representation:

[0088] ;

[0089] in, Indicates global average pooling; This represents the student characteristics after average pooling. This process represents student features. It compresses two-dimensional spatial features into a global vector representation, allowing the local convolutional features of the student model to converge on global information from the entire image, facilitating matching with the global attention of the Transformer teacher.

[0090] Subsequently, in order to further map student features to a high-dimensional attention space compatible with teacher attention features, this embodiment introduces learnable projection transformations and nonlinear mappings:

[0091] ;

[0092] in, Student attention characteristics Let ReLU be the learnable projection matrix, and let ReLU denote the nonlinear activation function. This represents the Sigmoid activation function. The final result is... As an attention map generated from CNN features, it is used to indicate the importance of various spatial locations in an image.

[0093] 2.3. Synthesis of Public Attention Features

[0094] Because the teacher model (Transformer) and the student model (CNN) have inherent differences in architecture and feature representation, directly aligning their attention distributions may lead to unstable knowledge transfer or even negative transfer, thus affecting the distillation effect and the final reconstruction quality of the student model. To address this issue, this invention maps the teacher and student attention into a unified joint space for comprehensive learning, making cross-architecture distillation more stable and efficient.

[0095] The specific steps are as follows: [Analyze teacher attention characteristics] Characteristics of student attention The joint representation is obtained by concatenating along the channel dimension:

[0096] ;

[0097] in, Indicating characteristics of public attention, This represents the combination of teacher's attention and student's attention. , Let be the projection matrix, and ReLU denote the nonlinear activation function. This represents the Sigmoid activation function. The result is... This represents the consensus attention distribution after comprehensive learning.

[0098] Through this public attention mapping, the attention information of teachers and students can be integrated in a unified space. The student model can not only absorb the global context information provided by the teacher, but also take into account the local structure perception ability of its own convolutional features, thereby providing stable and controllable guidance for cross-architecture knowledge transfer.

[0099] 2.4. Optimize the objective function

[0100] To achieve efficient distillation, this embodiment constructs a spatially constrained distillation loss function within the joint attention region:

[0101] ;

[0102] Where B represents the batch size. Indicates the sample number. This represents the attentional characteristics of the b-th student. This shows the b-th characteristic of public attention. This represents the attentional characteristic of the b-th teacher. represents the square of the L2 norm, and ⊙ represents element-wise multiplication, used to strengthen the alignment of teacher and student attention within the consensus region.

[0103] By combining the aforementioned common attention feature synthesis and spatial constraint distillation loss, the student model, under a lightweight convolutional architecture, can not only learn the global contextual information and long-range dependencies provided by the teacher model, but also maintain the ability of local convolutional features to perceive structural and texture details; and obtain a more stable and controllable attention pattern guidance throughout the training process.

[0104] Step 3: Global Spectrum Fusion

[0105] This step is implemented through the global spectrum fusion module.

[0106] To further leverage the advantages of the teacher model SwinIR in global frequency modeling while avoiding interference from high-frequency noise in image super-resolution reconstruction, this embodiment proposes a Global Spectral Fusion (GSF) module. This module transfers the low-frequency stable features inherent in the teacher's global features to the student model EDSR, enabling it to achieve stronger global perception and detail reconstruction capabilities while maintaining a lightweight convolutional structure. This, in turn, improves the generalization performance and visual reconstruction quality of the super-resolution task.

[0107] 3.1. Extraction of Teacher Spectral Features

[0108] Teacher global characteristics Where B represents the batch size and D represents the feature dimension. To facilitate alignment with student features and subsequent operations, the intermediate dimensions are first compressed, i.e., the dimension with length 1 is removed, resulting in a two-dimensional matrix representation:

[0109] ;

[0110] Where B represents the batch size, D represents the feature dimension, and `squeezed` indicates that redundant dimensions of length 1 are removed to obtain a more compact tensor. To extract its spectral features, this embodiment performs a Fast Fourier Transform (FFT) on the vector to obtain its decomposition in the frequency domain:

[0111] ;

[0112] in, Indicates the spectral characteristics of teachers; Fast Fourier Transform.

[0113] 3.2. Student Spectral Alignment

[0114] This embodiment obtains the student's spectral representation through Fast Fourier Transform:

[0115] ;

[0116] in, Fast Fourier Transform; This represents the student characteristics after average pooling. It is a student spectral characteristic.

[0117] And use a learnable mapping matrix Align it to a dimension consistent with the teacher's spectral space:

[0118] ;

[0119] in, It is a student spectral characteristic; It is a learnable mapping matrix. It is the aligned student spectral characteristics.

[0120] 3.3. Optimize the objective function

[0121] To ensure that the student model fully learns the teacher's low-frequency knowledge, this embodiment designs a spectral distillation loss:

[0122] ;

[0123] Where B represents the batch size. Indicates the sample sequence number; This represents the spectral characteristics of the b-th aligned student. The b-th element represents the spectral characteristics of the teacher. This indicates the extraction of low-frequency components.

[0124] This loss function enhances the student model's ability to perceive the overall image structure and stable texture by aligning the low-frequency features of the student and teacher in the frequency domain, while suppressing the interference of high-frequency noise on the training process, thereby improving the quality and robustness of the reconstructed image.

[0125] Through the global spectrum fusion module, the student model, based on a lightweight convolutional network, can effectively absorb low-frequency information from the teacher model, enhance the perception of global structure by convolutional operations, and provide reliable frequency domain guidance for image reconstruction.

[0126] Step 4: Overall Optimization Goal

[0127] In addition to the two distillation losses mentioned above, it also includes task loss, which is the reconstructed image output by the student model. Compared to true high-resolution images L1 image reconstruction loss:

[0128] ;

[0129] in B represents the L1 image reconstruction loss for the task; B represents the batch size. Indicates the sample sequence number; This represents the b-th true high-resolution image; This represents the reconstructed image of the b-th student model.

[0130] The final overall optimization objective function is:

[0131] ;

[0132] in, This indicates space-constrained distillation loss; Indicates the loss during spectral distillation; The L1 image reconstruction loss represents the task. These are adjustable weighting coefficients used to balance the importance of spatial relation distillation and spectral knowledge distillation during training. Through this design, the student model can learn the global spatial attention and spectral features provided by the teacher model while maintaining a lightweight architecture, resulting in significant improvements in structural integrity, texture detail, and overall visual perception.

[0133] In addition, it should be noted that the high-resolution image reconstructed by the teacher model is not further processed. The distillation loss only includes two intermediate spatial constraint distillation losses and spectral distillation losses. It only has the loss of the intermediate layer and does not include the loss of the teacher-reconstructed image and the student-reconstructed image.

[0134] Step 5: Super-resolution reconstruction steps: After the model training is completed, input the low-resolution image to be processed and output the reconstructed high-resolution image.

[0135] Example 2

[0136] As another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, is used to implement the method described in the previous embodiment 1.

[0137] Computer-readable storage media can be non-volatile computer-readable storage media. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0138] Furthermore, the method embodiments provided in this disclosure can be executed on a computer terminal, server, or similar computing device.

[0139] To verify the effectiveness of the method of the present invention, a series of experiments were conducted.

[0140] The teacher model employs SwinIR, whose structure includes a convolutional layer for shallow feature extraction to capture low-level texture information. The backbone consists of four stacked residual Swin Transformer modules, each containing a multi-layer window attention structure with a window size of 8 and a feature dimension of 60, effectively modeling global image dependencies. After backbone feature processing, the model finally achieves ×4 upsampling through convolutional layers and a PixelShuffle operation to generate a high-resolution image.

[0141] The student model employs EDSR, which also uses convolutional layers to extract shallow features and sixteen residual blocks for deep feature extraction. Each residual block contains two 3×3 convolutional layers, enabling it to progressively learn detailed information from low-resolution images. Finally, ×4 upsampling is achieved through convolutional layers and PixelShuffle to obtain the reconstructed image. .

[0142] In this experiment, distillation is performed only at the feature layer between the teacher and student models. The student network learns the spatial attention features and spectral features of the teacher model through feature alignment in a convolutional structure to assist in high-quality reconstruction.

[0143] During training, this invention uses the DIV2K dataset as the training set and performs ×4 downsampling on the high-resolution images using bicubic interpolation to generate the corresponding low-resolution input. To enhance the model's generalization ability, data augmentation was performed on the training data, including random horizontal flipping and rotation operations. The Adam optimizer was used, with an initial learning rate of 1×10⁻⁻⁻⁴. 4The momentum parameter β = (0.9, 0.99) is used without weight decay, and exponential moving average (EMA, decay = 0.999) is combined to stabilize model parameter updates. The learning rate is scheduled using the MultiStepLR strategy, decaying to 0.1 of the original value at the 100k and 200k iterations, with a total of 250k training iterations and no warm-up phase.

[0144] In terms of loss design, this invention combines three types of loss functions, including spatially constrained distillation loss. Spectral distillation loss and the L1 image reconstruction loss of the task ,in Both weights are set to 1.0.

[0145] During the validation phase, model performance is evaluated every 5k iterations. The performance metrics include Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR measures the pixel-level difference between the generated image and the real image, measured in dB (decibels); a higher value indicates that the reconstructed image is closer to the original image. SSIM evaluates the structural, brightness, and contrast similarity of the image, with a value ranging from [0,1]; values ​​closer to 1 indicate that the reconstructed image's structure is closer to the original image.

[0146] Table 1. Performance comparison of different methods on ×4 super-resolution tasks

[0147]

[0148] As can be seen from the experimental results in Table 1:

[0149] The PSNR of the traditional EDSR model on Set5, Set14 and BSDS100 are 29.07, 26.59 and 26.33, respectively, and the SSIM are 0.8119, 0.7183 and 0.6852, respectively, which shows the limitations of standard lightweight convolutional networks in global dependency modeling.

[0150] The EDSR obtained by using conventional KD knowledge distillation showed significant improvement on all test sets, indicating that knowledge transfer from the teacher model helps improve the structure perception and detail reconstruction capabilities of the student network.

[0151] The method proposed in this invention further improves PSNR and SSIM, for example, reaching 32.01 and 0.8921 respectively on Set5, which are about 0.17 dB and 0.0023 higher than conventional KD, respectively. This shows that by extracting the global spatial information and spectral features of the deep features of the teacher model and guiding the student network to perform feature distillation, the global modeling ability and detail reconstruction effect of lightweight CNN can be effectively enhanced.

[0152] In summary, the feature layer distillation method proposed in this invention achieves improved super-resolution performance of a lightweight student model on multiple datasets without increasing the computational burden of inference, verifying the effectiveness and practicality of the global feature guidance strategy.

[0153] Furthermore, in the experiments of this invention, to intuitively evaluate the super-resolution reconstruction performance of the student model under different distillation methods, we performed a visual comparison of typical samples in the test set. The visualization comparison results are as follows: Figure 3 As shown, the reconstruction results include the original high-resolution image (GT), the downsampled low-resolution input image (LR), the undistilled student model (EDSR), the student model based on traditional knowledge distillation (EDSR-KD), and the student model distilled by the method of this invention (EDSR-Ours).

[0154] By observing the visualizations of the Set5, Set14, and BSDS100 datasets, we can find that:

[0155] In the recovery of high-frequency fine structures, EDSR (the present invention) has a more significant advantage over EDSR (Student Model) and EDSR (KD). Taking bird images from the Set5 dataset as an example, this method can more accurately restore the detailed texture and edge clarity of the bird's eye region; in baboon images from the Set15 dataset, this method reconstructs high-frequency fine line structures such as whiskers more clearly and stably; in architectural scenes from the BSDS100 dataset, this method can recover more continuous and regular wall brick seams and roof textures, demonstrating strong structural consistency and detail expression capabilities.

[0156] Based on the above observations, the visualization comparison of this invention further demonstrates that through the cross-architecture distillation mechanism guided by global features, the student model can learn the spatial attention feature information and spectral features contained in the teacher model under a lightweight convolutional structure, thereby achieving higher reconstruction accuracy and visual quality in complex texture reconstruction tasks.

[0157] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should be protected by the present invention.

Claims

1. A lightweight image super-resolution method based on ViT-CNN heterogeneous distillation, characterized in that, Includes the following steps: The data input steps are as follows: input low-resolution images into the pre-trained teacher model and the student model to be trained, where the teacher model is a ViT model based on the Transformer architecture and the student model is a lightweight model based on CNN. The steps for spatial relationship alignment are as follows: extract deep features from the teacher model and generate teacher attention features, extract deep features from the student model and generate student attention features; fuse the two attention features to generate common attention features; calculate the spatial constraint distillation loss based on the teacher attention features, student attention features and common attention features to achieve cross-architecture spatial knowledge transfer and alignment. The steps for generating teacher attention features specifically include: Flatten the deep feature map of the teacher model into a token sequence format; Global average pooling is performed on the token sequence to obtain the global features of the teacher model; The teacher model calculates the spatial relationship weights between the global features of the teacher model and the token sequence through a self-attention mechanism, and then normalizes them using Softmax to obtain the teacher attention features. The steps for generating student attention features specifically include: Global average pooling is performed on the deep feature maps of the student model to obtain the global features of the student. By using learnable projection transformations and nonlinear mappings, the student global features are mapped to a high-dimensional attention space compatible with the teacher's attention features, thus obtaining the student attention features. The steps for synthesizing common attention features are as follows: Teacher attention characteristics and student attention characteristics are spliced ​​together along the channel dimension; The concatenated features are then passed through a projection network containing linear layers, layer normalization, ReLU activation function, and Sigmoid activation function to synthesize common attention features. The formula for calculating space-constrained distillation loss is as follows: ； in, This indicates space-constrained distillation loss, and B represents the batch size. Indicates the sample number. This represents the attentional characteristics of the b-th student. Show the first A characteristic of public attention, Indicates the first Teacher attention characteristics represents the square of the L2 norm, and ⊙ represents element-wise multiplication; The steps of global spectrum fusion are as follows: extract the global features of the deep features of the teacher model and transform them to the frequency domain to obtain the teacher's spectrum features; extract the global features of the deep features of the student model and transform them to the frequency domain to obtain the student's spectrum features; map the student's spectrum features to the same dimension as the teacher's spectrum features, and calculate the spectral distillation loss between the low-frequency components of the two to achieve frequency domain knowledge transfer. The steps for loss calculation are as follows: Calculate the image reconstruction loss based on the super-resolution reconstruction output of the student model and the real high-resolution image; use the weighted sum of spatial constraint distillation loss, spectral distillation loss and image reconstruction loss as the total loss to train and optimize the student model.

2. The lightweight image super-resolution method based on ViT-CNN heterogeneous distillation according to claim 1, characterized in that, The formula for calculating the loss in spectral distillation is as follows: ； in, This indicates the spectral distillation loss, and B represents the batch size. Indicates the sample sequence number; This represents the spectral characteristics of the b-th aligned student. The b-th element represents the spectral characteristics of the teacher. This indicates the extraction of low-frequency components.

3. The lightweight image super-resolution method based on ViT-CNN heterogeneous distillation according to claim 1, characterized in that, The formula for calculating image reconstruction loss is as follows: ； in B represents the L1 image reconstruction loss for the task; B represents the batch size. Indicates the sample sequence number; This represents the b-th true high-resolution image; This represents the reconstructed image of the b-th student model; The final overall optimization objective function is: ； in, This indicates space-constrained distillation loss; Indicates the loss during spectral distillation; The L1 image reconstruction loss represents the task. These are the weighting coefficients.

4. The lightweight image super-resolution method based on ViT-CNN heterogeneous distillation according to claim 1, characterized in that, It also includes a super-resolution reconstruction step: after the model is trained, the input is a low-resolution image to be processed, and the output is a reconstructed high-resolution image.

Images