Image super-resolution method based on layered sensing distillation

Abstract

The invention discloses an image super-resolution method based on hierarchical perception distillation. The method comprises the following operation steps: step 1, inputting a low-resolution image data set; 2, a shallow feature extraction process; 3, a deep feature extraction and enhancement process: processing the image through a deeper network layer to extract complex high-level features; step 4, image reconstruction and optimization: after feature extraction and enhancement, the model integrates the extracted features; and 5, outputting a reconstruction result: outputting the processed and reconstructed image as a high-resolution version, wherein the output result shows significant visual improvement. Compared with the prior art, the method has the advantages that the network structure of the super-resolution reconstruction model is optimized, and the advantages of the convolutional neural network and the visual transformer are combined, so that the model which is lighter and more efficient is developed.

Description

Technical Field

[0001] The present invention relates to the field of image resolution methods, and in particular to an image super-resolution method based on layered perceptual distillation. Background Art

[0002] With the rapid development of image super-resolution reconstruction technology, high-quality super-resolution images have become increasingly important in various fields. This technology reconstructs high-resolution images with higher details and clarity from low-resolution images, meeting the urgent needs for high-resolution images in many fields including medical imaging, remote sensing, security monitoring, digital photography, etc. For example, in the medical field, super-resolution technology can improve the resolution of medical images such as CT and MRI, helping doctors obtain more accurate diagnostic information without increasing the radiation exposure of patients; in the field of public safety, the use of super-resolution technology can significantly improve the quality of surveillance images, thereby improving the accuracy of face recognition and license plate recognition, and providing more reliable evidence support. However, traditional super-resolution algorithms often rely on deep and complex network architectures, which leads to challenges of high computational complexity and a large number of parameters, making these algorithms difficult to deploy on edge devices with limited computing resources. To address these problems, this study is dedicated to optimizing the network structure of the super-resolution reconstruction model and developing a more lightweight and efficient model by combining the advantages of convolutional neural networks and visual transformers. These lightweight models can not only run efficiently in resource-constrained environments, but also reduce computing and storage requirements while ensuring image quality, thereby broadening the possibilities of super-resolution technology in practical applications.

[0003] Image super-resolution reconstruction technology has the following main technical problems under the existing convolutional neural network and visual transformer framework:

[0004] (1) Insufficient deep feature fusion: Current super-resolution models often have difficulty fully utilizing multi-level feature information when processing images with complex textures and structures, resulting in insufficient performance in detail recovery. This is mainly because traditional deep learning models are designed to be unable to effectively integrate local details and global context information at the same time, especially in lightweight network structures.

[0005] (2) High computational complexity: Existing super-resolution methods usually require high computing resources and long processing time. This high computing demand has become a major bottleneck, especially in real-time applications and mobile devices. The complex model structure and high number of parameters are difficult to deploy in resource-constrained environments, limiting the practical application of super-resolution technology.

[0006] (3) Difficulty in balancing lightweight and performance: Improving the reconstruction performance while keeping the model lightweight is a challenging task. Existing technologies often have difficulty balancing efficiency and effect in optimizing the number of parameters, computational complexity, and memory usage, making it difficult to achieve the expected image quality and processing speed on resource-constrained devices. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a network structure for optimizing the super-resolution reconstruction model, so as to develop a more lightweight and efficient model by combining the advantages of convolutional neural networks and visual transformers.

[0008] In order to solve the above problems, the technical solution of the present invention is: an image super-resolution method based on layered perceptual distillation, comprising the following steps:

[0009] Step 1: Input low-resolution image dataset: First, input a dataset containing low-resolution images (denoted as X), which contains N images. These low-resolution images enter the network after preprocessing and serve as the basic data for reconstructing high-resolution images. The input process ensures that the data is standardized and adjusted to a size suitable for network input before entering the model;

[0010] Step 2: Shallow feature extraction process: In the initial stage, the basic features are extracted from the low-resolution image through the shallow feature extraction module. Shallow feature extraction focuses on the low-level features of the image such as edge, color and simple texture;

[0011] Step 3: Deep feature extraction and enhancement process: The image is processed through deeper network layers to extract complex high-level features. The deep learning model enhances the details and texture representation through multi-scale feature extraction. At the same time, the introduction of the attention mechanism enables the network to dynamically adjust its attention to different image areas, giving priority to processing more informative image parts, ensuring that the details are fully restored in the final reconstruction process;

[0012] Step 4: Image reconstruction and optimization: After feature extraction and enhancement, the model integrates these extracted features and gradually restores the low-resolution image to a high-resolution image through upsampling layers and sub-pixel convolution layers. This process not only optimizes the image resolution, but also ensures that the output image is visually closer to the original high-resolution image through meticulous feature fusion.

[0013] Step 5: Output reconstruction results: The processed and reconstructed image is output as a high-resolution version. The output results show significant visual improvement and restore the missing details and textures in the original low-resolution image.

[0014] Furthermore, the shallow feature extraction process in step 2 requires the use of a larger number of convolutions, using three convolutional layers.

[0015] Furthermore, in step 4, the image reconstruction uses self-attention technology to process global detail information, and introduces multiple scale convolution blocks as a new feature extraction strategy to extract values, keys and queries, which improves the accuracy of feature reconstruction through intrinsic calculations, allowing the model to automatically highlight key visual features.

[0016] The advantages of the present invention compared with the prior art are:

[0017] The present invention proposes a novel, lightweight and efficient image super-resolution reconstruction technology, called an image super-resolution method based on hierarchical perceptual distillation. This method can effectively aggregate contextual information in the feature extraction module, promote the gradual refinement of feature extraction, and thus more accurately reconstruct high-frequency details. In this technology, the channel attention mechanism reconstructs the feature details of the image area in a multi-branch parallel manner, and the long-distance dependence of self-attention significantly enhances the ability to capture and reconstruct global information. DETAILED DESCRIPTION

[0018] In order to make the contents of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be described clearly and completely below.

[0019] An image super-resolution method based on layered perceptual distillation includes the following steps:

[0020] Step 1: Input low-resolution image dataset: First, input a dataset containing low-resolution images (denoted as X), which contains N images. These low-resolution images enter the network after preprocessing and serve as the basic data for reconstructing high-resolution images. The input process ensures that the data is standardized and adjusted to a size suitable for network input before entering the model;

[0021] Three convolutional layers are used. The key to this layer is to integrate global spatial relationships to capture a wide range of contextual information. This multi-level feature extraction method greatly enhances the performance and adaptability of the network. The specific process can be expressed as:

[0022]

[0023] where F d_i Indicates the distillation characteristics. r_i Represents a refinement feature. cd Represents the features after channel dimension reduction. In order to better integrate the extracted features, we use contextual collaborative attention and multi-scale deep convolution sparse attention for deep feature fusion, which can well balance the enhanced details and global context information.

[0024]

[0025] Among them, F agg is an aggregate feature. Represents a 1×1 convolutional layer that changes the number of channels in order to keep the same shape as the input. Represents channel attention calculation. Represents the self-attention computation.

[0026] Step 2: Shallow feature extraction process: In the initial stage, the basic features are extracted from the low-resolution image through the shallow feature extraction module. Shallow feature extraction focuses on the low-level features of the image such as edge, color and simple texture;

[0027] Step 3: Deep feature extraction and enhancement process: The image is processed through deeper network layers to extract complex high-level features. The deep learning model enhances the details and texture representation through multi-scale feature extraction. At the same time, the introduction of the attention mechanism enables the network to dynamically adjust its attention to different image areas, giving priority to processing more informative image parts, ensuring that the details are fully restored in the final reconstruction process;

[0028] Step 4: Image reconstruction and optimization: After feature extraction and enhancement, the model integrates these extracted features and gradually restores the low-resolution image to a high-resolution image through upsampling layers and sub-pixel convolution layers. This process not only optimizes the image resolution, but also ensures that the output image is visually closer to the original high-resolution image through meticulous feature fusion.

[0029] Step 5: Output reconstruction results: The processed and reconstructed image is output as a high-resolution version. The output results show significant visual improvement and restore the missing details and textures in the original low-resolution image.

[0030] The shallow feature extraction process in step 2 requires a large number of convolutions, using three convolutional layers.

[0031] In step 4, the image reconstruction uses self-attention technology to process global detail information, and introduces multiple scale convolution blocks as a new feature extraction strategy to extract values, keys and queries. The accuracy of feature reconstruction is improved through intrinsic calculation, so that the model automatically highlights key visual features.

[0032] In specific usage, for the input feature X, first, the query, key and value are generated by the deep convolution block. Then, the deformation function is applied to the query, key and value respectively to generate new query, key and value. Secondly, the self-attention of multiple scales is spliced ​​along the channel dimension. The key features can be marked in a multi-scale manner, thereby achieving a good balance between performance and parameters. A point-by-point convolution is used to integrate with the generated attention. Finally, in order to transform the features, the regular feed-forward network performs the same operation on each pixel position separately. It uses two 1×1 convolutions, one for expanding the feature channel (usually by factor γ=4), and the second for reducing the channel to the original input dimension. It contains deep convolutions to encode information from spatially adjacent pixel positions, which is useful for learning local image structures for effective restoration.

[0033]

[0034] X1=W pc X1+X

[0035]

[0036] Among them, Attention(Q1,K1,V1) represents the features after self-attention calculation, X1 represents the output features after concatenating multiple scale features, and W pc , represents point-wise convolution, Represents a deep convolution, a learnable parameter used before the Rel u function, which manipulates the size of the dot product of Q1 and K1. Re lu is used to obtain the most useful attention, and through the self-attention part, generates the output aggregate feature X1. Then X1 is input into the feedforward network to generate the output X2.

[0037] In step 4, the L1 loss function is selected to optimize the image super-resolution reconstruction process. The L1 loss, also known as the minimum absolute deviation, measures the reconstruction quality by calculating the absolute value difference between the predicted image and the real image. The calculation formula of the L1 loss is as follows:

[0038]

[0039] In this formula, y i represents the pixel value in the real image, Represents the corresponding pixel value in the predicted image, and N is the total number of pixels. By minimizing this loss, the model is trained to more accurately reconstruct high-resolution images while reducing errors in the reconstruction process, thereby improving the overall image quality. The use of L1 loss ensures the visual accuracy and quality of the reconstruction results, which is particularly critical in image super-resolution applications.

[0040] In step one, model training requires a data set and a training algorithm, while the performance of the policy model requires multiple evaluation indicators.

[0041] Dataset

[0042] Training dataset: D IV2K: This dataset consists of 800 high-definition images and is also widely used in super-resolution research. The D IV2K dataset is regarded as a standard testing platform for super-resolution tasks due to its high-quality images and diverse content. F l ickr2K: This dataset contains 2,650 high-resolution images and comes from the F l ickr website.

[0043] Test dataset: Set5: A small benchmark dataset commonly used to evaluate image super-resolution algorithms. It contains 5 images covering a range of different themes and scenes, such as natural scenery and portraits.

[0044] Data augmentation techniques

[0045] The present invention adopts a variety of standard data enhancement techniques during the training process, including random rotation and horizontal flipping. These operations can help the model learn how to process images from different angles and directions, so as to better adapt to various situations that may be encountered in practical applications.

[0046] Training Algorithms

[0047] The present invention integrates six feature blocks in the backbone network, and sets the channel size to 48 in the distillation module and the attention mapping module. The model is trained using the Adam optimizer, and the optimization parameters are set to β1=0.9, β2=0.99. The learning rate starts from 0.002 and is strategically halved according to [100k, 500k, 800k, 900k, 950k] iterations until one million iterations are completed.

[0048] Evaluation Metrics

[0049] Peak signal-to-noise ratio (PSNR) is a widely used metric for measuring image quality, especially in the field of image and video compression. It measures the similarity between the original image and the processed image. Based on the error level between the two, PSNR is calculated by taking the logarithm of the mean square error (MSE) between the original image and the reconstructed or compressed image and converting it to decibels (dB). The calculation formula is as follows:

[0050]

[0051] Among them, MAX Iis the maximum possible pixel value of the image (255 for 8-bit images), MSE is the average of the squares of the differences between the original image and the reconstructed image, and the higher the PSNR value, the better the image quality and the smaller the error.

[0052] The structural similarity index (SSIM) is used to measure the visual similarity between two images. Unlike pixel-based error measurements (such as MSE and PSNR), SSIM takes into account the structural information of the image, namely brightness, contrast, and structure. The SSIM value is between -1 and 1, with 1 representing exact similarity. The calculation of SSIM involves three comparison measurements: brightness comparison, contrast comparison, and structure comparison. The calculation formula is as follows:

[0053]

[0054] Among them, x and y are the two image windows to be compared, μ x , μ y are the means of x and y respectively, is the variance of x and y, σ xy is xy The covariance of , c1 and c2 are small constants used to maintain stability. SSIM provides a quality evaluation method for image processing that is closer to human perception.

[0055] The present invention and its implementation methods are described above, and such description is not restrictive, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design structures and embodiments similar to the technical solution without creativity, which should all fall within the protection scope of the present invention.

Claims

1. An image super-resolution method based on hierarchical perceptual distillation, characterized by: The steps include: Step 1: Input low-resolution image dataset: First, input a dataset containing low-resolution images (denoted as X), which contains N images. These low-resolution images enter the network after preprocessing and serve as the basic data for reconstructing high-resolution images. The input process ensures that the data is standardized and adjusted to a size suitable for network input before entering the model; Step 2: Shallow feature extraction process: In the initial stage, the basic features are extracted from the low-resolution image through the shallow feature extraction module. Shallow feature extraction focuses on the low-level features of the image such as edge, color and simple texture; Step 3: Deep feature extraction and enhancement process: The image is processed through deeper network layers to extract complex high-level features. The deep learning model enhances the details and texture representation through multi-scale feature extraction. At the same time, the introduction of the attention mechanism enables the network to dynamically adjust its attention to different image areas, giving priority to processing more informative image parts, ensuring that the details are fully restored in the final reconstruction process; Step 4: Image reconstruction and optimization: After feature extraction and enhancement, the model integrates these extracted features and gradually restores the low-resolution image to a high-resolution image through upsampling layers and sub-pixel convolution layers. This process not only optimizes the image resolution, but also ensures that the output image is visually closer to the original high-resolution image through meticulous feature fusion. Step 5: Output reconstruction results: The processed and reconstructed image is output as a high-resolution version. The output results show significant visual improvement and restore the missing details and textures in the original low-resolution image.

2. The image super-resolution method based on hierarchical perceptual distillation according to claim 1, characterized in that: The shallow feature extraction process in step 2 requires a large number of convolutions, using three convolutional layers.

3. The image super-resolution method based on hierarchical perceptual distillation according to claim 1, characterized in that: In step 4, the image reconstruction uses self-attention technology to process global detail information, and introduces multiple scale convolution blocks as a new feature extraction strategy to extract values, keys and queries. The accuracy of feature reconstruction is improved through intrinsic calculation, so that the model automatically highlights key visual features.

Images