Methods, apparatus, computer equipment and storage media for single-image super-resolution reconstruction

Abstract

This application relates to a method, apparatus, computer device, and storage medium for single-image super-resolution reconstruction. The method includes: performing full binarization optimization on a VIT model; sequentially binarizing and specifically optimizing the multi-head self-attention mechanism, feedforward network, and convolutional layer to obtain an optimized binarization module; introducing a consistency-aware binarization mechanism into the optimized binarized multi-head self-attention mechanism; fusing computation to enhance the similarity between the query and key matrix; adjusting the distribution differences of the three through learnable parameters and residual connections; constructing a full-binarization super-resolution model; iteratively training the model to a predetermined number of training epochs and performance metrics; inputting the low-resolution image to be reconstructed into the model; and outputting the corresponding high-resolution image. This method can improve the model's feature representation capability and achieve high-quality single-image super-resolution reconstruction in low-resource scenarios.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer device, and storage medium for super-resolution reconstruction of a single image. Background Technology

[0002] Single Image Super-Resolution (SISR) aims to reconstruct high-quality, high-resolution images from low-resolution images, and has significant application value in scenarios such as motion image enhancement, security monitoring, medical imaging, and remote sensing interpretation. With the development of deep learning, super-resolution methods based on convolutional neural networks (CNNs) have significantly improved reconstruction accuracy. In recent years, Transformer-based super-resolution models have further utilized self-attention mechanisms to capture long-range dependencies, achieving better results in detail restoration and texture reconstruction. However, the Transformer architecture typically introduces higher computational overhead and storage requirements, limiting its deployment and application on practical hardware platforms.

[0003] To improve the usability of super-resolution models on edge devices, academia and industry have proposed various technical approaches, including model quantization, knowledge distillation, and efficient network architecture design. Among these, binary neural networks (BNNs) stand out as a form of extreme quantization, which constrains weights and activations as { The method {1,+1} can bring significant storage compression and can leverage bitwise operations such as XNOR-Popcount to achieve advantages in inference speed and energy consumption. Therefore, it is considered an important direction for achieving efficient inference in resource-constrained scenarios.

[0004] Although existing binarized Transformers (such as the binary ViT method for classification / detection tasks) have been explored in high-level vision tasks, when they are directly transferred to low-level vision reconstruction tasks such as super-resolution, they often exhibit significant performance degradation and fail to meet the requirements for detail representation and pixel-level accurate modeling. Therefore, there is an urgent need to design a dedicated binarized Transformer scheme for the characteristics of super-resolution tasks. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, apparatus, computer equipment, and storage medium for single-image super-resolution reconstruction that can significantly reduce model computation and storage overhead, effectively suppress attention bias and feature degradation caused by binarization, improve the feature representation capability of the fully binarized Transformer, and achieve high-quality reconstruction in low-resource scenarios, in order to address the above-mentioned technical problems.

[0006] A method for super-resolution reconstruction of a single image, the method comprising: Obtain an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images; The Vision Transformer model is optimized by full binarization. Specifically, the multi-head self-attention mechanism, the feedforward network, and the convolutional layer are binarized in sequence. The binarized structure is then optimized to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. A consistency-aware binarization mechanism is introduced into the optimized binarized multi-head self-attention mechanism. The similarity between the query matrix and the key matrix is ​​enhanced by fusion computation. Learnable parameters and residual connections are introduced to adjust the distribution differences of the query matrix, key matrix and value matrix, and a fully binarized Transformer super-resolution model is constructed. The fully binarized Transformer super-resolution model is iteratively trained using the image dataset until a predetermined number of training rounds and performance metrics are reached, resulting in a trained super-resolution reconstruction model. Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

[0007] In one embodiment, the Vision Transformer architecture employs the SwinIR model.

[0008] In one embodiment, targeted optimization of the binarized multi-head self-attention mechanism includes: A LayerNorm layer is introduced into the binarized multi-head self-attention mechanism to normalize the binarized features; A softmax-free attention mechanism is used to directly binarize the inner product of the query matrix and the key matrix.

[0009] In one embodiment, targeted optimization of the binarized feedforward network includes: Residual extended skip connections are introduced into each layer of the binarized feedforward network to expand the dimension of the input features; The GELU activation function is introduced into the binarized feedforward network as a nonlinear activation function.

[0010] In one embodiment, targeted optimization of the binarized convolutional layer includes: A residual alignment unit is added to the binarized convolutional layer to fuse the output of the binarized convolution operation with the features of the residual connection, and then the fused feature data is input into the PReLU activation function for correction processing.

[0011] In one embodiment, the consistency-aware binarization mechanism includes: A fusion calculation is performed on the query matrix and the key matrix, a learnable spatial scale parameter is introduced, and a residual connection structure is added. By cooperating with the spatial scale parameter and the residual connection structure, the distribution of the query matrix, the key matrix and the value matrix are adjusted.

[0012] In one embodiment, the fully binarized Transformer super-resolution model includes a base model BiTransSR and an optimized model BiTransSR-W; Compared to BiTransSR, BiTransSR-W increases the channel width while reducing the number of network layers of the Swing Transformer in each unit.

[0013] This application also provides a single-image super-resolution reconstruction apparatus, the apparatus comprising: The data acquisition module is used to acquire an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images. The model binarization optimization module is used to perform full binarization optimization on the Vision Transformer model. Specifically, the multi-head self-attention mechanism, feedforward network, and convolutional layer are binarized in sequence, and the structure after binarization is optimized again to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. The core mechanism embedding module is used to introduce a consistency-aware binarization mechanism into the optimized binarized multi-head self-attention mechanism. It enhances the similarity between the query matrix and the key matrix through fusion computation, and introduces learnable parameters and residual connections to adjust the distribution differences of the query matrix, key matrix and value matrix, thus constructing a fully binarized Transformer super-resolution model. The model training module is used to iteratively train the fully binarized Transformer super-resolution model using the image dataset until a predetermined number of training rounds and performance indicators are reached, thereby obtaining a trained super-resolution reconstruction model. The super-resolution reconstruction module is used to acquire the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

[0014] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described single-image super-resolution reconstruction method.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described single-image super-resolution reconstruction method.

[0016] The aforementioned single-image super-resolution reconstruction method, apparatus, computer equipment, and storage medium optimize the VisionTransformer model by performing full binarization optimization. Specifically, the multi-head self-attention mechanism, feedforward network, and convolutional layer are binarized sequentially, and the binarized structure is further optimized to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. A consistency-aware binarization mechanism is introduced into the optimized binarized multi-head self-attention mechanism. The similarity between the query matrix and the key matrix is ​​enhanced through fusion calculation, and learnable parameters and residual connections are introduced to adjust the distribution differences of the query matrix, key matrix, and value matrix, thus constructing a fully binarized Transformer super-resolution model. The fully binarized Transformer super-resolution model is iteratively trained using an image dataset until a predetermined number of training rounds and performance indicators are reached, resulting in a trained super-resolution reconstruction model. A low-resolution image is input into the super-resolution reconstruction model, and the corresponding high-resolution reconstructed image is output.

[0017] This method binarizes and optimizes the multi-head self-attention mechanism, feedforward network, and convolutional layers of the Vision Transformer model sequentially. A consistency-aware binarization mechanism is then introduced into the optimized binarized multi-head self-attention mechanism. This enhances the similarity between the query matrix and the key matrix through fusion computation. Furthermore, learnable parameters and residual connections are used to adjust the distribution differences between the query matrix, key matrix, and value matrix. This effectively suppresses attention bias and feature representation degradation caused by binarization, reducing information distortion. While achieving a lightweight, fully binarized transformation of the Vision Transformer model, significantly reducing computational and storage overhead, and improving hardware deployment adaptability, this method effectively preserves and enhances the model's feature representation capabilities, enabling it to better capture image feature associations. Simultaneously, a dataset containing low-resolution sample images and corresponding high-resolution sample images is used to construct the fully binarized Transformer. The super-resolution model is iteratively trained until the predetermined number of training rounds and performance indicators are reached. This fully optimizes the model parameters and ensures that the trained super-resolution reconstruction model has stable and reliable pixel-level accurate modeling capabilities. It can effectively convert the low-resolution image to be reconstructed into the corresponding high-resolution image, realizing high-quality single-image super-resolution reconstruction in low-resource scenarios. Moreover, the overall method and process are coherent. Optimization is carried out based on the pre-trained model without designing the network architecture from scratch, which simplifies the model development and training process and provides an efficient and feasible technical solution for single-image super-resolution reconstruction. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a single-image super-resolution reconstruction method in one embodiment; Figure 2 This is a schematic diagram of the SwinIR model in one embodiment; Figure 3 This is a schematic diagram comparing BiMHSA, BiFFN, and BiConv before and after the improvement in one embodiment. Figure 3 (a) shows the structure of the original BiMHSA network and a schematic diagram of the structures of BiFFN and BiConv within it. Figure 3 (b) shows the structure of the improved BiMHSA network and a schematic diagram of the structures of BiFFN and BiConv therein; Figure 4 This is a schematic diagram comparing the structures of the BiTransSR and BiTransSR-W models in one embodiment, where... Figure 4 (a) is a schematic diagram of the BiTransSR model structure. Figure 4 (b) is a schematic diagram of the BiTransSR model structure; Figure 5 This is a structural block diagram of a single-image super-resolution reconstruction device in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] To address the existing problems in the technology, such as the high computational and storage overhead of Transformer super-resolution models making them difficult to deploy on edge devices, the significant performance degradation of directly transferring binarized Transformers to low-level super-resolution visual tasks, and the tendency of binarization to cause degradation of Q / K / V representations in self-attention layers, masking of residual information transmission, and decreased feature representation capabilities, thus failing to maintain high-quality super-resolution reconstruction while reducing model complexity, this application addresses these issues. Figure 1 As shown, a single-image super-resolution reconstruction method is provided, which specifically includes the following steps: Step S100: Obtain an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images.

[0021] Step S110: The Vision Transformer model is fully binarized and optimized. In this step, the multi-head self-attention mechanism, the feedforward network, and the convolutional layer are binarized in sequence. The binarized structure is then optimized to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer.

[0022] Step S120: In the optimized binarized multi-head self-attention mechanism, a consistency-aware binarization mechanism is introduced. The similarity between the query matrix and the key matrix is ​​enhanced by fusion calculation. Learnable parameters and residual connections are introduced to adjust the distribution differences of the query matrix, key matrix and value matrix, and a fully binarized Transformer super-resolution model is constructed.

[0023] Step S130: The fully binarized Transformer super-resolution model is iteratively trained using the image dataset until the predetermined number of training rounds and performance indicators are reached, thereby obtaining the trained super-resolution reconstruction model.

[0024] Step S140: Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

[0025] In step S100, the image dataset is a high- and low-resolution pixel-level paired dataset specifically for the Single Image Super-Resolution (SISR) task. Both datasets are publicly recognized standard datasets in the field and are divided into two categories according to their purpose: training dataset and test dataset. Both types of datasets contain multiple low-resolution sample images, and each low-resolution sample image has a unique corresponding high-resolution sample image, providing accurate and effective data support for the model to learn the feature mapping relationship from low-resolution images to high-resolution images.

[0026] In this embodiment, the training dataset uses the DIV2K super-resolution public dataset, which contains 800 high-resolution original images. The corresponding low-resolution sample images are generated from the high-resolution images through downsampling, blurring, and other operations that simulate image degradation in real-world scenes, ensuring the consistency of pixel pairing between high- and low-resolution images. This allows the model to effectively learn the super-resolution reconstruction rules of images in different scenes. The test dataset uses standard test sets commonly used in SISR tasks, specifically Set5, Set14, B100, Urban100, and Manga109. Each test set contains 5, 14, 100, 100, and 109 pairs of high- and low-resolution paired images, respectively. The image types cover various scenes such as natural landscapes, portraits, urban buildings, and cartoon textures, which can comprehensively verify the reconstruction performance of the subsequently trained super-resolution model in different image scenes.

[0027] In step S110, the selected Vision Transformer model is the SwinIR model (i.e., the full-precision baseline model), and its basic network structure is as follows: Figure 2 As shown, this model is a Transformer model specifically designed for single-image super-resolution tasks, and it forms the structural basis for this full binarization optimization.

[0028] In this embodiment, with Figure 3 (a) shows the binarized multi-head self-attention mechanism (BiMHSA), binarized feedforward network (BiFFN), and binarized convolutional layer (BiConv) in the standard BiSwinIR model as the binarization benchmark. Binarization transformations are then performed sequentially on the multi-head self-attention mechanism, feedforward network, and convolutional layer in the SwinIR model. Furthermore, to address issues such as feature degradation and information transmission obstruction after binarization, targeted optimizations are made to each binarization structure, ultimately yielding the desired result. Figure 3 (b) shows the improved and optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer.

[0029] In this embodiment, the binarization attention mechanism in BiViT can be written as:

[0030] In the above formula, Q, K, and V represent the query, key, and value matrices, respectively. To address the quantization error caused by binarization, BiViT proposed "Softmax-aware binarization," which dynamically calculates a threshold for the attention vector of each row. The threshold formula is:

[0031] In the above formula, The first after softmax Line 1 One value, These are custom coefficients. This is a step function that returns 1 or 0.

[0032] In this embodiment, the native multi-head self-attention mechanism in the SwinIR model is first binarized to obtain the basic binarized multi-head self-attention mechanism (BiMHSA), corresponding to... Figure 3(a) The BiMHSA module of the standard BiSwinIR. Since binarization typically leads to a decrease in similarity between Q, K, and V, traditional attention mechanisms cannot effectively capture long-range dependencies. Therefore, this basic BiMHSA is further optimized in two ways: First, a LayerNorm layer is introduced into the binarized multi-head self-attention mechanism to normalize the binarized features, enhancing the information content of the binarized features and alleviating the feature representation degradation problem caused by binarization. Second, the traditional Softmax attention calculation method is abandoned, and a Softmax-free attention mechanism is adopted, directly performing binarization on the inner product of the query matrix and the key matrix, improving the computational efficiency of the attention mechanism while avoiding the sparsity problem caused by Softmax calculation. After the above optimizations, the following is obtained: Figure 3 (b) shows the improved binarized multi-head self-attention mechanism. Finally, the formula for calculating the attention score was modified as follows:

[0033] In the above formula, Represents a symbolic function. This indicates a binarization operation. This represents the scaling factor for the feature dimension.

[0034] In this embodiment, the native feedforward network in the SwinIR model is binarized to obtain the basic binarized feedforward network (BiFFN), corresponding to... Figure 3 (a) The BiFFN module of the standard BiSwinIR. Since the binarized convolution operation may lead to a long-tailed distribution, resulting in a large difference between the feature value range of the output and the residual connection, a residual alignment module is added to reduce this difference and ensure the effective transfer of information in the model.

[0035] Specifically, in the targeted optimization of the basic BiFFN, firstly, residual extended skip connections are introduced into each layer of the binarized feedforward network, and the dimensionality of the input features is expanded. This structural design effectively preserves feature information and alleviates the information loss problem caused by the binarized linear layers. Secondly, the GELU activation function is introduced into the binarized feedforward network as a non-linear activation function, improving the non-linear expressive power of the binarized feedforward network. After the above optimizations, the following is obtained: Figure 3 (b) shows the improved binarized feedforward network. The optimized binarized feedforward network is represented as follows:

[0036]

[0037] In this embodiment, the native convolutional layers in the SwinIR model are binarized to obtain the basic binarized convolutional layer (BiConv), corresponding to... Figure 3 (a) The BiConv module of the standard BiSwinIR. Since binarized convolution operations may lead to long-tailed distributions, resulting in a large difference between the feature value ranges of the output and the residual connections, a residual alignment module is added to reduce this difference and ensure the effective transfer of information in the model.

[0038] Specifically, when optimizing the basic BiConv, a residual alignment unit is added to the binarized convolutional layer. First, the output of the binarized convolution operation is fused with the features of the residual connections. Then, the fused feature data is input into the PReLU activation function for correction. Through the cooperation of the residual alignment unit and the PReLU activation function, the distribution difference between the binarized convolution output and the residual branch features is eliminated, ensuring the effective transmission of residual information in the model. After the above optimization, the following is obtained: Figure 3 (b) shows the improved binarized convolutional layer. The optimized binarized convolutional layer is represented as follows:

[0039] In step S110, based on Figure 2 The full-precision baseline model of SwinIR was used, and binarization and targeted optimization of the three core network structures were completed, ultimately yielding... Figure 3 (b) shows the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. The above optimized modules constitute the basic architecture of the fully binarized Transformer super-resolution model, laying the structural foundation for the subsequent introduction of a consistent perception binarization mechanism and the construction of a complete super-resolution model.

[0040] In step S120, with Figure 3 (b) The improved BiMHSA module is used as the core carrier. The Consistency-aware Binarization (CAB) mechanism proposed in this application is introduced to specifically solve the problems of query matrix (Q), key matrix (K), and value matrix (V) representation degradation, similarity reduction and distribution mismatch caused by binarization operation. Through the collaborative design of feature fusion, learnable parameter adjustment and residual connection, the feature characterization ability of the binarized self-attention layer is restored and improved. Finally, the optimized BiMHSA with embedded CAB mechanism is modularly integrated with the optimized BiFFN and BiConv obtained in step S110 to build the basic architecture of the fully binarized Transformer super-resolution model.

[0041] In this embodiment, the CAB mechanism is specifically embedded into the BiMHSA module optimized in step S110 (corresponding to...). Figure 3 (b) The core computation node of the improved BiMHSA module. This module has completed the design of introducing the LayerNorm layer and a softmax-free attention mechanism, providing a good feature processing foundation for the implementation of the CAB mechanism. The core design goal of the CAB mechanism is to compensate for the information loss of Q, K, and V features caused by binarization operations through active feature fusion and distribution adjustment in the binarization attention computation framework without softmax, thereby enhancing the similarity between Q and K from the root and correcting the feature distribution differences among the three. This solves the technical pain points of modeling bias in the binarization self-attention layer and weakened token relationship characterization ability, allowing the binarization BiMHSA module to still effectively capture the long-range dependencies of the image and adapt to the pixel-level feature learning requirements of super-resolution tasks.

[0042] Specifically, a fusion calculation is performed on the query matrix and the key matrix, a learnable spatial scale parameter is introduced, and a residual connection structure is added. By combining the spatial scale parameter with the residual connection structure, the distribution of the query matrix, the key matrix, and the value matrix is ​​adjusted.

[0043] Specifically, the calculation formula for the consistency-aware binarization mechanism is as follows:

[0044] In the above formula, and These are the learned spatial scale parameters.

[0045] In this embodiment, the fully binarized Transformer super-resolution model includes the basic model BiTransSR and the optimized model BiTransSR-W. Compared to BiTransSR, BiTransSR-W increases the channel width while reducing the number of Swing Transformer network layers in each unit. Both are constructed based on the aforementioned optimized binarized convolutional layers, a binarized multi-head self-attention mechanism with embedded CAB mechanism, and an optimized binarized feedforward network. Both adopt a fully binarized architecture design, uniformly constraining the model weights and activation values ​​to a binary form of {+1,-1}. Inference acceleration can be achieved by relying on XNOR-Popcount bit operations, preserving the super-resolution feature characterization capability while maintaining lightweight design.

[0046] BiTransSR is the basic model, strictly following the network hierarchy and module stacking logic of SwinIR. With extreme lightweight design as its core, it significantly reduces the number of model parameters and computational load, making it suitable for edge devices with limited computing power and storage, such as mobile terminals and portable security cameras. BiTransSR-W, on the other hand, makes targeted structural parameter adjustments to the BiTransSR architecture. Compared to BiTransSR, it increases the channel width from 60 to 120, while reducing the number of Swin Transformer network layers in each module from 6 to 3. Through this collaborative adjustment strategy, it improves the model's feature extraction and expression capabilities, captures more image detail features, and effectively controls the increase in computational complexity caused by the increased channel width, achieving a dual balance between feature expression capability and lightweight design.

[0047] The structural differences between the two models are intuitively reflected in Figure 4 In the comparison diagram of the BiTransSR and BiTransSR-W model structures, the overall process of image feature input, binarization feature processing, and high-resolution image output remains consistent, with only differences in core structural parameters. This ensures both technological commonality and deployment universality, while also flexibly adapting to different hardware resource conditions and super-resolution application requirements. BiTransSR emphasizes extreme lightweight design, making it suitable for resource-constrained scenarios. BiTransSR-W, on the other hand, achieves improved reconstruction performance with a slight increase in the number of parameters and computational load, making it suitable for edge computing scenarios with higher requirements for image reconstruction results and relatively abundant computing resources. This gives the proposed solution good scenario scalability and engineering feasibility.

[0048] In step S130, after training the fully binarized Transformer super-resolution model described above, a comprehensive evaluation is performed on multiple standard datasets. The lightweight effect of the model is measured by the number of parameters and computational cost (OPs), and the super-resolution efficiency is measured by metrics such as SSIM and PSNR. The formula for calculating computational cost (OPs) is as follows:

[0049] In the above formula, BOPs represents the number of binary operations, and FLOPs represents the number of floating-point operations.

[0050] In step S140, this step is the actual inference application stage of the model. In practical applications, the low-resolution images to be reconstructed can come from various super-resolution applicable scenarios such as mobile image acquisition, security monitoring capture, and remote sensing imaging. The image type is compatible with the scene features of the aforementioned training and test datasets, and no additional preprocessing operations such as block cropping are required, directly meeting the needs of real engineering applications.

[0051] In this embodiment, the low-resolution original image is input into the trained fully binarized Transformer super-resolution reconstruction model. The model will rely on the fully binarized architecture and XNOR-Popcount bit operations to complete efficient inference. It will sequentially pass through optimized binarized convolutional layers, a binarized multi-head self-attention mechanism with embedded CAB mechanism, and an optimized binarized feedforward network to complete feature extraction, long-range dependency capture, detail feature recovery, and high-resolution feature mapping of the low-resolution image. Finally, the model outputs a high-resolution reconstructed image corresponding to the input low-resolution image, completing the entire single-image super-resolution reconstruction process.

[0052] In step S140, the basic model BiTransSR or the optimized model BiTransSR-W can be flexibly selected for inference based on the hardware resources and reconstruction effect requirements of the actual application scenario. In edge scenarios where computing power and storage resources are highly limited, BiTransSR is selected to achieve lightweight and fast inference. In scenarios with higher requirements for image detail reconstruction quality and relatively abundant computing resources, BiTransSR-W is selected, achieving better high-resolution reconstruction results with a slight increase in inference time. The inference processes of the two models are consistent, with differences only in reconstruction accuracy and inference efficiency, giving the technical solution of this application good flexibility in practical applications.

[0053] This paper also demonstrates the effectiveness of the proposed method using experimental data. In the experimental results, a fully binary Transformer single-image super-resolution method was constructed based on the PyTorch framework. The baseline model used was SwinIR-light, and three binarization methods—BiBERT, BiT, and BiViT—were employed. Table 1 shows a comparison of the parameter count and computational cost between the proposed method and the baseline model. The model training underwent 300,000 iterations, with each batch containing 32 image patches of 64×64 pixels. The optimizer used was Adam, with an initial learning rate of 2×10⁻⁶. -4 It decays to half its original value on the 250,000th iteration.

[0054] Table 1. Parameter and computational cost distribution of the proposed method and SwinIR-light.

[0055] The pre-training dataset used was DIV2K, which, after processing, contained 800 high-resolution images and corresponding low-resolution images. The trained model was evaluated on the Set5, Set14, B100, Urban100, and Manga109 datasets, which contain 5, 14, 100, 100, and 109 pairs of high- and low-resolution images, respectively.

[0056] Furthermore, the proposed method and the SwinIR-based image super-resolution binarization model were tested on the five evaluation datasets mentioned above. The experimental results for multiple models are shown in Table 2. It can be seen that the proposed method achieves the best performance among all binarization models.

[0057] Table 2 compares the performance of the proposed method with the SwinIR-based image super-resolution binarization model on the evaluation dataset.

[0058] Next, the proposed method and an image super-resolution binarization model based on convolutional neural networks were tested on the Set5 and B100 datasets. The experimental results for multiple models are shown in Table 3. It can be seen that the proposed method achieves a good balance between performance and model complexity.

[0059] Table 3 compares the performance of the proposed method with that of the image super-resolution binarization model based on convolutional neural networks.

[0060] Experimental results show that the proposed method achieves performance similar to that of a full-precision network by optimizing the binarized multi-head self-attention mechanism, improving the binarized feedforward network and convolutional layers, and introducing a consistent-aware binarization approach, while greatly reducing the number of parameters and computational cost. This significantly enhances the representational ability of the full-binary Transformer network, effectively improves image super-resolution performance, and has good scalability.

[0061] In the aforementioned single-image super-resolution reconstruction method, the self-attention layer is optimized through Consistent Aware Binarization (CAB), significantly improving the representational capability of the binarization transformer. In traditional binarized Transformer models, the similarity between Q, K, and V is weakened by the binarization operation, leading to a decrease in the performance of the self-attention layer. However, by introducing CAB, the similarity between Q and K is effectively maintained, reducing information distortion and enabling the model to recover more detailed information in super-resolution tasks, thus improving the quality of the reconstructed image. This method significantly reduces computational and storage overhead. By adopting a fully binarized structure, not only are model weights and activation values ​​compressed to binary ({+1, -1}), but the computation process is also accelerated through XOR-Popcount operations. Compared with traditional full-precision models, the proposed method achieves a significant reduction in computational and storage requirements. Specifically, the number of model parameters and computational operations (OPs) are compressed by approximately 84.2% and 93.8%, respectively, significantly improving the deployment efficiency and running speed of the model on hardware platforms.

[0062] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0063] In one embodiment, such as Figure 5 As shown, a single-image super-resolution reconstruction device is provided, comprising: a data acquisition module 200, a model binarization optimization module 210, a core mechanism embedding module 220, a model training module 230, and a super-resolution reconstruction module 240, wherein: The data acquisition module 200 is used to acquire an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images. The model binarization optimization module 210 is used to perform full binarization optimization on the Vision Transformer model. Specifically, the multi-head self-attention mechanism, the feedforward network, and the convolutional layer are binarized in sequence, and the structure after binarization is optimized again to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. The core mechanism embedding module 220 is used to introduce a consistency-aware binarization mechanism into the optimized binarized multi-head self-attention mechanism. It enhances the similarity between the query matrix and the key matrix through fusion computation, and introduces learnable parameters and residual connections to adjust the distribution differences of the query matrix, key matrix and value matrix, thereby constructing a fully binarized Transformer super-resolution model. The model training module 230 is used to iteratively train the fully binarized Transformer super-resolution model using the image dataset until a predetermined number of training rounds and performance indicators are reached, so as to obtain the trained super-resolution reconstruction model. The super-resolution reconstruction module 240 is used to acquire the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

[0064] Specific limitations regarding the single-image super-resolution reconstruction device can be found in the limitations of the single-image super-resolution reconstruction method described above, and will not be repeated here. Each module in the aforementioned single-image super-resolution reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0065] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a single-frame image super-resolution reconstruction method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0066] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0067] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Obtain an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images; The pre-trained Vision Transformer model was used as the full-precision baseline model for full binarization optimization. Specifically, the multi-head self-attention mechanism, feedforward network, and convolutional layer were binarized in sequence, and the binarized structure was further optimized to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. A consistency-aware binarization mechanism is introduced into the optimized binarized multi-head self-attention mechanism. The similarity between the query matrix and the key matrix is ​​enhanced by fusion computation. Learnable parameters and residual connections are introduced to adjust the distribution differences of the query matrix, key matrix and value matrix, and a fully binarized Transformer super-resolution model is constructed. The fully binarized Transformer super-resolution model is iteratively trained using the image dataset until a predetermined number of training rounds and performance metrics are reached, resulting in a trained super-resolution reconstruction model. Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

[0068] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images; The pre-trained Vision Transformer model was used as the full-precision baseline model for full binarization optimization. Specifically, the multi-head self-attention mechanism, feedforward network, and convolutional layer were binarized in sequence, and the binarized structure was further optimized to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. A consistency-aware binarization mechanism is introduced into the optimized binarized multi-head self-attention mechanism. The similarity between the query matrix and the key matrix is ​​enhanced by fusion computation. Learnable parameters and residual connections are introduced to adjust the distribution differences of the query matrix, key matrix and value matrix, and a fully binarized Transformer super-resolution model is constructed. The fully binarized Transformer super-resolution model is iteratively trained using the image dataset until a predetermined number of training rounds and performance metrics are reached, resulting in a trained super-resolution reconstruction model. Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A single image super-resolution reconstruction method, characterized in that, The method includes: Obtain an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images; The Vision Transformer architecture designed for single-image super-resolution reconstruction is fully binarized and optimized. Specifically, the multi-head self-attention mechanism, feedforward network, and convolutional layer are binarized sequentially. The binarized structure is then further optimized to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. A consistency-aware binarization mechanism is introduced into the optimized binarized multi-head self-attention mechanism. The similarity between the query matrix and the key matrix is ​​enhanced by fusion computation. Learnable parameters and residual connections are introduced to adjust the distribution differences of the query matrix, key matrix and value matrix, and a fully binarized Transformer super-resolution model is constructed. The fully binarized Transformer super-resolution model is iteratively trained using the image dataset until a predetermined number of training rounds and performance metrics are reached, resulting in a trained super-resolution reconstruction model. Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

2. The single-image super-resolution reconstruction method according to claim 1, characterized in that, The VisionTransformer architecture uses the SwinIR model.

3. The single-image super-resolution reconstruction method according to claim 2, characterized in that, Targeted optimization of the binarized multi-head self-attention mechanism includes: A LayerNorm layer is introduced into the binarized multi-head self-attention mechanism to normalize the binarized features; A softmax-free attention mechanism is used to directly binarize the inner product of the query matrix and the key matrix.

4. The single-image super-resolution reconstruction method according to claim 2, characterized in that, Targeted optimization of the binarized feedforward network includes: Residual extended skip connections are introduced into each layer of the binarized feedforward network to expand the dimension of the input features; The GELU activation function is introduced into the binarized feedforward network as a nonlinear activation function.

5. The single-image super-resolution reconstruction method according to claim 2, characterized in that, Targeted optimization of the binarized convolutional layer includes: A residual alignment unit is added to the binarized convolutional layer to fuse the output of the binarized convolution operation with the features of the residual connection, and then the fused feature data is input into the PReLU activation function for correction processing.

6. The single-image super-resolution reconstruction method according to any one of claims 3-5, characterized in that, The consistency-aware binarization mechanism includes: A fusion calculation is performed on the query matrix and the key matrix, a learnable spatial scale parameter is introduced, and a residual connection structure is added. By cooperating with the spatial scale parameter and the residual connection structure, the distribution of the query matrix, the key matrix and the value matrix are adjusted.

7. The single-image super-resolution reconstruction method according to claim 6, characterized in that, The fully binarized Transformer super-resolution model includes the basic model BiTransSR and the optimized model BiTransSR-W; Compared to BiTransSR, BiTransSR-W increases the channel width while reducing the number of SwinTransformer network layers in each unit.

8. A single-image super-resolution reconstruction device, characterized in that, The device includes: The data acquisition module is used to acquire an image dataset, which includes multiple low-resolution sample images and corresponding high-resolution sample images. The model binarization optimization module is used to perform full binarization optimization on the Vision Transformer model. Specifically, the multi-head self-attention mechanism, feedforward network, and convolutional layer are binarized in sequence, and the structure after binarization is optimized again to obtain the optimized binarized multi-head self-attention mechanism, binarized feedforward network, and binarized convolutional layer. The core mechanism embedding module is used to introduce a consistency-aware binarization mechanism into the optimized binarized multi-head self-attention mechanism. It enhances the similarity between the query matrix and the key matrix through fusion computation, and introduces learnable parameters and residual connections to adjust the distribution differences of the query matrix, key matrix and value matrix, thus constructing a fully binarized Transformer super-resolution model. The model training module is used to iteratively train the fully binarized Transformer super-resolution model using the image dataset until a predetermined number of training rounds and performance indicators are reached, thereby obtaining a trained super-resolution reconstruction model. The super-resolution reconstruction module is used to acquire the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the super-resolution reconstruction model, and output the corresponding high-resolution reconstructed image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images