An image super-resolution reconstruction method, system, device and medium

Abstract

The application discloses an image super-resolution reconstruction method, system, device and medium, relates to the technical field of image super-resolution reconstruction, and comprises the following steps: extracting an image feature map; dividing an input feature map into multiple routing units, introducing a hybrid depth condition calculation in a deep feature extraction network composed of multiple residual blocks; and based on the output of the deep feature extraction network, reconstructing an image with a resolution higher than a threshold value. By introducing a hybrid depth condition calculation mechanism with displacement perception, the application designs a structured routing with displacement perception and a budget tendency for the shift window mechanism, so that the shift window sub-block is protected in terms of computing resources, and the cross-window information interaction link is more complete, so that block-like artifacts and fractures across the window boundary are less likely to occur when recovering long edges, repeated textures and large structures.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution reconstruction technology, and in particular to an image super-resolution reconstruction method, system, device and medium. Background Technology

[0002] Super-resolution (SR) image reconstruction aims to recover image details with a resolution higher than a threshold from a single image with a resolution lower than a threshold. It is a typical low-level visual reconstruction problem. Unlike high-level visual tasks such as classification and detection, SR focuses more on pixel-level texture details, edge shapes, and consistency across regions. Therefore, the model needs to strike a balance between global structure modeling and local high-frequency detail recovery.

[0003] Early reconstruction (SR) methods primarily relied on interpolation or prior-based optimization frameworks, which often struggled to recover realistic details in complex textures and fine structures. Subsequently, deep learning propelled SR technology into an end-to-end learning phase, with convolutional neural networks significantly improving reconstruction quality through strategies such as residual learning and multi-scale feature fusion. However, convolutional models are limited by their local receptive field, and capturing long-range dependencies often requires deeper networks or more complex modules. In high-magnification or extremely complex texture scenarios, issues such as texture repetition and edge distortion still easily arise.

[0004] In recent years, visual Transformers have utilized self-attention mechanisms to establish dependencies between pixels / blocks over a wider range. In particular, the SwinTransformer, through windowed self-attention and shifting window mechanisms, achieves cross-window information interaction while maintaining computational efficiency. Based on this idea, the SwinIR model stacks residual Swin Transformer blocks into a deep feature extraction backbone, achieving excellent results in image restoration tasks (including SR). However, SR tasks require high-resolution output, large feature map sizes, and high channel counts. SwinIR still incurs significant computational overhead during the inference stage; furthermore, its computation is spatially evenly distributed, failing to adaptively concentrate more computational power on texture-rich or edge-critical regions based on image content complexity.

[0005] In video image super-resolution reconstruction, to reduce computational costs, existing technologies have proposed various dynamic or sparse computation approaches. For example, based on the local content complexity of the input image, the network depth or computational paths involved in deep feature extraction can be adaptively reduced. Among these, the hybrid depth processing approach of "selectively performing recomputation based on importance" provides a general framework for introducing conditional computation while maintaining the overall network structure. However, directly applying such sparsification strategies to image reconstruction networks based on shifted window structures can lead to problems such as block artifacts, edge breaks, or texture fragmentation in the reconstructed image due to the improper weakening of cross-window information interaction and the disruption of feature map spatial continuity. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by providing an image super-resolution reconstruction method, system, device, and medium to solve the problems in the prior art.

[0007] The present invention specifically provides the following technical solution: An image super-resolution reconstruction method includes: Retrieve single-frame images with resolutions below a threshold from a video sequence; The single-frame image with a resolution below a threshold is input into the improved SwinIR network model to obtain a reconstructed image with a resolution above the threshold; wherein, the improved SwinIR network model uses SwinIR as the backbone network and introduces displacement-aware hybrid depth conditional sub-blocks in each residual block of its deep feature extraction unit; specifically including: Initial features are extracted from a single frame image to generate a preliminary feature map, which is then divided into multiple routing units; each routing unit represents a feature map grid. In each displacement-aware hybrid depth conditional sub-block, the importance of the routing unit is evaluated alternately using a regular window and a shift window. A first feature transformation is performed on all routing units to obtain basic features, and a second feature transformation is performed on routing units with importance higher than a threshold to obtain incremental features. The initial features, basic features, and incremental features input to each routing unit are differentially fused to obtain output features. The computational cost of the second feature transformation is greater than that of the first feature transformation. The output features of all displacement-aware hybrid depth conditional sub-blocks are upsampled and pixel-mapped to reconstruct an image with a resolution higher than the threshold.

[0008] Preferably, the importance of the routing unit is evaluated alternately using a regular window and a shifted window, specifically: Based on the conventional window segmentation perspective, the importance of routing units is evaluated, and a first importance score is obtained; Based on the equivalent viewpoint aligned with the shift window, a supplementary evaluation of the boundary region is performed to obtain a second importance score; By combining the first importance score and the second importance score, a final set of routing units to be selected for the second feature transformation is determined; the set of routing units is allocated according to a set ratio.

[0009] Preferably, when the routing unit set is allocated according to a set ratio, it further includes: At the hierarchical scale, the budget is allocated based on the computational resource requirements of residual blocks at different depths in the deep feature extraction network. At the block level, within the same residual block, budgets are allocated to each displacement-aware mixed depth condition sub-block, and displacement-aware priors are introduced to obtain displacement window sub-blocks that are higher than the set computational budget tendency.

[0010] Preferably, when allocating using both hierarchical and block-level scales, the method further includes setting a budget lower limit and balancing mechanism to limit the budget for all sub-blocks.

[0011] Preferably, the first feature transformation is a full basic branch, used to perform a uniform feature transformation with a computational cost below a threshold on all routing units; the second feature transformation is a sparse incremental branch, which performs a complete Transformer sub-block computation only on the selected routing units.

[0012] Preferably, it also includes training a deep feature extraction network, specifically: In the first stage, under near-intensive computation conditions, the network parameters, routing evaluation module, and budget allocation module are pre-trained. In the second stage, the proportion of routing units participating in the second feature transformation is gradually reduced, entering a sparse computing mode, and the network parameters are jointly fine-tuned.

[0013] This invention provides an image super-resolution reconstruction system, comprising: The data acquisition module is used to acquire single-frame images with a resolution lower than a threshold in a video sequence; The model processing module is used to input the single-frame image with a resolution lower than a threshold into the improved SwinIR network model to obtain a reconstructed image with a resolution higher than the threshold; wherein, the improved SwinIR network model uses SwinIR as the backbone network and introduces displacement-aware hybrid depth conditional sub-blocks in each residual block of its deep feature extraction unit; specifically including: The routing unit partitioning module is used to extract the initial features of a single frame image, generate a preliminary feature map, and divide the preliminary feature map into multiple routing units; the routing unit represents a feature map grid. The sub-block feature acquisition module is used to alternately evaluate the importance of the routing unit in each displacement-aware hybrid depth condition sub-block using a regular window and a shift window, and perform a first feature transformation on all routing units to obtain basic features. For routing units with importance higher than a threshold, a second feature transformation is performed to obtain incremental features. The initial features, basic features, and incremental features input to each routing unit are differentially fused to obtain output features. The computational cost of the second feature transformation is greater than that of the first feature transformation. The reconstruction module is used to upsample and pixel map the output features of all displacement-aware hybrid depth conditional sub-blocks to reconstruct an image with a resolution higher than a threshold.

[0014] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor performs the steps of the above-described image super-resolution reconstruction method.

[0015] The present invention provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described image super-resolution reconstruction method.

[0016] Compared with the prior art, the present invention has the following significant advantages: This invention introduces a displacement-aware hybrid depth conditional processing mechanism into a deep feature extraction network composed of multiple residual blocks. It designs displacement-aware structured routing and budget bias for the shift window mechanism, protecting the computational resources of shift window sub-blocks and ensuring a more complete cross-window information interaction link. This reduces the likelihood of block artifacts and breaks at window boundaries when restoring long edges, repetitive textures, and large-scale structures. Furthermore, at each Transformer sub-block, a base-incremental fusion structure is adopted to provide continuous, low-cost base branch feature transformations for all regions, learning detailed increments relative to the base branch only in important regions. This prevents unselected regions from being skipped directly, significantly reducing meshing artifacts caused by abrupt changes in feature map space. This results in better visual performance in terms of texture continuity and edge smoothness. The sub-block routing and two-scale budget allocation allow the model to dynamically allocate recomputational resources based on different depths, block types, and the complexity of different image regions, favoring computation in areas with rich textures, critical edges, or higher cross-window fusion requirements, thereby improving the ability to depict complex textures and subtle structures. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall framework in an example of the present invention; Figure 2 This is a schematic diagram of the overall structure of the SwinIR skeleton and the hybrid depth packaging layer in this invention; Figure 3 This is a schematic diagram of the sub-block routing and bi-branch calculation path within the residual block in this invention; Figure 4 This is a schematic diagram of the structured selection and dual-view overlay of displacement sensing in this invention; Figure 5 This is a schematic diagram of the basic branch of this invention - incremental fusion (Base-Delta Fusion); Figure 6 This is a schematic diagram of the hierarchical-block level two-scale budget allocation and displacement prior in this invention; Figure 7 This is a flowchart of an image super-resolution reconstruction method provided in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] To achieve the goal of "reducing the computational cost of SR models while maintaining reconstruction quality", existing technologies can be broadly categorized into the following typical solutions.

[0020] Static lightweighting and structural pruning schemes reduce computational cost by decreasing the number of network layers and channels, or by using lighter convolutional / attention modules. These schemes are simple to implement and offer stable inference, but their computational savings are independent of image content, representing a "one-size-fits-all" static reduction. When faced with complex textures, fine lines, or high magnification, insufficient model capacity can easily lead to loss of detail; while in smooth regions, unnecessary computational redundancy may still exist, making it difficult to achieve content-adaptive computational allocation.

[0021] Local computation schemes based on masks or irregular sparsity: By predicting spatial masks, convolution or attention computations are performed only at certain locations. Theoretically, this type of scheme can achieve content-adaptive computational allocation, but the masks often exhibit irregular shapes, resulting in highly discrete computation and storage access patterns, making it difficult to achieve stable gains on general-purpose parallel hardware. Furthermore, for Swing structures that rely on window integrity and shifted window interactions, irregular masks disrupt the complete organization of tokens within the window, making it difficult to efficiently implement window attention. Moreover, the hard decision of "compute / not compute" creates abrupt changes in space, easily introducing discontinuities in reconstruction.

[0022] Conditional computation-based expert models or hybrid depth schemes: Through routing selection, different samples or regions take different computational paths. Typical approaches include hybrid expert models (MoE) and hybrid depth models (MoD). MoD emphasizes that only some tokens take "heavy computation paths" in each layer, while the rest take "light paths or direct passes," which has the advantages of relatively regular tensor shapes and hardware friendliness. However, directly applying standard MoD to SR and SwinIR still has obvious drawbacks: On the one hand, SR output is sensitive to spatial consistency. If unselected regions are simply passed through, it is easy to cause feature breaks, manifested as block artifacts or texture fragmentation. On the other hand, the shifted window blocks of SwinIR are responsible for cross-window information fusion. If uniform budget or sparsification that does not distinguish block types is still used on these blocks, it will weaken cross-window interaction and amplify boundary errors, further inducing cross-window artifacts.

[0023] In summary, existing solutions either lack "structured sparse adaptation for the Swin shift window mechanism" or fail to guarantee the spatial continuity and cross-window consistency required by SR under sparse conditions. Therefore, a new dynamic sparse computing solution that can combine the structural characteristics of SwinIR while taking into account reconstruction continuity and cross-window information flow protection is still needed.

[0024] This invention addresses the challenge of single-image super-resolution reconstruction. It tackles the difficulties of balancing reconstruction accuracy and computational cost in existing SwinIR-type window Transformer models, as well as the problems of cross-window artifacts and feature discontinuities introduced when directly transferring general sparse computation mechanisms to shifted window structures. To solve this problem, a hybrid depth computing method based on "displacement awareness, block-by-block routing, and base-incremental fusion" is proposed, matching the Swin shifted window structure. The core technical problems this invention aims to solve include at least the following aspects:

[0025] 0. This invention addresses the problem of "sparseness disrupting cross-window information exchange in shifted windows." The Swin architecture relies on alternating regular window attention and shifted window attention to achieve cross-window information exchange. In SR tasks, cross-window interaction is particularly crucial for recovering long edges, repetitive textures, and cross-region consistency. If sparsification is applied indiscriminately within the shifted window module, the cross-window information flow can be weakened or even broken, resulting in significant cross-window block artifacts or boundary discontinuities in the output image. One objective of this invention is to explicitly protect the cross-window fusion capability undertaken by the shifted window while introducing conditional computation, ensuring that the model maintains continuity and consistency during the recovery of complex textures and long-range structures.

[0026] 0. This invention addresses the issue of "feature map spatial fragmentation caused by direct access to unselected regions." Traditional token-selective sparse computation typically skips the recomputation path for unselected tokens or performs only identity mapping. While this strategy has relatively controllable impact in NLP tasks, it leads to local information loss and spatial discontinuity in SR tasks: unselected regions lack necessary feature transformations and detail compensation, resulting in texture holes, detail collapse, or mesh artifacts after multiple layers are stacked. A second objective of this invention is to provide continuous, low-cost "basic branch feature transformations" for all regions, and to ensure that recomputation only learns the "detail increments" relative to the basic branches, thereby guaranteeing spatial continuity and reconstruction stability.

[0027] 0. This invention addresses the problem of "different depths and block types having different computational demands yet being constrained by a unified budget." SwinIR is composed of multiple stacked residual blocks, with each depth layer having varying degrees of dependence on high-frequency details, structural information, and context. Within the same residual block, there are alternating sub-blocks of two types: regular windows and shifted windows, whose requirements for cross-window interaction differ significantly. Using a uniform sparsity ratio or shared routing rules would either lead to insufficient processing of edge textures in shallow layers or insufficient budget allocation for critical shifted window sub-blocks, ultimately resulting in unrealistic textures or cross-window artifacts. The third objective of this invention is to establish a budget allocation mechanism oriented towards network depth and block type, enabling adaptive allocation of computational budgets at both the "layer-block" and "level" scales, and structurally giving shifted window sub-blocks a computational bias more aligned with their functional positioning.

[0028] Fourth, this invention addresses the problem of "instability in budget learning caused by discrete selection." Sparse routing typically involves discrete operations such as Top-K selection. If the early training phase is in a state of near-all selection or random selection, it will be difficult for the budget allocation parameters to obtain effective gradients, leading to instability in the training process, and even routing collapse (some blocks do not receive budget for a long time) or oscillation (frequent budget jumps). The fourth objective of this invention is to provide a differentiable budget learning auxiliary mechanism and a progressive training strategy, enabling stable gradient signals for budget allocation in the early stages of training, and maintaining model convergence and reconstruction quality during the gradual transition from intensive to sparse computation.

[0029] With the above objectives, this invention aims to introduce conditional computation adapted to the shift window structure without changing the overall framework of the SwinIR backbone. This allows the model to concentrate computational power more precisely on key texture and edge regions, reduce cross-window artifacts through displacement-aware structured selection, and maintain spatial continuity through base-incremental fusion, thereby improving the visual quality and stability of SR reconstruction.

[0030] This invention proposes an image super-resolution reconstruction method. Based on the SwinIR network skeleton, this method introduces a conditional computation approach with mixed depths during its deep feature extraction stage. It also designs "displacement-aware structured routing" and "block-by-block budget allocation" to address the shift window mechanism of the Swin structure, and proposes a dual-branch computation path of "base-incremental fusion" to ensure feature continuity. In its implementation, this invention can be divided into two main parts: model architecture and training strategy. The model architecture emphasizes routing unit partitioning, block-by-block routing, displacement-aware budgeting, and base-incremental fusion; the training strategy emphasizes differentiable budget proxying and progressively sparse training from dense to sparse. Figure 1 This is a schematic diagram of the overall process of the method, including the following steps:

[0031] Step 1: Obtain single-frame images with resolutions below a threshold from the video sequence.

[0032] Step 2: Input the single-frame image with a resolution below the threshold into the improved SwinIR network model to obtain a reconstructed image with a resolution above the threshold; wherein, the improved SwinIR network model uses SwinIR as the backbone network and introduces displacement-aware hybrid depth conditional sub-blocks in each residual block of its deep feature extraction unit; specifically including: Step 21: Extract the initial features of a single frame image, generate a preliminary feature map, and divide the preliminary feature map into multiple routing units; the routing unit represents a feature map grid.

[0033] Input Alignment and Routing Unit Partitioning: To ensure that conditional computation has spatially regular computational units and avoid boundary fragmentation, this invention performs dynamic alignment processing on input features before entering deep feature extraction. Specifically, the feature map is aligned in the height and width directions according to the "routing unit" size, and necessary padding is performed using methods such as boundary reflection. This ensures that subsequent window partitioning and routing unit partitioning strictly cover the entire feature map without any missing units. Unlike alignment based solely on the attention window size, this invention allows routing units to be in a fixed multiple relationship with the attention window. This maintains the window attention implementation method while providing units for coarser-grained and more stable routing decisions. This design helps reduce selection breaks at window boundaries and makes sparse computation easier to organize in the form of regular tensors in engineering implementation.

[0034] Overall Network-Level Insertion Approach: This invention, without altering the SwinIR input / output interface or the shallow feature extraction and reconstruction modules, inserts a hybrid depth conditional computation mechanism as a "wrapping layer" into the deep feature extraction stage. Specifically, residual Swin Transformer blocks are used as the basic stacking units, and modules capable of performing routing decisions and bi-branch computation are introduced outside each residual block. This ensures that the overall network maintains end-to-end residual learning and feature stacking. This network-level insertion guarantees compatibility with existing SwinIR engineering implementations while ensuring that conditional computation primarily operates on the most computationally intensive deep feature extraction part, providing greater room for improvement in reducing reconstruction artifacts and enhancing texture representation. Figure 2 This is a schematic diagram of the overall structure of the SwinIR skeleton and the mixed depth packaging layer.

[0035] Step 22: In each displacement-aware hybrid depth conditional sub-block, the importance of the routing unit is evaluated alternately using a regular window and a shift window. A first feature transformation is performed on all routing units to obtain basic features. A second feature transformation is performed on routing units with importance higher than a threshold to obtain incremental features. The initial features, basic features, and incremental features input to each routing unit are differentially fused to obtain output features. The computational cost of the second feature transformation is greater than that of the first feature transformation.

[0036] Blockwise Routing within Residual Blocks: To avoid coarse-grained mismatch caused by a "one-size-fits-all" approach to residual blocks, this invention decentralizes routing granularity to each Swin Transformer sub-block within the residual block. That is, within the same residual block, routing decisions and condition calculations are performed separately for sub-blocks at different depths and in different shift states. Each sub-block is configured with an independent routing evaluation unit to score the importance of routing units with input features for the current sub-block. Within this sub-block, only some important routing units undergo full path recalculation, while the remaining routing units only undergo low-cost basic branches. Due to functional and semantic differences between sub-blocks, blockwise routing can learn differentiated selection patterns and avoid insufficient representation caused by shared routers, thereby maintaining or improving the stability and detail integrity of SR reconstruction while reducing computation. Figure 3 This is a schematic diagram of the sub-block routing and bi-branch calculation path within the residual block.

[0037] Displacement-Aware Structured Routing: Addressing the crucial role of shifted window sub-blocks in cross-window information interaction, this invention proposes a displacement-aware structured routing strategy. During routing decisions, not only is the importance of routing units evaluated from the conventional window partitioning perspective to obtain a first importance score, but an equivalent perspective aligned with the shifted window is also introduced to supplement the evaluation of boundary regions, obtaining a second importance score. Combining the first and second importance scores, the set of routing units is allocated proportionally to the set of routing units "covered by the shifted perspective," determining the final set of routing units selected for recalculation, thereby ensuring that critical textures and long edges near cross-window boundaries are not missed due to selection bias. The mathematical expression of this invention is as follows:

[0038] Importance assessment from two perspectives: , .

[0039] Displacement perspective budget breakdown: , .

[0040] Dual-perspective Topk selection and merging: , , .

[0041] Structured group selection (group score definition): .

[0042] in, This represents the first importance score of the i-th routing unit from a conventional perspective. This represents the second importance score of the i-th routing unit from the perspective of shift equivalence. For the scoring function, Let be the feature representation of the i-th routing unit from the perspective of a regular window partitioning. This represents the feature representation of the same routing unit from an equivalent viewpoint aligned with the shift window. The number of units selected from the shift perspective. The preset total number of routing units, For hyperparameters, The number of units selected from a conventional perspective. This indicates selecting the highest score from the importance score vector s. Each routing unit returns its index or set. The score for the j-th group. For the j-th structured group, This represents an aggregate function.

[0043] Meanwhile, to avoid spatial abrupt changes caused by irregular scattered selection, this invention adopts structured grouping selection for routing units, making the selected set spatially distributed in a relatively continuous block shape, which better meets the spatial continuity requirements of SR. By combining displacement sensing and structured selection, this invention can significantly reduce cross-window block artifacts and improve cross-window consistency. Figure 4 A schematic diagram of structured selection and dual-view overlay for displacement sensing.

[0044] A dual-branch computation path for basic-incremental fusion: To address the feature discontinuity problem caused by sparse conditional computation, this invention designs a dual-branch structure of "full basic path + sparse incremental path" within each sub-block. The basic branch performs a unified basic feature transformation on all routing units using low-computation modules, providing continuous and stable low-frequency and structural information; the incremental path performs complete Swin Transformer sub-block computation only on the selected routing units, used to generate higher-frequency and more refined detail representations. For example, the full basic branch: ;Sparse incremental branch Basic-incremental differential fusion output: ;where the mask function Gated scaling factor , Input features for the i-th routing unit, The output of the basic branch, For the output of the sparse incremental branch, For the final fusion output, For learnable scalar parameters, For activation function, Let be the gating parameter corresponding to the i-th routing unit.

[0045] The final output is fused using the "difference between the incremental and the basic branches," allowing the recalculation path to focus on supplementing details rather than reconstructing all features. This mechanism ensures that unselected regions also have reasonable feature representation, significantly reducing spatial fragmentation and meshing artifacts caused by skipped calculations. Figure 5 Diagram of Base-Delta Fusion.

[0046] To ensure that sub-blocks of different depths and types receive computing resources matching their functions, this invention establishes a two-scale budget allocation mechanism: "hierarchical" and "block-level." For example... Figure 6 A schematic diagram of budget allocation and displacement priors at both the hierarchical and block levels.

[0047] At the hierarchical scale, the budget is allocated in a learnable manner based on the network depth location, enabling the deep texture reconstruction stage to obtain more sufficient computational resources for detail. The mathematical expression is as follows: ; in, To indicate the first The computational budget allocated to each layer. For learnable parameter vectors, As a smoothing factor, This represents the total number of layers in the network.

[0048] At the block level, budgets are further allocated to each sub-block within the same residual block, and displacement-aware priors are introduced to give the shifted window sub-blocks a higher budget bias, structurally ensuring the integrity of the cross-window information interaction link. The mathematical expression is as follows:

[0049] ; ; Simultaneously, a budget floor and balancing mechanism are set to prevent training starvation caused by excessively low budgets in individual sub-blocks over a long period, ensuring that routing decisions are stable and learnable across the entire network. The mathematical expression is as follows: ; in, This represents the block-level allocation result. For the balance coefficient, This represents the total number of sub-blocks under the same allocation scale.

[0050] Step 23: Upsample and pixel map the output features of all displacement-aware hybrid depth conditional sub-blocks to reconstruct an image with a resolution higher than the threshold.

[0051] Training strategy: Differentiable Budget Proxy and Stable Learning: Since route selection involves discrete Top-K operations, direct end-to-end training may encounter problems such as difficulty in updating budget parameters and unstable route decisions in the early stages of training. To address this, this invention introduces a differentiable budget proxy mechanism in the early stages of training: while maintaining controllable network execution paths, a budget-related continuous scaling is applied to the output of basic branches, causing the loss function to form a propagable gradient signal with respect to the budget allocation parameters. This allows the importance distribution of different depths and sub-blocks to be learned without relying on discrete selection gradients. This mechanism effectively avoids route collapse and oscillations, improves training stability, and lays the foundation for subsequent entry into a true sparse mode.

[0052] Progressive Sparsity Training from Dense to Sparse: To gradually introduce conditional computation while ensuring reconstruction quality, this invention employs a progressive sparsity training process from dense to sparse. The core idea is to first complete stable pre-training of routers, budget allocation, and basic branches under near-dense computational conditions, enabling the model to learn how to assess importance and provide continuous basic branches. Subsequently, the proportion of routing units participating in recomputation paths is gradually reduced, entering true sparse operation, and fine-tuned jointly with the backbone network parameters, allowing the basic and incremental branches to be co-optimized under sparse conditions. This progressive training reduces performance fluctuations in the early stages of sparse training, minimizes cross-window artifacts and texture breaks, and makes the method more stable in terms of visual quality.

[0053] The conditional computation of this invention uses routing units as rule tensor units for selection, combined with dynamic alignment and structured group selection, which is beneficial for batch implementation on general-purpose GPUs / accelerators; the method is inserted in the form of a wrapper layer, which has good portability and can be extended to similar window-based Transformer-based image restoration tasks.

[0054] To further verify the beneficial effects of the present invention, in the same hardware and software testing environment, the image super-resolution reconstruction method proposed in this invention was compared with the baseline model (standard Swin SwinIR structure features). Under these conditions, the method effectively overcomes the bottlenecks of the original model's large computational load and slow inference. Specific results are as follows: 1. Computational cost reduction: Compared with the baseline model, the method of this invention significantly reduces the total computational cost by 37.92% while ensuring reconstruction quality. This indicates that the invention successfully eliminates redundant computation in smooth areas of the image through block-by-block routing and two-scale budget allocation mechanism.

[0055] 2. Improved inference speed: Thanks to the dynamic alignment design of the rule tensor unit and the basic-incremental dual-branch architecture, the reduction in computational load is successfully translated into actual inference acceleration. The method of this invention achieves an inference speedup of 1.8 times.

[0056] This invention proposes an image super-resolution reconstruction system, comprising: The data acquisition module is used to acquire single-frame images with resolutions below a threshold from the video sequence; the model processing module is used to input the single-frame images with resolutions below the threshold into the improved SwinIR network model to obtain reconstructed images with resolutions above the threshold; wherein the improved SwinIR network model uses SwinIR as the backbone network and introduces displacement-aware hybrid depth conditional sub-blocks in each residual block of its deep feature extraction unit; specifically including: The routing unit partitioning module is used to extract initial features from a single frame image, generate a preliminary feature map, and divide the preliminary feature map into multiple routing units; the routing unit represents a feature map grid; the sub-block feature acquisition module is used to evaluate the importance of the routing unit in each displacement-aware mixed depth condition sub-block alternately using a regular window and a shift window, and perform a first feature transformation on all routing units to obtain basic features, and perform a second feature transformation on routing units with importance higher than a threshold to obtain incremental features; the initial features, basic features, and incremental features input to each routing unit are differentially fused to obtain output features; wherein, the computational cost of the second feature transformation is greater than that of the first feature transformation; the reconstruction module is used to upsample and pixel map the output features of all displacement-aware mixed depth condition sub-blocks to reconstruct an image with a resolution higher than a threshold.

[0057] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor performs the steps of an image super-resolution reconstruction method.

[0058] According to the disclosed embodiments, the computer device can communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth communication, etc.) or with any device that enables the computing device to communicate with one or more other computing devices (e.g., router, demodulator, etc.).

[0059] The present invention also provides a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of an image super-resolution reconstruction method.

[0060] According to the disclosed embodiments, the storage medium can be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0061] The above description, in conjunction with specific preferred embodiments, provides a more detailed explanation of the present invention. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention.

Claims

1. An image super-resolution reconstruction method, characterized in that, Including the following steps: Retrieve single-frame images with resolutions below a threshold from a video sequence; The single-frame image with a resolution below a threshold is input into the improved SwinIR network model to obtain a reconstructed image with a resolution above the threshold; wherein, the improved SwinIR network model uses SwinIR as the backbone network and introduces displacement-aware hybrid depth conditional sub-blocks in each residual block of its deep feature extraction unit; specifically including: Initial features are extracted from a single frame image to generate a preliminary feature map, which is then divided into multiple routing units; each routing unit represents a feature map grid. In each displacement-aware hybrid depth conditional sub-block, the importance of the routing unit is evaluated alternately using a regular window and a shift window. A first feature transformation is performed on all routing units to obtain basic features, and a second feature transformation is performed on routing units with importance higher than a threshold to obtain incremental features. The initial features, basic features, and incremental features input to each routing unit are differentially fused to obtain output features. The computational cost of the second feature transformation is greater than that of the first feature transformation. The output features of all displacement-aware hybrid depth conditional sub-blocks are upsampled and pixel-mapped to reconstruct an image with a resolution higher than the threshold.

2. The image super-resolution reconstruction method as described in claim 1, characterized in that, The importance of the routing unit is evaluated alternately using a regular window and a shifted window, specifically: Based on the conventional window segmentation perspective, the importance of routing units is evaluated, and a first importance score is obtained; Based on the equivalent viewpoint aligned with the shift window, a supplementary evaluation of the boundary region is performed to obtain a second importance score; By combining the first importance score and the second importance score, a final set of routing units selected to perform the second feature transformation is determined; the set of routing units is allocated according to a set ratio.

3. The image super-resolution reconstruction method as described in claim 2, characterized in that, When the routing unit set is allocated according to a set ratio, it also includes: At the hierarchical scale, the budget is allocated based on the computational resource requirements of residual blocks at different depths in the deep feature extraction network. At the block level, within the same residual block, budgets are allocated to each displacement-aware mixed depth condition sub-block, and displacement-aware priors are introduced to obtain displacement window sub-blocks that are higher than the set computational budget tendency.

4. The image super-resolution reconstruction method as described in claim 3, characterized in that, When allocating at both hierarchical and block levels, the following measures are also included: setting a budget floor and a balancing mechanism to limit the budget for all sub-blocks.

5. The image super-resolution reconstruction method as described in claim 1, characterized in that, The process of determining the final set of routing units selected to perform the second feature transformation further includes: using structured grouping to select the routing units to obtain a set of routing units that are spatially distributed in a relatively continuous block shape.

6. The image super-resolution reconstruction method as described in claim 1, characterized in that, The first feature transformation is a full basic branch, which is used to perform a uniform feature transformation with a computational cost of less than a threshold on all routing units; the second feature transformation is a sparse incremental branch, which performs a complete Transformer sub-block computation only on the selected routing units.

7. The image super-resolution reconstruction method as described in claim 1, characterized in that, This also includes training a deep feature extraction network, specifically: In the first stage, under near-intensive computation conditions, the network parameters, routing evaluation module, and budget allocation module are pre-trained. In the second stage, the proportion of routing units participating in the recalculation path is gradually reduced, entering a sparse computing mode, and network parameters are jointly fine-tuned.

8. An image super-resolution reconstruction system, characterized in that, include: The data acquisition module is used to acquire single-frame images with a resolution lower than a threshold in a video sequence; The model processing module is used to input the single-frame image with a resolution lower than a threshold into the improved SwinIR network model to obtain a reconstructed image with a resolution higher than the threshold; wherein, the improved SwinIR network model uses SwinIR as the backbone network and introduces displacement-aware hybrid depth conditional sub-blocks in each residual block of its deep feature extraction unit; specifically including: The routing unit partitioning module is used to extract the initial features of a single frame image, generate a preliminary feature map, and divide the preliminary feature map into multiple routing units; the routing unit represents a feature map grid. The sub-block feature acquisition module is used to alternately evaluate the importance of the routing unit in each displacement-aware hybrid depth condition sub-block using a regular window and a shift window, and perform a first feature transformation on all routing units to obtain basic features. For routing units with importance higher than a threshold, a second feature transformation is performed to obtain incremental features. The initial features, basic features, and incremental features input to each routing unit are differentially fused to obtain output features. The computational cost of the second feature transformation is greater than that of the first feature transformation. The reconstruction module is used to upsample and pixel map the output features of all displacement-aware hybrid depth conditional sub-blocks to reconstruct an image with a resolution higher than a threshold.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores a program that, when executed by the processor, causes the processor to perform the steps of an image super-resolution reconstruction method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the image super-resolution reconstruction method according to any one of claims 1 to 7.

Images