Mobile image super-resolution method based on prompt calibration and multi-scale cascade

Abstract

The present application relates to super-resolution image reconstruction technology, and discloses a mobile terminal image super-resolution method based on prompt calibration and multi-scale cascade, which comprises the following steps: acquiring a low-resolution image, using a blueprint separation convolution to perform shallow feature extraction on the low-resolution image to obtain an initial feature map; performing down-sampling on the initial feature map, and inputting the down-sampled feature map into a feature extraction network composed of a plurality of RMVP modules in cascade to perform deep feature extraction; performing up-sampling on the deep features output by the feature extraction network; performing feature fusion on the up-sampled features and the initial feature map through a jump connection; performing image reconstruction on the fused features to output a high-resolution image. The mobile terminal image super-resolution method based on prompt calibration and multi-scale cascade can achieve high image super-resolution performance while maintaining a lightweight structure to meet the real-time inference requirements of mobile terminals.

Description

Technical Field

[0001] This invention relates to super-resolution image reconstruction technology, and more particularly to a mobile image super-resolution method based on cue calibration and multi-scale cascading. Background Technology

[0002] Image super-resolution reconstruction (SR) is an important low-level computer vision task that aims to recover the corresponding high-resolution (HR) image from a degraded low-resolution (LR) image. High-resolution images have high pixel density, providing richer details that play a crucial role in subsequent visual tasks. Current image super-resolution reconstruction methods generally suffer from large network parameter counts and high computational complexity. In particular, models based on deep network architectures often suffer from slow inference speeds, making it difficult to meet the real-time super-resolution processing requirements of resource-constrained mobile devices.

[0003] Existing techniques, such as the introduction of multiple Swing TRansfoRmeR blocks to extract shallow and deep features, improve the performance of the model in image super-resolution tasks. Local window N-Gram interaction is introduced into visual TRansfoRmeR, expanding the perceptual range of degraded pixels through sliding window attention.

[0004] Existing methods enhance key features and long-range dependencies through attention mechanisms, effectively capturing global context to improve image super-resolution performance. However, they face bottlenecks such as large number of parameters and high computational complexity, which restrict model efficiency and inference speed. Summary of the Invention

[0005] This invention addresses the bottlenecks in existing super-resolution image processing technologies, such as large parameter counts and high computational complexity, which restrict model efficiency and inference speed. It provides a mobile image super-resolution method based on cue calibration and multi-scale cascading.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution.

[0007] A mobile image super-resolution method based on cue calibration and multi-scale cascading includes the following methods:

[0008] Acquire low-resolution images by inputting from a mobile device;

[0009] The initial feature map is extracted by using blueprint-separated convolution to perform shallow feature extraction on the low-resolution image.

[0010] Deep feature extraction involves downsampling the initial feature map and then inputting the downsampled feature map into a feature extraction network composed of multiple cascaded RMVP modules for deep feature extraction.

[0011] Grouped multi-scale feature aggregation: Grouped multi-scale feature aggregation processes deep features by channel grouping and parallel cascading;

[0012] Feature fusion involves upsampling the deep features output by the aggregated grouped multi-scale feature extraction network; then fusing the upsampled features with the initial feature map through skip connections.

[0013] The high-resolution image is output by reconstructing the fused features.

[0014] As a preferred embodiment, each RMVP module in the feature extraction network composed of multiple cascaded RMVP modules includes a reparameterizable multi-branch fusion module and a visually cued dynamic modulation module.

[0015] Preferably, the reparameterizable multi-branch fusion module consists of multiple parallel branches, and its implementation includes:

[0016] Receive input feature map; multi-branch feature extraction, which consists of multiple parallel heterogeneous convolutional branches during the training phase, is used to extract multi-scale features from the input feature map;

[0017] The first fusion feature is generated by fusing the multi-scale features output by the multi-branch feature extraction unit to generate the first fusion feature.

[0018] The second fusion feature is generated by inputting the first fusion feature into a convolutional layer with a kernel size of conv1×1 for channel dimensionality reduction and nonlinear transformation.

[0019] Preferably, the training phase consists of multiple parallel heterogeneous convolutional branches used to extract multi-scale features from the input feature map; the heterogeneous convolutional branches include a first branch, a second branch, a third branch, a fourth branch, a fifth branch, and a sixth branch.

[0020] The first branch is a conv3×3 convolution kernel branch, the second branch is a conv3×1 convolution kernel branch, the third branch is a conv1×3 convolution kernel branch, the fourth branch is a conv1×1 convolution kernel branch, the fifth branch is an element-wise multiplication operation between the learnable scale factor matrix and the input feature map, and the sixth branch is an element-wise addition operation between the learnable bias factor matrix and the input feature map.

[0021] Preferably, the visual cue-guided dynamic modulation module includes a visual cue generation unit for generating one or more learnable visual cue tokens; a feature interaction unit for calculating the correlation between the input feature map and the visual cue tokens to generate dynamic modulation coefficients; a modulation unit for adaptively enhancing or suppressing the input feature map using the dynamic modulation coefficients to generate modulated features; and outputting the modulated features.

[0022] As a preferred option: grouped multi-scale feature aggregation for deep features is performed by channel grouping and parallel cascading, including: receiving the input feature map and dividing the input feature map into N non-overlapping feature subgroups in the channel dimension, where N is an integer greater than 1;

[0023] Parallel multi-scale feature extraction is performed on the N feature subgroups after division. At least one feature subgroup is subjected to a depthwise convolution operation with K concatenations, where K is an integer greater than 1. The N feature subgroups after parallel multi-scale feature extraction are concatenated along the channel dimension to generate aggregated features. The aggregated features are then output.

[0024] The present invention has significant technical effects due to the adoption of the above technical solutions.

[0025] This invention is based on a mobile image super-resolution method using cue calibration and multi-scale cascading. It achieves high image super-resolution performance while maintaining an extremely lightweight structure to meet the real-time inference requirements of mobile devices.

[0026] This invention extracts multi-scale features through parallel heterogeneous convolution branches, and then introduces learnable scale factor and bias factor branches for dynamic feature calibration. By enhancing the representation ability through multi-branch structure during training, it can be equivalently reparameterized into a single convolution during inference, thus achieving a significant improvement in computational efficiency.

[0027] This invention utilizes a visually cued, dynamically modulated module to achieve adaptive enhancement and suppression of features through minimal learnable visual cues and an adaptive centralized attention mechanism.

[0028] The grouped multi-scale feature aggregation module of the present invention constructs a multi-scale receptive field by grouping features by channel and processing them in parallel cascades, based on stacking deep convolutions with few parameters, thereby reducing the amount of computation and improving the feature utilization rate. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the process of this invention.

[0030] Figure 2 This is a schematic diagram of the RMVP process of this invention.

[0031] Figure 3This is a schematic diagram of the multi-scale feature aggregation process of the present invention.

[0032] Among them: RMBF (RepaRameteRizable Multi-branch with Visual prompt) is a reparameterizable multi-branch fusion module; VPDM (Visual Prompt guided Dynamic Modulation module) is a visually prompted dynamic modulation module; GMFA (Grouped Multi-scale Feature Aggregation module) is a grouped multi-scale feature aggregation module; RMVP (RepaRameteRizable Multi-branch with Visual Prompt) is a reparameterizable multi-branch module integrating visual prompts; DIV2K (Diverse 2K Resolution high qualityimages) is an image super-resolution dataset with rich and diverse scene content; LSDIR (Large Scale Dataset for Image Restoration) is a large-scale image super-resolution dataset created to make up for the small size and lack of diversity of previous image restoration datasets; REDS (Realistic and Diverse Scenes) is a dataset for video super-resolution tasks; UDM10 is a dataset for video super-resolution tasks; and DWconv (Depthwise convolution) is a depthwise convolution. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0034] Example 1

[0035] A mobile image super-resolution method based on cue calibration and multi-scale cascading includes the following methods:

[0036] Acquire low-resolution images by inputting from a mobile device;

[0037] The initial feature map is extracted by using blueprint-separated convolution to perform shallow feature extraction on the low-resolution image.

[0038] Deep feature extraction involves downsampling the initial feature map and then inputting the downsampled feature map into a feature extraction network composed of multiple cascaded RMVP modules for deep feature extraction.

[0039] Feature fusion involves upsampling the deep features output by the feature extraction network; then fusing the upsampled features with the initial feature map through skip connections.

[0040] The high-resolution image is output by reconstructing the fused features.

[0041] Each RMVP module in the feature extraction network, which is composed of multiple cascaded RMVP modules, includes a reparameterizable multi-branch fusion module and a visually cued dynamic modulation module.

[0042] The reparameterizable multi-branch fusion module consists of multiple parallel branches. The implementation of the reparameterizable multi-branch fusion module includes:

[0043] Receive input feature map; multi-branch feature extraction, which consists of multiple parallel heterogeneous convolutional branches during the training phase, is used to extract multi-scale features from the input feature map;

[0044] The first fusion feature is generated by fusing the multi-scale features output by the multi-branch feature extraction unit to generate the first fusion feature.

[0045] The second fusion feature is generated by inputting the first fusion feature into a convolutional layer with a kernel size of 1×1 for channel dimensionality reduction and nonlinear transformation.

[0046] exist Figure 2 In the training phase, the system consists of multiple parallel heterogeneous convolutional branches used to extract multi-scale features from the input feature map. These heterogeneous convolutional branches include a first branch, a second branch, a third branch, a fourth branch, a fifth branch, and a sixth branch. The first branch is a conv3×3 convolutional kernel branch, the second branch is a conv3×1 convolutional kernel branch, the third branch is a conv1×3 convolutional kernel branch, the fourth branch is a conv1×1 convolutional kernel branch, the fifth branch performs element-wise multiplication of the learnable scale factor matrix with the input feature map, and the sixth branch performs element-wise addition of the learnable bias factor matrix with the input feature map. In the diagram, the learnable scale factor is ω, and the learnable bias factor is b. The first branch is a conv3×3 convolutional kernel branch with a learnable scaling factor of ω1 and a learnable bias factor of b1; the second branch is a conv3×1 convolutional kernel branch with a learnable scaling factor of ω2 and a learnable bias factor of b2; the third branch is a conv1×3 convolutional kernel branch with a learnable scaling factor of ω3 and a learnable bias factor of b3; the fourth branch is a conv1×1 convolutional kernel branch with a learnable scaling factor of ω4 and a learnable bias factor of b4.

[0047] Example 2

[0048] A mobile image super-resolution method based on cue calibration and multi-scale cascading. Figure 1 The methods include:

[0049] Acquire low-resolution images by inputting from a mobile device;

[0050] The initial feature map is extracted by using blueprint-separated convolution to perform shallow feature extraction on the low-resolution image.

[0051] Deep feature extraction involves downsampling the initial feature map and then inputting the downsampled feature map into a feature extraction network composed of multiple cascaded RMVP modules for deep feature extraction.

[0052] Feature fusion involves upsampling the deep features output by the feature extraction network; then fusing the upsampled features with the initial feature map through skip connections.

[0053] Grouped multi-scale feature aggregation: Grouped multi-scale feature aggregation processes deep features by channel grouping and parallel cascading;

[0054] The high-resolution image output is generated by reconstructing the image from the features after grouping and aggregating multi-scale features.

[0055] In a feature extraction network composed of multiple cascaded RMVP modules, each RMVP module includes reparameterizable multi-branch fusion and visually cued dynamic modulation.

[0056] The reparameterizable multi-branch fusion module consists of multiple parallel branches. The implementation of the reparameterizable multi-branch fusion module includes:

[0057] Receive input feature map; multi-branch feature extraction, which consists of multiple parallel heterogeneous convolutional branches during the training phase, is used to extract multi-scale features from the input feature map;

[0058] The first fusion feature is generated by fusing the multi-scale features output by the multi-branch feature extraction unit to generate the first fusion feature.

[0059] The second fusion feature is generated by inputting the first fusion feature into a convolutional layer with a kernel size of 1×1 for channel dimensionality reduction and nonlinear transformation.

[0060] The training phase consists of multiple parallel heterogeneous convolutional branches used to extract multi-scale features from the input feature map; the heterogeneous convolutional branches include the first branch, the second branch, the third branch, the fourth branch, the fifth branch, and the sixth branch.

[0061] The first branch is a conv3×3 convolutional kernel branch, the second branch is a conv3×1 convolutional kernel branch, the third branch is a conv1×3 convolutional kernel branch, the fourth branch is a conv1×1 convolutional kernel branch, the fifth branch is an element-wise multiplication operation between the learnable scaling factor matrix and the input feature map, and the sixth branch is an element-wise addition operation between the learnable bias factor matrix and the input feature map. The first branch, the conv3×3 convolutional kernel branch, has a learnable scaling factor of ω1 and a learnable bias factor of b1; the second branch, the conv3×1 convolutional kernel branch, has a learnable scaling factor of ω2 and a learnable bias factor of b2; the third branch, the conv1×3 convolutional kernel branch, has a learnable scaling factor of ω3 and a learnable bias factor of b3; the fourth branch, the conv1×1 convolutional kernel branch, has a learnable scaling factor of ω4 and a learnable bias factor of b4.

[0062] Figure 2 In the process, the dynamic modulation module guided by visual cues includes a visual cue generation unit, which generates one or more learnable visual cue tokens; it takes learnable visual cues as input and obtains weights through a shared router to perform adaptive weighted visual cues; another branch performs attention centering on the features through Sigmoid and generates dynamically modulated corrected features through dot product.

[0063] The feature interaction unit is used to calculate the correlation between the input feature map and the visual cue token and generate dynamic modulation coefficients; the modulation unit uses the dynamic modulation coefficients to adaptively enhance or suppress the input feature map and generate modulated features; and outputs the modulated features.

[0064] Grouped multi-scale feature aggregation for deep features involves channel-based grouping and parallel concatenation, including: receiving an input feature map; dividing the input feature map into N non-overlapping feature subgroups along the channel dimension, where N is an integer greater than 1; performing parallel multi-scale feature extraction on the N subgroups, wherein at least one subgroup undergoes a depthwise convolution operation consisting of K concatenations, where K is an integer greater than 1; concatenating the N subgroups after parallel multi-scale feature extraction along the channel dimension to generate aggregated features; and outputting the aggregated features.

[0065] Example 3

[0066] Based on the above embodiments, this embodiment achieves high image super-resolution performance while maintaining a lightweight structure to meet the real-time inference requirements of mobile devices.

[0067] An image super-resolution dataset for security scenarios is constructed to train the designed super-resolution network. The training process employs multi-stage progressive resolution training, and its specific implementation is as follows:

[0068] The first step involves pre-training the network using publicly available datasets DIV2K, LSDIR, REDS, and UDM10. The network then utilizes a blind super-resolution pipeline proposed by Real-ESRGAN (Real-Enhanced Super-Resolution Generative Adversarial Network), an augmented super-resolution generative adversarial network designed for the real world, to generate corresponding low-resolution images. The input low-resolution image size is set to 64×64. The Adam optimizer is used with parameters β1 = 0.9 and β2 = 0.999, an initial learning rate of 5e-4, and a cosine learning rate decay strategy. The number of iterations is 100k.

[0069] In the second stage, the low-resolution image size was set to 128×128, the batch size to 32, the learning rate to 5e-4, a cosine learning rate decay strategy was adopted, and the number of iterations was 100k.

[0070] The third stage introduces an additional security scenario dataset, setting the low-resolution image size to 256×256, the batch size to 32, the learning rate to 2e-4, and adopting a cosine learning rate decay strategy with 200k iterations.

[0071] In the fourth stage, the original weights are frozen, and new optimizable weights are introduced and optimized according to the following formula: Wfinal = FRozen(WpRe) + WleaRn, where Wfinal is the final weight; FRozen(WpRe) is the frozen pre-trained weight; WleaRn is the learnable weight; the batch size is 10, the learning rate is set to 1e-4, a cosine learning rate decay strategy is adopted, and the number of iterations is 50k.

[0072] The training loss in the first to third stages is the L1 loss. The training loss in the fourth stage is the sum of the weighted L1 and L2 losses, where L1 is the mean absolute error and L2 is the mean squared error, and the weight coefficients of L1 and L2 losses are 0.7 and 0.3, respectively.

[0073] The designed super-resolution network model first standardizes the input image X0∈ R^(3*H*W), and then concatenates the image X0 multiple times along the channel dimension to obtain the input X = Concat([X0, X0, ..., X0]) ∈ R^(3·N*H*W), where N, H and W are the number, length and width of the concatenated images, respectively, and R is the set of real numbers.

[0074] The initial feature F is extracted by blueprinting the convolution with a convolution kernel of conv3×3. 1，At this point, the feature map size of F1 is D*H*W. Subsequent calculations will introduce a large amount of computation. To address this, max pooling is introduced to reduce the feature map size to D*H / 2*W / 2 to reduce computation. This will help the model extract global feature information but will also lose some local details. Therefore, the lost information is recovered by adding the F1 feature to the subsequently upsampled features. This can significantly reduce computation while ensuring model performance.

[0075] The feature extraction module, named RMVP, is constructed using the novel reparameterizable multi-branch fusion module RMBF and the visually cued dynamic modulation module VPDM as its core. Specifically, multiple RMVPs are stacked to perform feature extraction from shallow to deep levels.

[0076] The RMBF module consists of multiple parallel branches. Four branches with convolutional kernels of conv3×3, conv3×1, conv1×3, and conv1×1 extract multi-scale features. Four sets of learnable scale and bias coefficients are introduced to improve the model's non-linear fitting ability and adaptive feature extraction capability using a very small number of parameters. Then, the features from these eight branches are concatenated along the channel dimension and dimensionality reduction is performed using conv1×1 convolutions to obtain the output feature F. 1i Meanwhile, this RMBF module can be reparameterized into a single convolutional layer with a conv3×3 kernel during the inference phase. Therefore, the RMBF module can ensure that it can fit complex nonlinear relationships during training and learn more knowledge from different perspectives, while ensuring minimal parameter and computational load during the inference phase, thus improving the inference speed of the model.

[0077] Figure 2 In this system, the visually cued dynamic modulation module achieves adaptive feature enhancement or suppression through zero-centralized attention and a small number of learnable cue tokens, thereby improving the performance of super-resolution tasks while avoiding the computational overhead of traditional attention. Figure 1 After extracting features at different levels using four stacked RMVP modules, the last layer of features is then upsampled by interpolation. This takes into account the inevitable loss of some information due to the feature map downsampling using pooling layers and the certain redundancy between different channels.

[0078] Figure 3 In this module, the Grouped Multi-Scale Feature Aggregation (GMFA) module extracts multi-scale features by grouping features and applying low-parameter depthwise convolutions to some groups. It then uses a cascaded depthwise convolution (DWconv) to extract features, reducing computational cost while improving feature utilization. After feature mapping using a GeLU activation function and a standard conv3×3 kernel, a parameterless pixel shuffle operation is used to obtain the final super-resolution image.

Claims

1. A mobile terminal image super-resolution method based on prompt calibration and multi-scale cascade, the method comprising: acquiring a low-resolution image, acquiring a low-resolution image through the input of a mobile terminal; extracting an initial feature map, performing shallow feature extraction on the low-resolution image using a blueprint separation convolution to obtain an initial feature map; extracting deep features, downsampling the initial feature map and inputting the downsampled feature map into a feature extraction network composed of a plurality of RMVP modules in cascade for deep feature extraction; grouped multi-scale feature aggregation, which processes deep features by channel grouping and parallel cascade; fusing features, upsampling the deep features output by the aggregated grouped multi-scale feature extraction network; and fusing the upsampled features with the initial feature map through a skip connection; outputting a high-resolution image, reconstructing the fused features into an image and outputting a high-resolution image; each RMVP module in the feature extraction network composed of a plurality of RMVP modules in cascade comprises a reparameterizable multi-branch fusion module and a dynamic modulation module based on visual prompt guidance; the reparameterizable multi-branch fusion module is composed of a plurality of parallel branches, and the implementation of the reparameterizable multi-branch fusion module comprises: receiving an input feature map, a multi-branch feature extraction unit composed of a plurality of parallel heterogeneous convolution branches in the training stage is used to extract multi-scale features from the input feature map; generating a first fused feature, which is used to fuse the multi-scale features output by the multi-branch feature extraction unit to generate a first fused feature; generating a second fused feature, the first fused feature is input into a convolution layer with a convolution kernel size of conv1×1 for channel dimension reduction and nonlinear transformation to generate a second fused feature.

2. The mobile terminal image super-resolution method based on prompt calibration and multi-scale cascade according to claim 1, characterized in that: The multi-branch feature extraction unit is composed of a plurality of parallel heterogeneous convolution branches in the training stage, which is used to extract multi-scale features from the input feature map; the heterogeneous convolution branches include a first branch, a second branch, a third branch, a fourth branch, a fifth branch and a sixth branch, the first branch is a branch with a convolution kernel of conv3×3, the second branch is a branch with a convolution kernel of conv3×1, the third branch is a branch with a convolution kernel of conv1×3, the fourth branch is a branch with a convolution kernel of conv1×1, the fifth branch is a learnable scale factor matrix that performs element-wise multiplication operation with the input feature map, and the sixth branch is a learnable bias factor matrix that performs element-wise addition operation with the input feature map.

3. The mobile image super-resolution method based on prompt calibration and multi-scale cascade according to claim 1, characterized in that: The dynamic modulation module based on visual prompt guidance comprises: a visual prompt generation unit for generating one or more learnable visual prompt tokens; a feature interaction unit for calculating the correlation between the input feature map and the visual prompt token to generate a dynamic modulation coefficient; a modulation unit for adaptively enhancing or suppressing the input feature map using the dynamic modulation coefficient to generate a modulated feature; and outputting the modulated feature.

4. The mobile image super-resolution method based on prompt calibration and multi-scale cascade according to claim 1, characterized in that: The grouped multi-scale feature aggregation processes deep features by channel grouping and parallel cascade, comprising: receiving an input feature map, dividing the input feature map into N non-overlapping feature subgroups in the channel dimension, where N is an integer greater than 1; The N feature subgroups after division are subjected to parallel multi-scale feature extraction processing, wherein at least one feature subgroup is subjected to a deep convolution operation containing K cascades, K being an integer greater than 1; The N feature subgroups after parallel multi-scale feature extraction processing are spliced in the channel dimension to generate aggregated features, and the aggregated features are output.

Images