Single image super-resolution processing method and device of adaptive residual correction network

Abstract

The application relates to the technical field of image processing, and discloses a single-image super-resolution processing method and device of an adaptive residual correction network, which comprises the following steps: performing feature extraction on a to-be-processed image to obtain initial image features, wherein the resolution of the to-be-processed image is a first resolution; performing multi-level adaptive feature fusion on the initial image features to obtain multi-level intermediate features; in each level of adaptive feature fusion, residual features generated by non-linear inference are corrected, the residual features contain heterogeneous features caused by mapping errors, the correction process converts the heterogeneous features into homogeneous features through a residual correction submodule, and residual deviations of the residual features are eliminated based on the homogeneous features; and performing image recovery according to the multi-level intermediate features to obtain a target image, wherein the resolution of the target image is a second resolution, and the second resolution is greater than the first resolution. The application realizes a single-image super-resolution processing method which takes into account both high fidelity and low calculation cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a single image super-resolution processing method and device of an adaptive residual correction network. BACKGROUND

[0002] Single Image Super-Resolution (SISR) technology aims to reconstruct a high-definition image from a low-resolution image, and has irreplaceable application value in the fields of monitoring and security, medical imaging, satellite remote sensing, etc. In recent years, SISR methods based on deep learning have gradually replaced traditional interpolation methods and become the mainstream of research in this field, but the existing technology still faces two major bottlenecks, making it difficult to balance high fidelity and low computational cost.

[0003] 1. Feature expression limitations and error accumulation problems of convolutional neural networks (CNNs):

[0004] Existing CNN-based SISR methods (such as residual networks ResNet and dense networks DenseNet) rely on local convolution operations to extract features, which are computationally efficient but have two key defects: first, the long-range dependence capture ability is weak due to the limitation of local receptive fields, making it difficult to restore global structural information of the image; second, in the residual learning process, nonlinear inference can easily produce heterogeneous mapping errors (i.e., irrelevant interference information is mixed into the residual features), and this type of error cannot be eliminated through traditional linear superposition, and will continue to accumulate as the network level deepens, eventually leading to fidelity problems such as edge blurring and detail distortion in the reconstructed image.

[0005] 2. High computational overhead problem of visual Transformers:

[0006] To solve the long-range dependence defect of CNNs, SISR methods based on Transformers (such as SwinIR and Restormer) use multi-head self-attention mechanisms to achieve global feature interaction, which has improved the reconstruction fidelity, but this mechanism has significant shortcomings: on the one hand, its computational complexity grows quadratically with the image resolution (complexity is , N is the number of image pixels), which makes the storage requirements and inference time of the model rise sharply, for example, when processing a 512 ×512 resolution image, the calculation amount of the Transformer is 3-5 times that of the CNN of the same size; on the other hand, from the perspective of computational efficiency and parallel computing, the Transformer architecture is not conducive to parallel computing due to its own mechanism characteristics, compared with the CNN model, which is more conducive to efficient parallel computing on hardware by virtue of local convolution and other operations, and is more convenient for practical application and deployment, which also leads to the difficulty of the Transformer architecture to adapt to resource-limited scenarios such as mobile terminals and embedded devices.

[0007] In summary, the contradiction between the insufficient fidelity of the CNN and the low efficiency of the Transformer in the prior art always exists, and how to design an SISR scheme that can suppress error accumulation through accurate residual correction to ensure reconstruction fidelity and rely on the CNN architecture to maintain low computational cost has become a key technical problem to be solved in the field. SUMMARY

[0008] In view of the above problems, the present application provides a single image super-resolution processing method and device of an adaptive residual correction network in order to overcome the above problems or at least partially solve the above problems.

[0009] The present application is implemented by the following technical solutions:

[0010] A single image super-resolution processing method of an adaptive residual correction network, the method is implemented based on an adaptive residual correction network, and includes the following steps:

[0011] Feature extraction is performed on a to-be-processed image to obtain initial image features, and the resolution of the to-be-processed image is a first resolution;

[0012] Multi-level adaptive feature fusion is performed on the initial image features to obtain multi-level intermediate features; in each level of adaptive feature fusion, a residual correction module is used to correct residual features generated by non-linear inference, the residual features contain heterogeneous features caused by mapping errors, and the correction process converts the heterogeneous features into homogeneous features through a residual correction submodule, and eliminates the residual deviation of the residual features based on the homogeneous features;

[0013] Image restoration is performed according to the multi-level intermediate features to obtain a target image, and the resolution of the target image is a second resolution, which is greater than the first resolution.

[0014] As optimization, the adaptive residual correction network comprises a feature extraction layer for performing convolution operation on the image to be processed to obtain initial image features, a plurality of cascaded adaptive feature fusion modules AFG, and an image reconstruction layer for performing image restoration on the multi-level intermediate features; wherein the process of performing multi-level adaptive feature fusion on the initial image features to obtain multi-level intermediate features comprises:

[0015] The initial image features are iteratively optimized by sequentially passing through a plurality of adaptive feature fusion modules AFG with adaptive feature fusion as the core to obtain intermediate features output by each adaptive feature fusion module AFG; wherein the input of the first adaptive feature fusion module AFG is the initial image features, and the input of the adaptive feature fusion module AFG other than the first adaptive feature fusion module AFG is the intermediate features output by the adaptive feature fusion module AFG of the previous level.

[0016] As optimization, each adaptive feature fusion module AFG comprises a plurality of residual correction modules RCB connected in series, and the adaptive feature fusion module AFG processes the input features according to the following steps:

[0017] The first input features are sequentially passed through a plurality of residual correction modules RCB to perform non-linear mapping and residual correction to obtain first output features; wherein the first input features of the first residual correction module RCB are the initial image features, and the first input features of the residual correction module RCB other than the first residual correction module RCB are the output features of the previous residual correction module;

[0018] The first input features and the first output features are fused by a weighted channel concatenation strategy WCC to obtain first fused features; the weighted channel concatenation strategy WCC is to concatenate the channel dimensions of the first input features and the first output features after dynamic weighting by learnable weights;

[0019] The first fused features are subjected to convolution compression processing to obtain intermediate features of the current level.

[0020] As optimization, the residual correction module comprises a first convolution layer, a residual correction submodule, a second convolution layer, a weighted channel concatenation unit, and a third convolution layer connected in sequence; and the residual correction module processes the first input features according to the following steps:

[0021] The first input features are subjected to non-linear mapping by the first convolution layer to obtain residual features;

[0022] The residual features are subjected to bias correction by the residual correction submodule to obtain corrected residual features;

[0023] The corrected residual features are transformed by the second convolutional layer to obtain transformed features;

[0024] The weighted channel splicing unit fuses the first input feature and the transformed feature to obtain the second fused feature;

[0025] The second fused feature is adjusted in channel dimension by the third convolutional layer to obtain the first output feature of the residual correction module.

[0026] As an optimization, the residual correction submodule includes a fourth convolutional layer, a fifth convolutional layer, and an activation function layer; the steps for obtaining corrected residual features through the residual correction submodule include:

[0027] The residual features are compressed in channel dimension by the fourth convolutional layer to obtain the first processing result;

[0028] The activation features are obtained by performing nonlinear activation on the first processing result through an activation function layer;

[0029] The activation features are restored by channel dimension through the fifth convolutional layer to obtain homogeneous features representing the mapping error;

[0030] The difference between the residual feature and the homogeneous feature is used to obtain the corrected residual feature after eliminating the bias.

[0031] As an optimization, the weighted channel splicing strategy is implemented through the following steps:

[0032] The first input feature is weighted according to the first weight to obtain the weighted input feature, and the first output feature is weighted according to the second weight to obtain the weighted output feature. The first weight and the second weight are determined by training the adaptive residual correction network.

[0033] The weighted input features and the weighted output features are concatenated by weighted channels to obtain the first fused feature.

[0034] As an optimization, the specific process of image restoration based on the multi-level intermediate features to obtain the target image is as follows:

[0035] Based on the weights of each of the adaptive feature fusion modules (AFG), the multi-level intermediate features are weighted and concatenated to obtain the final image features.

[0036] The final image features are restored using subpixel convolutional layers to obtain the target image.

[0037] This invention also discloses a single-image super-resolution processing apparatus with an adaptive residual correction network, used to execute the aforementioned single-image super-resolution processing method with an adaptive residual correction network, comprising:

[0038] The feature extraction module is used to extract features from the image to be processed to obtain initial image features, wherein the resolution of the image to be processed is a first resolution;

[0039] The fusion module group is used to perform multi-level adaptive feature fusion on the initial image features to obtain multi-level intermediate features. In each level of adaptive feature fusion, the residual features generated by nonlinear inference are corrected by the residual correction module. The residual features include heterogeneous features caused by mapping errors. The correction process transforms the heterogeneous features into homogeneous features through the residual correction submodule and eliminates the residual bias of the residual features based on the homogeneous features.

[0040] The recovery module group is used to perform image recovery based on the multi-level intermediate features to obtain a target image, wherein the resolution of the target image is a second resolution, which is greater than the first resolution.

[0041] As an optimization, the fusion module group includes multiple cascaded adaptive feature fusion modules AFG, and each of the adaptive feature fusion modules AFG contains multiple cascaded residual correction modules RCB;

[0042] Each residual correction module (RCB) processes the first input feature as follows:

[0043] The residual features are obtained by performing a nonlinear mapping on the first input features through the first convolutional layer.

[0044] The residual correction submodule RCB corrects the heterogeneous features in the residual features caused by mapping error, transforming them into homogeneous features to eliminate residual bias and obtain corrected residual features.

[0045] The corrected residual features are transformed by the second convolutional layer, and then combined with the first input features for weighted channel concatenation and channel dimension adjustment by the third convolutional layer to output the first output feature.

[0046] As an optimization, the specific process of image restoration performed by the restoration module group is as follows:

[0047] Based on the learnable weights of each of the adaptive feature fusion modules (AFG), a weighted channel concatenation operation is performed on the multi-level intermediate features to obtain the final image features.

[0048] The final image features are upsampled and processed using subpixel convolutional layers to generate a target image with a second resolution, where the second resolution is greater than the first resolution of the image to be processed.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] This invention fully leverages the advantages of convolutional neural networks in residual learning and feature fusion. It performs multi-level adaptive feature fusion on the image features of the image to be processed. In the adaptive feature fusion, the nonlinear inference error (heterogeneous features) in the residual features is transformed into homogeneous features, and the residual features are dynamically corrected. This effectively suppresses the error accumulation problem in traditional residual learning and ensures the fidelity of the target image. In addition, image restoration is performed based on multi-level intermediate features. It makes full use of shallow high-frequency detail features (such as edge contours) and deep semantic features (such as structural information) to restore the image, further ensuring the fidelity of the target image. Thus, a single-image super-resolution processing method that balances high fidelity and low computational cost is realized. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0052] Figure 1 This is a flowchart illustrating the steps of a single-image super-resolution processing method using an adaptive residual correction network, as provided in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of an adaptive feature fusion module provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of mapping error in nonlinear inference provided by an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of a residual correction module provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of an adaptive residual correction network provided in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of the structure of a single-image super-resolution processing device with an adaptive residual correction network provided in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0060] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] In related technologies, although visual transformers have shown a more significant advantage than convolutional neural networks in single-image super-resolution, transformers typically require more computational and storage resources than the widely studied convolutional neural networks, which limits their practical application in mobile terminals or embedded devices. Meanwhile, the characteristics of convolutional neural networks—their ability to perceive local receptive fields and share all convolutional weights across the entire feature map—give them favorable inductive bias, allowing them to remain dominant in the field of efficient single-image super-resolution.

[0062] Therefore, this application re-examines the residual learning and feature fusion commonly used in single-image super-resolution and proposes a single-image super-resolution processing method using an adaptive residual correction network. The key motivation of this method is to discover that residual features may be biased due to mapping errors in the residual learning process. By correcting these errors, the processing effect of single-image super-resolution can be improved. Thus, the method fully leverages the advantages of convolutional neural networks in residual learning and feature fusion, achieving a single-image super-resolution processing method that balances high fidelity and low computational cost.

[0063] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a single-image super-resolution processing method using an adaptive residual correction network, as provided in an embodiment of this application. Figure 1 As shown, the single-image super-resolution processing method based on an adaptive residual correction network may include steps S1 to S3:

[0064] Step S1: Extract features from the image to be processed to obtain initial image features. The resolution of the image to be processed is the first resolution.

[0065] Step S2: Perform multi-level adaptive feature fusion on the initial image features to obtain multi-level intermediate features; wherein, in each level of adaptive feature fusion, the residual features generated by nonlinear inference are corrected by the residual correction module. The residual features include heterogeneous features caused by mapping errors. The correction process transforms the heterogeneous features into homogeneous features through the residual correction submodule, and eliminates the residual bias of the residual features based on the homogeneous features.

[0066] Step S3: Perform image restoration based on the multi-level intermediate features to obtain the target image. The resolution of the target image is a second resolution, which is greater than the first resolution.

[0067] Among them, the image to be processed with the first resolution is a low-resolution image. For the image to be processed... (where h represents the length of the image to be processed and w represents the width of the image to be processed), feature extraction can be achieved through convolution. For example, using a... The convolutional layer extracts features from the image to be processed, obtaining the initial image features. (c represents the number of feature channels of the initial image features).

[0068] Multi-level adaptive feature fusion is performed on the initial image features, with each level yielding an intermediate feature. Adaptive feature fusion refers to the process of performing nonlinear inference (i.e., convolution) on image features (including initial or intermediate features) to obtain residual features, and then correcting these residual features. Specifically, nonlinear inference of image features leads to different errors between different coverage areas (i.e., heterogeneous features), which cannot be simply corrected by linear superposition. Therefore, the residual features from nonlinear inference are processed into homogeneous features, which are features that can be linearly superimposed, thus enabling correction of the residual features.

[0069] In some embodiments, multi-level intermediate features can be represented as Where n is the number of levels in the multi-level adaptive feature fusion, and in this embodiment, it is also the number of adaptive feature fusion modules (AFG). Multi-level adaptive feature fusion of image features can be expressed as:

[0070] ;

[0071] in, This represents the intermediate feature of the i-th level, i.e., the intermediate feature output by the i-th adaptive feature fusion module AFG. This represents the intermediate feature of the (i-1)th level, i.e., the intermediate feature output by the (i-1)th adaptive feature fusion module AFG. This indicates an adaptive feature fusion operation. These are the initial image features.

[0072] After obtaining the multi-level intermediate features (that is, the intermediate features corresponding to each level), the multi-level intermediate features are adaptively integrated, and the image is restored based on the adaptive integration result to obtain the target image; wherein, the resolution of the target image is greater than the resolution of the image to be processed, and the target image is a high-resolution image.

[0073] The technical solution implemented in this application fully leverages the advantages of convolutional neural networks in residual learning and feature fusion. It performs multi-level adaptive feature fusion on the image features of the image to be processed. In this adaptive feature fusion, nonlinear inference errors (heterogeneous features) in the residual features are transformed into homogeneous features, dynamically correcting the residual features and effectively suppressing the error accumulation problem in traditional residual learning, thus ensuring the fidelity of the target image. Furthermore, image restoration is performed based on multi-level intermediate features, fully utilizing shallow high-frequency detail features (such as edge contours) and deep semantic features (such as structural information) to further guarantee the fidelity of the target image. Thus, a single-image super-resolution processing method that balances high fidelity and low computational cost is achieved.

[0074] In conjunction with the above embodiments, in one embodiment, this application also provides a single-image super-resolution processing method using an adaptive residual correction network. In this method, the adaptive residual correction network includes a feature extraction layer that performs convolution operations on the image to be processed to obtain initial image features, multiple cascaded adaptive feature fusion modules (AFG), and an image reconstruction layer that restores the image based on the multi-level intermediate features.

[0075] The step S2 above, "performing multi-level adaptive feature fusion on the image features to obtain multi-level intermediate features", may include: sequentially iteratively optimizing the initial image features through multiple adaptive feature fusion modules (AFG) with adaptive feature fusion as the core, and obtaining the intermediate features output by each adaptive feature fusion module; wherein, the input feature of the first adaptive feature fusion module is the image features, and the input feature of the remaining adaptive feature fusion modules is the intermediate features output by the previous adaptive feature fusion module.

[0076] In this embodiment, multiple adaptive feature fusion modules in the adaptive residual correction network are connected in series. Each adaptive feature fusion module sequentially performs adaptive feature fusion on the image features, and the intermediate feature output by each adaptive feature fusion module is a level of intermediate feature. That is, the first adaptive feature fusion module performs adaptive feature fusion on the initial image features obtained in step S1, and subsequent adaptive feature fusion modules perform adaptive feature fusion on the intermediate features output by the previous adaptive feature fusion module, ultimately obtaining the intermediate features output by each adaptive feature fusion module, thus obtaining multi-level intermediate features.

[0077] Furthermore, such as Figure 2 As shown, each of the adaptive feature fusion modules (AFG) includes multiple cascaded residual correction modules (RCB). Each adaptive feature fusion module performs adaptive feature fusion on the input features according to steps A1 to A3:

[0078] Step A1: The first input feature is subjected to nonlinear inference and residual correction through the multiple residual correction modules in sequence to obtain the first output feature; wherein, the first input feature corresponding to the first residual correction module is the input feature of the adaptive feature fusion module, and the first input feature corresponding to the other residual correction modules is the feature output by the previous residual correction module.

[0079] Step A2: Weighted channel concatenation of the first input feature and the first output feature yields the first fused feature. In other words, the first input feature and the first output feature are fused using a weighted channel concatenation strategy (WCC) to obtain the first fused feature; the weighted channel concatenation strategy (WCC) dynamically weights the channel dimensions of the first input feature and the first output feature using learnable weights before concatenation.

[0080] Step A3: Based on the first fused feature, obtain the intermediate features of the current level. Perform convolutional compression processing on the first fused feature to obtain the intermediate features of the current level.

[0081] In this embodiment, the multiple residual correction modules in the adaptive feature fusion module are connected in series, and each residual correction module is used sequentially to perform adaptive feature fusion on the input features. That is, the first residual correction module is used to perform nonlinear inference and residual correction on the input features of the adaptive feature fusion module, and subsequent residual correction modules perform nonlinear inference and residual correction on the output features of the previous residual correction module, finally obtaining the first output feature (i.e., the output feature of the last residual correction module).

[0082] Considering the presence of undesirable errors in the nonlinear inference of residual learning, these errors are heterogeneous and non-cumulative, meaning the residual features contain heterogeneous characteristics. Therefore, convolution should not be used to accumulate errors within local regions. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of a mapping error in nonlinear inference provided in an embodiment of this application. If a The residual characteristics and a If we perform convolution with the convolution kernel, then... The error within a covered region should differ from the errors in other covered regions (heterogeneous features); if the stride of the convolution is 1, these heterogeneous features can be reorganized into... The shape, then rearranged into Therefore, the residual correction process can be analogized to the inverse of subpixel convolution (PixelShuffle). To simulate this process, a residual correction submodule is used to implement residual correction.

[0083] In one optional embodiment, the residual correction module includes a first convolutional layer, a residual correction submodule, a second convolutional layer, a weighted channel splicing unit, and a third convolutional layer connected in sequence; the residual correction module processes the first input feature according to the following steps:

[0084] A1.1. The first input features are nonlinearly mapped through the first convolutional layer to obtain residual features.

[0085] A1.2. The residual features are corrected by the residual correction submodule to obtain the corrected residual features; that is, the residual features are processed into homogeneous features by the residual correction submodule, and the residual features are corrected based on the homogeneous features to obtain the corrected residual features.

[0086] A1.3. The corrected residual features are transformed by the second convolutional layer to obtain transformed features.

[0087] A1.4 The first input feature and the transformed feature are fused by the weighted channel splicing unit to obtain the second fused feature.

[0088] A1.5. The second fused feature is adjusted in channel dimension using a third convolutional layer to obtain the first output feature of the residual correction module. The second fused feature is then... The convolution process is used to obtain the output features of the residual correction module.

[0089] like Figure 4 As shown, the structure of the residual correction submodule is as follows: The fourth convolutional layer, ReLU function, The fifth convolutional layer and the ReLU function (activation function layer). Specifically, step A1.2, "processing the residual features into homogeneous features through the residual correction submodule, and correcting the residual features based on the homogeneous features to obtain corrected residual features," may specifically include: performing... The convolution process is performed to obtain the first processing result; the first processing result is then activated and then... The convolution process is used to obtain homogeneous features; the residual features are subtracted from the homogeneous features to obtain the corrected residual features.

[0090] Thus, for a given residual characteristic Through two Convolutional layers adjust the channels of residual features to This generates a homogeneous feature that can be directly subtracted (linearly calculated). In some embodiments, the corrected residual feature... It can be represented as:

[0091] ;

[0092] in, Indicates two Convolutional layer This represents the learnable parameters in the residual correction submodule. Furthermore, to impose non-negativity and sparsity constraints on the error, the ReLU function can be used to process the correction residual characteristics at the end.

[0093] In this embodiment of the application, to better achieve feature fusion, the residual connection is extended to weighted channel concatenation, and feature fusion is achieved through weighted channel concatenation in each residual correction module. That is, in steps A1.3 and A1.4, the corrected residual features are convolved (e.g., After convolution, the second fused feature is obtained by weighted channel concatenation with the first input feature. Finally, the output feature of the residual correction module is obtained through step A1.5.

[0094] like Figure 4 As shown, the residual correction module includes: a residual correction submodule and two... Convolutional layers, ReLU function and Convolutional layer. Specifically, the residual correction module utilizes in step A1.1 The convolutional layer performs convolution processing on the first input features to obtain residual features; then, the ReLU function is used to process the residual features, and the processing result is used in step A1.2 to obtain corrected residual features; in steps A1.3 and A1.4, through... The convolutional layer performs convolution processing on the corrected residual features and applies them according to the third weight. and the fourth weight Weighted channel concatenation is performed to obtain the second fused feature; finally, the second fused feature is processed in step A1.5. The convolution process is used to obtain the output features of the residual correction module.

[0095] In some embodiments, the output characteristics of the residual correction module It can be represented as:

[0096] ;

[0097] in, This indicates pointwise convolution (i.e., performing...) (convolution processing) Indicates the characteristics of the corrected residuals;

[0098] After each residual correction module in the adaptive feature fusion module performs nonlinear inference and residual correction on the first input feature, step A2 is executed to concatenate the first input feature and the first output feature using weighted channels to obtain the first fused feature. Specifically, the first input feature is weighted according to a first weight to obtain a weighted input feature, and the first output feature is weighted according to a second weight to obtain a weighted output feature. The first weight and the second weight are determined by training the adaptive residual correction network. The weighted input feature and the weighted output feature are then concatenated using weighted channels to obtain the first fused feature.

[0099] Thus, by performing weighted channel concatenation on the first input feature and the second input feature in the adaptive feature fusion module, image feature fusion can be better achieved, resulting in more accurate intermediate features. In some embodiments, the intermediate features output by the (i+1)th adaptive feature fusion module... It can be represented as:

[0100] ;

[0101] in, This represents the intermediate feature output by the i-th adaptive feature fusion module. This indicates pointwise convolution (i.e., performing...) (convolution processing) This represents the first output feature, which is the output feature of the last residual correction module in the adaptive feature fusion module. Indicates the first weight. This indicates the second weight.

[0102] The training methods for obtaining the first, second, third, and fourth weights are as follows:

[0103] Phase 1: High-frequency sensing initialization.

[0104] Stage 1.1: Extracting Edge Features: Using the Sobel operator to extract edge feature maps from low-resolution images. ;

[0105] Stage 1.2: After normalizing E, it is expanded into initial weights matching the feature channels through a 1×1 convolution. The first weight... Second weight For example:

[0106] , ; For the Sigmoid function, For the outer product, The weights are all-one vectors along the channel dimension, ensuring that the initial values ​​are higher in the edge regions. This stage makes the weights sensitive to high-frequency details from the early stages of training, reducing the cost of blind exploration. Third and fourth weights , Similarly, it is also based on edge feature map initialization.

[0107] Phase 2: Dynamic loss is linked to weight decay.

[0108] Phase 2.1 Constructing a multi-part loss function. A task-aware loss function is designed, incorporating the effectiveness of feature selection for weights into the loss calculation, while dynamically adjusting the weight decay intensity:

[0109] Basic loss: Preserve L1 loss (to ensure pixel-level accuracy) + Perceptual loss (to ensure visual quality):

[0110] ; ;

[0111] L1 loss is used to measure the sum of absolute differences between the reconstructed image and the real high-resolution image at the pixel level. N represents the total number of pixels involved in the loss calculation, which means calculating the differences for all N pixels in the image and then averaging them. This represents the value of the i-th pixel in the reconstructed image (the high-resolution image predicted by the model). This represents the true value of the i-th pixel in a real high-resolution image. , This represents the weight of the corresponding loss.

[0112] Weight-guided loss: Penalizes the inconsistency between weights and high-frequency features, directing weights towards key features.

[0113] ;

[0114] Dynamic weight decay: As the training epochs increase, the penalty for large weights is gradually reduced (to avoid excessive weight shrinkage).

[0115] ; is the initial decay coefficient, T is the decay period, and epoch is the current cycle.

[0116] Phase 2.2, Total Loss and Parameter Update. The final total loss is: This stage links weight learning with the high-frequency feature protection target, while avoiding premature convergence to averaged weights. An optimizer (such as Adam) is used to minimize the total loss through backpropagation, thereby iteratively updating the values ​​of the first, second, third, and fourth weights. The weights used to guide the loss.

[0117] Phase 3: Cross-level weight coordination constraints. The weight update magnitude of different modules is dynamically adjusted by utilizing the importance of hierarchical features, so that the weights of higher-level modules (responsible for semantics) and lower-level modules (responsible for details) complement each other.

[0118] Phase 3.1, Layer-by-Layer Learning Rate Scaling. Calculate the information entropy of intermediate features at each layer. Using information entropy Scaling the learning rate of the weights at the corresponding level: , For all levels; This represents the learning rate of the weights corresponding to the i-th level after scaling. The base learning rate is the initial learning rate setting without layer scaling. This represents the maximum value among the intermediate feature entropies of all levels (where j represents each level). By taking the maximum value, the entropy of the i-th level is compared with that of the most informative level overall, thus achieving a reasonable scaling of the learning rate.

[0119] Phase 3.2, Constraints on Weight Differences Between Adjacent Levels: Limiting the differences in weights between adjacent levels: ; The loss represents the collaborative constraint loss across weights, which measures the degree of difference between weights in adjacent layers. The purpose is to avoid abrupt changes in weights, thereby preventing features from jumping between different layers; n represents the total number of layers, that is, there are a total of n layers participating in the calculation of this loss. This represents the i-th level corresponding to the th level. Weight; This represents the (i+1)th level corresponding to the first level. Weight; for The norm is used to calculate the sum of the absolute differences between two vectors (here, the weight vectors of adjacent levels), and can intuitively reflect the magnitude of the differences between the weights of adjacent levels.

[0120] Phase 3.3, Parameter Update: The total loss is incorporated, and the optimizer is iterated and updated again to allow the first, second, third, and fourth weights to continue to be optimized under the constraint of hierarchical collaboration.

[0121] In other words, during model training, the acquisition of the first, second, third, and fourth weights is automatically completed through a preset three-stage learning rule, the rules of which are defined as follows:

[0122] High-frequency feature initialization rules:

[0123] The rule that the initial weight values ​​should be strongly correlated with the high-frequency details (such as edges and textures) of the input image is as follows: the Sobel operator is used to extract the edge feature map of the low-resolution image, and the edge features are expanded to the initial values ​​that match the weight channels through 1×1 convolution, so that the weights are assigned higher initial weights to the high-frequency regions in the early stage of training (the first, second, third and fourth weights are all initialized according to this rule).

[0124] Dynamic loss constraint rules:

[0125] The pre-defined weight learning requires a rule that specifically strengthens the ability to select high-frequency features:

[0126] Basic loss rule: A weighted sum of L1 loss and perceptual loss is used to constrain the pixel-level and perceptual-level differences between the reconstructed image and the real high-resolution image;

[0127] Weight guidance rule: By penalizing the deviation of weights from high-frequency feature maps, the weights are forced to focus on high-frequency regions;

[0128] Decay scheduling rule: As the training rounds increase, the penalty intensity for large weights is reduced according to the exponential decay formula to avoid excessive shrinkage of weights.

[0129] Hierarchical collaborative optimization rules:

[0130] The pre-defined rule is that the weights of different levels must maintain complementarity and continuity:

[0131] Learning rate adaptation rule: The weight update magnitude is dynamically adjusted based on the information entropy of intermediate features at each level (the richer the information, the higher the learning rate);

[0132] Difference constraint rule: By restricting the abrupt changes in weights of adjacent layers through the L1 norm, the smoothness of cross-layer feature fusion is guaranteed.

[0133] With the aforementioned preset rules, the model can automatically complete the initialization, iterative optimization, and hierarchical collaborative adjustment of weights without human intervention during training, ultimately obtaining the optimal weight values ​​adapted to the super-resolution task.

[0134] The technical solution implemented in this application adjusts the channels of the residual features to process heterogeneous features into homogeneous features, and corrects the residual features based on the homogeneous features. This effectively suppresses the error accumulation problem in traditional residual learning and ensures the fidelity of the target image. Furthermore, each adaptive feature fusion module and the multiple residual correction modules in the adaptive feature fusion module perform feature fusion by weighted channel splicing, further ensuring the effect of single image super-resolution processing.

[0135] In conjunction with the above embodiments, in one implementation, this application also provides a single-image super-resolution processing method based on an adaptive residual correction network. In this method, step S3, "performing image restoration based on the multi-level intermediate features to obtain the target image," specifically includes sub-steps S3.1 to S3.2:

[0136] Step S3.1: Based on the weight of each adaptive feature fusion, the multi-level intermediate features are weighted and concatenated to obtain the final image features.

[0137] Step S3.2: Use subpixel convolutional layers to perform image restoration on the final image features to obtain the target image.

[0138] In this embodiment, to better achieve feature fusion, the residual connection is extended to weighted channel concatenation, and feature fusion is achieved through weighted channel concatenation at each layer of the adaptive residual correction network. Therefore, based on the weight of each adaptive feature fusion, multi-level intermediate features are weighted and concatenated to obtain fully fused final image features.

[0139] The weights for adaptive feature fusion are determined through training an adaptive residual correction network. Specifically, based on the weights of each adaptive feature fusion, the multi-level intermediate features are weighted and concatenated, including: weighting each level's intermediate feature using its corresponding adaptive feature fusion weights, and then concatenating the weighted intermediate features to obtain concatenated intermediate features; the concatenated intermediate features are then sequentially processed... convolution, The convolution process yields the final image features.

[0140] In some embodiments, the final image features It can be represented as:

[0141] ;

[0142] in, The weights of the adaptive feature fusion at layer i are represented by ( , , (These are the weights for the 0th, 1st, and nth adaptive feature fusions, respectively). express Convolutional layers and Convolutional layer; This represents the operation of concatenating features along the channel dimension, resulting in a specific feature. The intermediate features of splicing individual channels.

[0143] Subpixel convolutional layers (PixelShuffle) are used to rearrange and combine image features from multiple channels (i.e., rearrange the pixels in the image features) to form a new high-resolution image. Therefore, after obtaining the final image features, subpixel convolutional layers are used to process these features, rearranging the pixels to obtain a target image with a second resolution. ,in, , And r represents the scaling factor.

[0144] The technical solution implemented in this application fuses multi-level intermediate features through weighted channel stitching, further ensuring the effectiveness of single-image super-resolution processing.

[0145] like Figure 5 As shown, Figure 5 This is a schematic diagram of an adaptive residual correction network provided in an embodiment of this application. The adaptive residual correction network includes: a feature extraction module, a fusion module group consisting of multiple adaptive feature fusion modules, and an image restoration module. The feature extraction module is composed of... The image restoration module consists of convolutional layers. Convolutional layers Convolutional layers, PixelShuffle, and Composition of convolutional layers.

[0146] Specifically, the single-image super-resolution processing process based on the adaptive residual correction network is as follows: First, through the feature extraction module... The convolutional layer extracts features from the image to be processed, obtaining initial image features. Next, the initial image features are adaptively fused sequentially through a group of fusion modules consisting of multiple adaptive feature fusion modules (AFG), yielding intermediate features output by each AFG module. Finally, the intermediate features at each level are weighted using the corresponding adaptive feature fusion weights, and the weighted intermediate features at each level are concatenated to obtain concatenated intermediate features. These concatenated intermediate features are then processed sequentially... convolution, Convolution is performed to obtain the final image features; PixelShuffle is then used to process these final image features, and the results are then processed... Convolution processing yields the target image.

[0147] In this embodiment, the adaptive residual correction network is a CNN-based network. Based on this network, multi-level adaptive feature fusion is performed on the image features of the image to be processed. During this adaptive feature fusion, nonlinear inference errors (heterogeneous features) in the residual features are transformed into homogeneous features, dynamically correcting the residual features. This effectively suppresses the error accumulation problem in traditional residual learning, ensuring the fidelity of the target image. Furthermore, image restoration is performed based on multi-level intermediate features, fully utilizing shallow high-frequency detail features (such as edge contours) and deep semantic features (such as structural information). Additionally, feature fusion is performed throughout the network using weighted channel concatenation, further ensuring the effectiveness of single-image super-resolution processing. Therefore, no complex architecture design or training strategy is required, facilitating efficient model inference and parallelization.

[0148] This application also provides a single-image super-resolution processing device based on an adaptive residual correction network, referring to... Figure 6 As shown, Figure 6 This is a schematic diagram of a single-image super-resolution processing device based on an adaptive residual correction network provided in an embodiment of this application. The device includes:

[0149] The feature extraction module is used to extract features from the image to be processed to obtain image features, wherein the resolution of the image to be processed is a first resolution;

[0150] The fusion module group is used to perform multi-level adaptive feature fusion on the initial image features to obtain multi-level intermediate features. In each level of adaptive feature fusion, the residual features generated by nonlinear inference are corrected by the residual correction module. The residual features include heterogeneous features caused by mapping errors. The correction process transforms the heterogeneous features into homogeneous features through the residual correction submodule and eliminates the residual bias of the residual features based on the homogeneous features.

[0151] The recovery module is used to perform image recovery based on the multi-level intermediate features to obtain a target image, wherein the resolution of the target image is a second resolution, which is greater than the first resolution.

[0152] It is understood that the single-image super-resolution processing device based on adaptive residual correction network in the embodiments of this application can implement the single-image super-resolution processing method based on adaptive residual correction network in the above embodiments. The single-image super-resolution processing device based on adaptive residual correction network and the single-image super-resolution processing method based on adaptive residual correction network have the same advantages over the prior art, and will not be repeated here.

[0153] This application also provides an electronic device, see embodiments thereof. Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps of the single-image super-resolution processing method based on an adaptive residual correction network described in the embodiments of this application.

[0154] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the single-image super-resolution processing method based on an adaptive residual correction network described in this application.

[0155] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the single-image super-resolution processing method based on an adaptive residual correction network described in this application.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0157] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0161] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0162] The above provides a detailed description of a single-image super-resolution processing method and apparatus based on an adaptive residual correction network provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A single image super-resolution processing method of an adaptive residual correction network, characterized by, The method is implemented based on an adaptive residual correction network and includes the following steps. Feature extraction is performed on the to-be-processed image to obtain initial image features, and the to-be-processed image has a first resolution. Multi-level adaptive feature fusion is performed on the initial image features to obtain multi-level intermediate features; in each level of adaptive feature fusion, a residual correction module is used to correct residual features generated by non-linear inference, and the residual features contain heterogeneous features caused by mapping errors, and the correction process includes: processing the residual features by a residual correction submodule, the residual correction submodule includes a 1×1 fourth convolutional layer, an activation function layer and a 1×1 fifth convolutional layer connected in sequence, the residual features are first compressed in channel dimension by the fourth convolutional layer, then are activated by the activation function layer, and finally are restored in channel dimension by the fifth convolutional layer to obtain homogeneous features representing mapping errors, and the residual features are subtracted from the homogeneous features to eliminate the residual bias of the residual features; Image restoration is performed according to the multi-level intermediate features to obtain a target image, and the target image has a second resolution, which is greater than the first resolution.

2. The single image super-resolution processing method of adaptive residual correction network according to claim 1, characterized in that, The adaptive residual correction network includes a feature extraction layer for performing convolutional operation on the to-be-processed image to obtain initial image features, a plurality of cascaded adaptive feature fusion modules AFG, and an image reconstruction layer for performing image restoration on the multi-level intermediate features; wherein the multi-level adaptive feature fusion on the initial image features to obtain the multi-level intermediate features includes: Iterative optimization is performed on the initial image features by a plurality of adaptive feature fusion modules AFG in sequence with adaptive feature fusion as the core to obtain intermediate features output by each adaptive feature fusion module AFG; wherein the input of the first adaptive feature fusion module AFG is the initial image features, and the input of the adaptive feature fusion module AFG other than the first adaptive feature fusion module AFG is the intermediate features output by the adaptive feature fusion module AFG of the previous level.

3. The single image super-resolution processing method of adaptive residual correction network according to claim 2, characterized in that, Each adaptive feature fusion module AFG includes a plurality of series-connected residual correction modules RCB, and the adaptive feature fusion module AFG processes the input features according to the following steps: Non-linear mapping and residual correction are performed on the first input features by a plurality of residual correction modules RCB in sequence to obtain first output features; wherein the first input features of the first residual correction module RCB are the initial image features, and the first input features of the residual correction module RCB other than the first residual correction module RCB are the output features of the previous residual correction module; Fusion is performed on the first input features and the first output features by a weighted channel concatenation strategy WCC to obtain first fusion features; the weighted channel concatenation strategy WCC is to concatenate the first input features and the first output features after dynamically weighting the channel dimensions of the first input features and the first output features by learnable weights; The first fusion feature is subjected to convolution compression processing to obtain an intermediate feature of a current level.

4. The single image super-resolution processing method of adaptive residual correction network according to claim 3, characterized in that, The residual correction module comprises a first convolution layer, a residual correction submodule, a second convolution layer, a weighted channel splicing unit and a third convolution layer connected in sequence; the residual correction module processes the first input feature according to the following steps: The first input feature is subjected to nonlinear mapping by the first convolution layer to obtain a residual feature; The residual feature is subjected to bias correction by the residual correction submodule to obtain a corrected residual feature; The corrected residual feature is subjected to feature conversion by the second convolution layer to obtain a converted feature; The first input feature and the converted feature are fused by the weighted channel splicing unit to obtain a second fusion feature; The third convolution layer is used to adjust the channel dimension of the second fusion feature to obtain the first output feature of the residual correction module.

5. The single image super-resolution processing method of adaptive residual correction network according to claim 4, characterized in that, The residual correction submodule comprises a fourth convolution layer, a fifth convolution layer and an activation function layer; the step of obtaining a corrected residual feature by the residual correction submodule comprises: The residual feature is subjected to channel dimension compression by the fourth convolution layer to obtain a first processing result; The first processing result is subjected to nonlinear activation by the activation function layer to obtain an activated feature; The activated feature is subjected to channel dimension recovery by the fifth convolution layer to obtain a homogeneous feature representing mapping error; The residual feature and the homogeneous feature are subtracted to obtain a corrected residual feature with eliminated bias.

6. The single image super-resolution processing method of adaptive residual correction network according to claim 3, characterized in that, The weighted channel splicing strategy is implemented by the following steps: The first input feature is weighted according to a first weight to obtain a weighted input feature, and the first output feature is weighted according to a second weight to obtain a weighted output feature, wherein the first weight and the second weight are determined by training of the adaptive residual correction network; The weighted input feature and the weighted output feature are subjected to weighted channel splicing to obtain a first fusion feature.

7. The single image super-resolution processing method of adaptive residual correction network according to any one of claims 1-6, characterized in that, The specific process of image restoration according to the multi-level intermediate features is as follows: The multi-level intermediate features are subjected to weighted channel splicing according to the weight of each adaptive feature fusion module AFG to obtain a final image feature; The final image feature is subjected to image restoration by a sub-pixel convolution layer to obtain a target image.

8. An apparatus for single image super-resolution with adaptive residual correction network, configured to perform the method of any one of claims 1-7. It comprises: a feature extraction module configured to extract features from a to-be-processed image to obtain initial image features, wherein the to-be-processed image has a first resolution; The fusion module group is used for multi-level adaptive feature fusion on the initial image features to obtain multi-level intermediate features. In each level adaptive feature fusion, a residual correction module is used to correct a residual feature generated by non-linear inference, and the residual feature contains heterogeneous features caused by mapping errors. The correction process includes: processing the residual feature by a residual correction submodule, which includes a 1*1 fourth convolution layer, an activation function layer and a 1*1 fifth convolution layer connected in sequence. The residual feature is first compressed in channel dimension by the fourth convolution layer, then activated by the activation function layer, and finally recovered in channel dimension by the fifth convolution layer to obtain homogeneous features representing mapping errors. The residual feature is subtracted from the homogeneous features to eliminate the residual bias of the residual feature. The recovery module group is used for image recovery according to the multi-level intermediate features to obtain a target image, and the resolution of the target image is a second resolution, which is greater than the first resolution.

9. The single image super-resolution processing apparatus of adaptive residual correction network according to claim 8, wherein, The fusion module group includes a plurality of cascaded adaptive feature fusion modules AFG, and each adaptive feature fusion module AFG includes a plurality of series-connected residual correction modules RCB. Each residual correction module RCB processes the input first input feature in the following manner: a first convolution layer is used to perform non-linear mapping on the first input feature to obtain a residual feature; the residual correction submodule RCB is used to correct the heterogeneous features in the residual feature caused by mapping errors, convert them into homogeneous features to eliminate the residual bias, and obtain a corrected residual feature; a second convolution layer is used to perform feature conversion on the corrected residual feature, and then the first input feature is combined for weighted channel splicing and channel dimension adjustment of the third convolution layer to output a first output feature.

10. The single image super-resolution processing apparatus of adaptive residual correction network according to claim 9, wherein, The specific process of the recovery module group for image recovery is as follows: According to the learnable weights of each adaptive feature fusion module AFG, a weighted channel splicing operation is performed on the multi-level intermediate features to obtain a final image feature; a sub-pixel convolution layer is used to perform up-sampling and other image recovery processes on the final image feature to generate a target image with a second resolution, and the second resolution is greater than the first resolution of the image to be processed.

Images