Dual-domain learning method for single image super-resolution based on dynamic spectrum hybrid network, electronic device and storage medium

Abstract

The application discloses a dual-domain learning method for single-image super-resolution based on a dynamic spectrum hybrid network, electronic equipment and a storage medium, which effectively integrates spatial domain and frequency domain features to improve the quality and applicability of image super-resolution, and comprises the following steps: in a decomposition stage, low-frequency features and high-frequency features are extracted from a given input image through convolution layers with a step of 1 and 3, respectively; in step 2, the low-frequency features are obtained through a spatial feature after a plurality of cascaded residual dense blocks, and the high-frequency features are obtained through a frequency feature after a plurality of cascaded complex blocks; in step 3, the spatial feature and the frequency feature are enhanced and fused by a feature enhancement and selection unit; and in step 4, the spatial feature and the frequency feature are processed through separate but interconnected paths by a dynamic feature fusion encoder, and finally, a super-resolution image output by the dynamic feature fusion encoder is output.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a dual-domain learning method for single image super-resolution based on a dynamic spectrum hybrid network, an electronic device and a storage medium. BACKGROUND

[0002] The role of image super-resolution (SR) technology is to reconstruct a low-resolution image into a high-resolution version. In the field of computer vision, this technology is mainly applied in security monitoring, medical imaging, satellite imaging, etc., and has high requirements for the detail recovery of images.

[0003] Although the development of deep learning in recent years has significantly improved the effect of super-resolution technology, the existing technology mainly adopts a single domain (usually the spatial domain) processing method, which leads to poor results when dealing with frequency-related image characteristics. For example, traditional convolutional neural network (CNN) based methods mainly focus on the explicit details of local pixels, and they often ignore the frequency characteristics of the image, which are key factors affecting the overall quality of the image, resulting in good recovery results in spatial details but poor performance in preserving high-frequency characteristics of the image, which easily leads to distortion or excessive smoothing of the details of the reconstructed image, especially in application scenarios where subtle texture and edge information need to be represented in the image (such as medical imaging, satellite pictures). In addition, some existing methods attempt to make up for the shortcomings of spatial domain methods by introducing frequency domain processing, such as using Fourier transform methods, but these methods usually lack a dynamic adjustment mechanism to adjust the processing weight of frequency and spatial features according to the specific needs of image content, resulting in the inability to fully utilize the complementarity between the two characteristics. This static feature processing strategy limits the adaptability and effectiveness of super-resolution technology in a variety of application environments.

[0004] In summary, the existing technology still faces significant challenges in dynamically and effectively integrating spatial and frequency domain characteristics in the process of improving image super-resolution, especially when dealing with images with complex textures and high-frequency details. SUMMARY

[0005] The purpose of the present application is to provide a dual-domain learning method for single image super-resolution based on a dynamic spectrum hybrid network, an electronic device and a storage medium, which solves the problems existing in the prior art and effectively integrates spatial and frequency domain features to improve the quality and applicability of image super-resolution.

[0006] In order to achieve the above purpose, the solution of the present application is:

[0007] A dual-domain learning method for single image super-resolution based on a dynamic spectrum hybrid network, comprising:

[0008] Step 1, in the decomposition stage, for a given input image , low-frequency features and high-frequency features are extracted respectively by convolution layers with steps , , , , , and the calculation formula is expressed as

[0009] ;

[0010] Step 2, after decomposition, spatial features are obtained by cascade residual dense blocks , and frequency features are obtained by cascade complex blocks , , and the calculation formula is expressed as

[0011] ;

[0012] Step 3, spatial features and frequency features processed in step 2 are enhanced and fused by feature enhancement and selection units, specifically:

[0013] In the feature enhancement and selection unit, spatial features and frequency features are first processed by convolution layers to obtain convolution feature maps , then deep convolution is applied on the convolution feature maps by spatial attention mechanism, and sigmoid activation is performed to generate spatial attention maps , and the calculation formula is expressed as

[0014] ;

[0015] wherein, denotes a sigmoid activation function; denotes a deep convolution layer with learnable weights, , denote the weights and bias of the deep convolution layer, respectively;

[0016] The spatial attention map is executed with the convolution feature map element-wise multiplication of the spatial participation feature map and refining it into a refined feature map through a convolutional layer , then through a global average pooling (GAP) layer to obtain a descriptor ;

[0017] The channel attention map is obtained by the formula , where , , denote the weights of the two fully connected layers, denote the real space, denote the number of channels of the image, denote the compression ratio, , denote the bias of the two fully connected layers, denote the ReLU activation function;

[0018] The channel attention map is multiplied element-wise with the refined feature map to realize a scaled refined feature , thereby obtaining a per-channel enhanced feature map , and processed using a convolutional layer to obtain an output feature map , which is the final output of the feature enhancement and selection unit , which is the result of the composite feature enhancement and selection mechanism of the feature enhancement and selection unit;

[0019] Step 4, the spatial features and frequency features obtained in step 2 are processed by separate but interconnected paths by a dynamic feature fusion encoder:

[0020] The frequency features are transformed into the frequency domain by a fast Fourier transform to obtain frequency encoded features , whose computational formula is expressed as ; the frequency encoded features are substituted into the formula to obtain a set of learning weights , where denotes the global average pooling operation, denotes the fully connected network, denotes the Softmax function;

[0021] An adaptive convolution block is set in the dynamic feature fusion encoder, which is composed of complex-valued convolution and is used to process the real component of complex features and imaginary components to capture the full spectrum of the dynamics of the features , whose computational expression is , where represents the number of convolution kernels, represents the index of the convolution kernel, represents the weight corresponding to the th convolution kernel, represents the th complex-valued convolution;

[0022] At the same time, the spatial features are processed concisely through convolution cascades, followed by gating operations and subsequent fusion, whose computational expression is

[0023] ;

[0024] , where represents the enhanced spatial features; represents the simplified composite function; represents the last convolution layer before the fusion of spatial features and frequency features; represents two consecutive convolution layers; represents element-wise multiplication; represents the spatial features processed by a single convolution branch;

[0025] The fusion-enhanced features are obtained through the formula to re-integrate the spatial information, where represents the inverse fast Fourier transform;

[0026] Finally, the full spectrum is merged with the fusion-enhanced features to obtain the feature set , and through a convolution layer with ReLU activation, the super-resolution image output by the dynamic feature fusion encoder is obtained .

[0027] Preferably, in step 4, the ComplexFractiReLU activation function and strategic feature fusion are used in the architecture of the dynamic feature fusion encoder, and the overall operation of ComplexFractiReLU is expressed by the following computational formula:

[0028] ;

[0029] ;

[0030] , where represents the use of a radius with the number of boxes to calculate the fractal dimension of the function of the polynomial fitting on the double logarithmic graph of the unique number of boxes to cover the data points in the complex plane; represents the complex input of the function, , respectively represent the real part and the imaginary part of the complex input; represents the polynomial fitting; represents the radius; represents the number of boxes covering the data points on the complex plane when the radius is scaled; represents the CFReLU activation function.

[0031] An electronic device comprising a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor the dual-domain learning method for single image super-resolution based on dynamic spectrum hybrid network.

[0032] A computer readable storage medium, the storage medium stores a computer program; the computer program is executed in the computer, so that the computer executes the dual-domain learning method for single image super-resolution based on dynamic spectrum hybrid network.

[0033] After adopting the above technical solutions, the present application has the following technical effects:

[0034] The present application can construct a dynamic hybrid spectrum synthesis network (DHS2Net), which can cope with the challenge of reconstructing a high-resolution image from a low-resolution image by utilizing the spectral characteristics of the image; the present application can dynamically adjust the processing strategy of the spatial domain and the frequency domain features according to the complexity of the image content by introducing a dynamic feature fusion encoder (DFFE) and a feature enhancement selection unit (FESU), thereby more finely reconstructing the image details; wherein, when performing complex value processing, complex value convolution and complex value activation function are adopted, which allows the network to simultaneously process the amplitude and phase information of the image, which is particularly important for maintaining high-frequency details, and can significantly improve the visual quality and realism of the image. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the system architecture diagram of the embodiment of the present application;

[0036] Figure 2 is the dynamic feature fusion encoder architecture diagram of the embodiment of the present application;

[0037] Figure 3 is the CFReLU activation function architecture diagram of the embodiment of the present application;

[0038] Figure 4The correlation between signal frequency and fractal dimension and the slope of CFReLU at different fractal dimensions are shown in the specific embodiments of the present invention.

[0039] Figure 5 This is a comparison chart of experimental performance of a specific embodiment of the present invention;

[0040] Figure 6 This is a comparison image of the detailed regions output by different models in a specific embodiment of the present invention;

[0041] Figure 7 This is a comparison chart of the high-frequency regions output by different models in a specific embodiment of the present invention. Detailed Implementation

[0042] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0043] refer to Figures 1-7 As shown, this invention discloses a dual-domain learning method for single-image super-resolution based on a dynamic spectrum hybrid network, comprising:

[0044] Step 1: In the decomposition stage, for a given input image... Each through step size for , convolutional layers , To extract low-frequency features High-frequency characteristics Its calculation formula is expressed as:

[0045] ;

[0046] Step 2: After decomposition, low-frequency features pass Cascaded residual dense blocks Spatial features were obtained later. High-frequency characteristics pass Cascaded complex value blocks Frequency characteristics were then obtained. Its calculation formula is expressed as:

[0047] ;

[0048] Step 3, Spatial features after processing in Step 2 Frequency characteristics enhancement and selection unit (FESU) to retain important features while discarding noise or less useful information, so as to produce a clearer and more accurate high-resolution image, the feature enhancement and selection unit acts as an adaptive filter therein, dynamically adjusting spatial features , frequency features to prioritize key information paths, specifically:

[0049] In the feature enhancement and selection unit, first spatial features , frequency features are processed through a convolutional layer to obtain a convolutional feature map Then a spatial attention mechanism is applied to the convolutional feature map through a deep convolution, followed by a sigmoid activation to generate a spatial attention map , whose computational formula is expressed as:

[0050] ;

[0051] wherein, denotes a sigmoid activation function; denotes a deep convolutional layer with learnable weights, , denote the weights and bias of the deep convolutional layer, respectively;

[0052] The spatial attention map is multiplied element-wise with the convolutional feature map to obtain a spatial participation feature map , which is then refined into a refined feature map through a convolutional layer, and then a descriptor is obtained through a global average pooling (GAP) layer;

[0053] The channel attention map is then obtained through the following formula:

[0054] ;

[0055] wherein, , denote the weights of two fully connected (FC) layers, denote the real space, denote the number of channels of the image, denote the compression ratio (or scaling relationship); , denote the bias of the two fully connected layers, denote the ReLU activation function, applied after the first fully connected layer;

[0056] Executing channel attention map With the element-wise multiplication of the refined feature map to achieve scaled refined features , so as to obtain a channel-by-channel enhanced feature map , and using a convolutional layer to process the output feature map obtained by the feature enhancement and selection unit to obtain the output feature map which is the result of the composite feature enhancement and selection mechanism of the feature enhancement and selection unit;

[0057] Step 4, the spatial features and frequency features obtained in step 2 are processed by separate but interconnected paths by a dynamic feature fusion encoder (DFFE);

[0058] wherein the frequency features are transformed into the frequency domain by fast Fourier transform to obtain frequency encoded features , the calculation formula of which is expressed as:

[0059] ;

[0060] The frequency encoded features are substituted into the following calculation formula to obtain a set of learning weights :

[0061] ;

[0062] wherein, represents a global average pooling operation, represents a fully connected network (Fully Connected Network), represents a Softmax function;

[0063] An adaptive convolution block is provided in the dynamic feature fusion encoder, which is composed of complex convolution and is used to process the real component and the imaginary component of the complex feature to capture the complete spectrum of feature dynamics , the calculation formula of which is expressed as:

[0064] ;

[0065] wherein, represents the number of convolution kernels, represents the index of the convolution kernel; represents the weight corresponding to the th convolution kernel; represents the a complex-valued convolution;

[0066] Meanwhile, the spatial feature The concise processing is performed through the convolution cascade, and then the gating operation and subsequent fusion are performed, and the calculation formula is expressed as:

[0067]

[0068] wherein, represents the enhanced spatial feature; represents a composite function expressed in a simplified manner; represents the last convolution layer before the fusion of the spatial feature and the frequency feature; represents two consecutive convolution layers; represents an element-wise multiplication; represents the spatial feature processed by a single convolution branch;

[0069] The fusion enhanced feature is obtained through the following calculation formula to re-integrate the spatial information:

[0070]

[0071] wherein, represents an inverse fast Fourier transform;

[0072] Finally, the complete spectrum is merged with the fusion enhanced feature to obtain a feature set , and the feature set is refined through a convolution layer with ReLU activation to obtain a super-resolution image output by the dynamic feature fusion encoder .

[0073] Through the above scheme, the application can construct a dynamic hybrid spectrum synthesis network (DHS2Net), which utilizes the spectral characteristics of the image to cope with the challenge of reconstructing a high-resolution image from a low-resolution image. The application introduces a dynamic feature fusion encoder (DFFE) and a feature enhancement selection unit (FESU), which can dynamically adjust the processing strategy of the spatial domain and the frequency domain features according to the complexity of the image content, thereby more finely reconstructing the image details. When performing complex-valued processing, complex-valued convolution and complex-valued activation functions are used, which allows the network to simultaneously process the amplitude and phase information of the image, which is particularly important for preserving high-frequency details, and can significantly improve the visual quality and realism of the image.

[0074] The following shows a specific embodiment of the application.

[0075] ​​In step 4 above, the proposed Dynamic Feature Fusion Encoder (DFFE) architecture innovatively uses the ComplexFractiReLU activation function and strategic feature fusion, which is expected to establish a general foundation for complex feature representations in deep learning architectures. See also Figure 3 The ComplexFractiReLU (CFReLU) activation function is a novel component in the field of super-resolution neural networks. It provides an innovative approach by integrating fractal theory into complex-valued neural computation; this function dynamically adjusts the activation slope based on the estimated fractal dimension of the input data, a concept not previously applied in activation functions. The overall operation of CFReLU can be expressed by the following formula:

[0076] ;

[0077] ;

[0078] in, Indicates the use of radius Number of unique boxes A polynomial fit on a double logarithmic plot is used to calculate a function of fractal dimension to cover data points in the complex plane; Represents the complex input of the function. , These represent the real and imaginary parts of the complex number input, respectively. Indicates polynomial fitting; Indicates radius; Indicated by radius The number of frames covering data points on the complex plane when the scale is used; This represents the CFReLU activation function. The CFReLU activation function described above can adjust its behavior according to the complexity of the input, thus achieving a more nuanced and flexible activation environment, which is particularly advantageous for processing features in super-resolution tasks. See also... Figure 4 The provided experimental data offers a compelling visualization of the CFReLU activation function:

[0079] a. Figure 4 The left half of the diagram illustrates the relationship between fractal dimension and frequency, where different frequencies exhibit different fractal dimensions, indicating that the complexity of the feature varies with frequency.

[0080] b. Figure 4 The right half of the diagram illustrates the effect of fractal dimension on the slope of the activation function; it is observed that a higher fractal dimension corresponds to a steeper slope, meaning that the activation of complex features is more pronounced. This adaptability is expected to be particularly advantageous for super-resolution models, as it allows for differential amplification of features according to their inherent complexity, thereby enabling more detailed and accurate reconstruction of high-resolution images from low-resolution inputs.

[0081] The technical effects of the present application are demonstrated by experimental data below.

[0082] Here, the DHS2Net model of the present application is compared with existing SISR methods, including: VDSR, DRCN, FSRCNN, EDSR, IMDN, CARN, RCAN, HAN, SwinIR.

[0083] The results of the objective evaluation using magnification factors of ×2, ×3 and ×4 are recorded in Table 1. Figure 5 The data presented in Table 1 reflect a comprehensive evaluation of the super-resolution capabilities of the DHS2Net model. Figure 6 7 The ability of DHS2Net to reconstruct fine details in patches selected from the BSD100 dataset is visually demonstrated, with the performance of the model being particularly evident in complex textures and patterns, very close to ground truth, indicating its fine-grained enhancement capabilities. Figure 5 Quantitative correlations are provided, in which the DHS2Net model exhibits excellent performance, particularly in terms of ×2 and ×3 magnification factors; notably, at the ×3 scale, the DHS2Net model outperforms other methods on the Urban100 dataset, highlighting its ability to reconstruct complex urban scenes. These indicators not only represent the quality of resolution enhancement of the DHS2Net model, but also its consistency across various image contents and scales.

[0084] The present application also discloses an electronic device comprising a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the above-mentioned method by the processor.

[0085] The present application also discloses a computer-readable storage medium, the storage medium stores a computer program; the computer program, when executed in a computer, causes the computer to execute the above-mentioned method. The storage medium includes various storage medium that can store program code, such as U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0086] The above embodiments and drawings do not limit the product form and style of the present application, and any appropriate changes or modifications made by those skilled in the art shall be considered as not departing from the scope of the present application.​

Claims

1. A dual-domain learning method for single image super-resolution based on dynamic spectrum hybrid network, characterized in that Comprise: Step 1: In the decomposition stage, for a given input image... Each through step size for , convolutional layers , To extract low-frequency features High-frequency characteristics Its calculation formula is expressed as ； Step 2, low frequency feature after decomposition By one cascaded residual dense block Spatial feature is obtained , high frequency feature By one cascaded complex block Frequency feature is obtained Its calculation formula is expressed as: ； Step 3, spatial features processed in Step 2 , frequency features enhanced and fused by the feature enhancement and selection unit, respectively, in detail: In the feature enhancement and selection unit, first spatial features , frequency features are obtained through convolution layer processing to obtain a convolution feature map , and then a deep convolution is applied to the convolution feature map through a spatial attention mechanism, and a sigmoid activation is performed to generate a spatial attention map , and the calculation formula is expressed as: ； wherein, denotes a sigmoid activation function; denotes a deep convolutional layer with learnable weights, , denote weights, biases of the deep convolutional layer, respectively. Performing spatial attention map with element-wise multiplication of the convolutional feature map spatial participation feature map and refining it into a refined feature map by a convolutional layer and then obtaining a descriptor by a global average pooling (GAP) layer ; The channel attention map is obtained by the formula wherein , , denote the weights of two fully connected layers, denote the real space, denote the number of channels of the image, denote the compression ratio, , denote the bias of two fully connected layers, denote the ReLU activation function; Performing channel attention map with refined feature map element-wise multiplication to achieve scaled refined feature , thus obtaining a channel-wise enhanced feature map , and using a convolutional layer processing to obtain an output feature map that is the final output of the feature enhancement and selection unit , which is the result of the composite feature enhancement and selection mechanism of the feature enhancement and selection unit ; Step 4, fusing the dynamic features by a dynamic feature fusion encoder for the spatial features obtained at step 2 and frequency features are processed through separate but interconnected paths: Frequency characteristics by fast fourier transform transformed into the frequency domain to obtain frequency encoding characteristics , the calculation formula of which is expressed as ; the frequency encoding characteristics are substituted into the formula to obtain a set of learning weights , wherein represents a global average pooling operation, represents a fully connected network, represents a Softmax function; An adaptive convolution block is arranged in a dynamic feature fusion encoder which consists of complex-valued convolutions for processing real and imaginary components of complex features to capture the full spectrum of feature dynamics whose computational expression is given by where denotes the number of convolution kernels, denotes the index of a convolution kernel, denotes the weight corresponding to the th convolution kernel, denotes the th complex-valued convolution;​ At the same time, spatial features The concise processing is performed through the convolution cascade, and then the gating operation and subsequent fusion are performed, and the calculation formula is expressed as: ； wherein, denotes the enhanced spatial feature; denotes a composite function for simplified expression; denotes the last convolutional layer before the spatial feature and the frequency feature are fused; denotes two consecutive convolutional layers; denotes element-wise multiplication; denotes the spatial feature processed by a single convolutional branch. The fusion enhanced features are obtained by the formula to re-integrate the spatial information, wherein denotes the inverse fast Fourier transform;​ Finally, the complete spectrum with fusion enhanced features merging to get the feature set and refining it through a convolutional layer with ReLU activation to get the super-resolution image output of the dynamic feature fusion encoder .

2. The dual-domain learning method for single image super-resolution based on dynamic spectrum hybrid network according to claim 1, wherein In step 4, a ComplexFractiReLU activation function and strategic feature fusion are used in the architecture of the dynamic feature fusion encoder, and the overall operation of ComplexFractiReLU is expressed by the following calculation formula: ； ； wherein, represents using radius represents the number of boxes a polynomial fit on a log-log plot to calculate the fractal dimension of the function to cover the data points in the complex plane; represents the complex input of the function, , represents the real part, the imaginary part of the complex input, respectively; represents a polynomial fit; represents the radius; represents the number of boxes covering the data points on the complex plane when scaled by the radius ; represents a CFReLU activation function.

3. An electronic device, characterized in that... The application comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor to perform the dual-domain learning method for single image super-resolution based on the dynamic spectrum hybrid network according to claim 1 or 2.

4. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer program causes the computer to perform the dual-domain learning method for single image super-resolution based on the dynamic spectrum hybrid network according to claim 1 or 2.

Images