Double-domain learning method for single-image super-resolution based on dynamic spectrum hybrid network, electronic equipment and storage medium

Abstract

The invention discloses a double-domain learning method for single-image super-resolution based on a dynamic spectrum hybrid network, an electronic device and a storage medium, which effectively integrate spatial domain and frequency domain features to improve the quality and applicability of image super-resolution, and comprises the following steps: step 1, in a decomposition stage, carrying out decomposition; low-frequency features and high-frequency features of a given input image are extracted through convolutional layers with the step lengths # imgabs0 # being # imgabs1 # and # imgabs2 #; 2, after low-frequency features pass through # imgabs 3 # cascaded residual dense blocks, spatial features are obtained, and after high-frequency features pass through # imgabs 4 # cascaded complex value blocks, frequency features are obtained; 3, the spatial features and the frequency features are enhanced and fused by a feature enhancement and selection unit; and step 4, processing the spatial features and the frequency features by the dynamic feature fusion encoder through independent but interconnected paths, and finally outputting a super-resolution image output by the dynamic feature fusion encoder.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a dual-domain learning method, electronic device and storage medium for single image super-resolution based on a dynamic spectrum mixing network. Background Art

[0002] Image Super-Resolution (SR) technology is used to reconstruct low-resolution images into high-resolution versions. In the field of computer vision, this technology is mainly used in security monitoring, medical imaging, satellite imaging and other fields, and places high demands on image detail restoration.

[0003] Although the development of deep learning in recent years has significantly improved the effect of super-resolution technology, the existing technology mainly adopts a single domain (usually spatial domain) processing method, which leads to poor results when processing frequency-related image characteristics. For example, the traditional convolutional neural network (CNN)-based methods mainly focus on the explicit details of local pixels, and they often ignore the frequency characteristics of the image, which is a key factor affecting the overall quality of the image. As a result, although they can achieve good restoration effects on spatial details, they are not satisfactory in retaining the high-frequency characteristics of the image, which easily leads to distortion or over-smoothing of the details of the reconstructed image, especially in application scenarios where subtle texture and edge information need to be expressed in the image (such as medical imaging, satellite images). In addition, some existing methods try to make up for the shortcomings of spatial domain methods by introducing frequency domain processing, such as the Fourier transform method, but these methods usually lack a dynamic adjustment mechanism and cannot adjust the processing weights of frequency and spatial features according to the specific needs of the image content, resulting in the inability to fully utilize the complementarity between the two characteristics. This static feature processing strategy limits the adaptability and effect of super-resolution technology in a changing application environment.

[0004] In summary, existing technologies still face significant challenges in dynamically and effectively integrating spatial and frequency domain characteristics in the process of improving image super-resolution, especially when processing images with complex textures and high-frequency details. Summary of the invention

[0005] The purpose of the present invention is to provide a dual-domain learning method, electronic device and storage medium for single image super-resolution based on a dynamic spectral mixing network, to solve the problems existing in the prior art, and to effectively integrate spatial domain and frequency domain features to improve the quality and applicability of image super-resolution.

[0006] In order to achieve the above object, the solution of the present invention is: A dual-domain learning method for single image super-resolution based on a dynamic spectral mixture network, comprising: Step 1: In the decomposition stage, for a given input image , respectively, by step size for , The convolutional layer , To extract low frequency features , high frequency features , and its calculation formula is expressed as ; Step 2: After decomposition, low-frequency features pass Cascaded residual dense blocks Then obtain the spatial features , high frequency features pass Cascaded complex valued blocks Then get the frequency characteristics , and its calculation formula is expressed as: ; Step 3: Spatial features after processing in step 2 , frequency characteristics The enhancement and fusion are performed by feature enhancement and selection units respectively, specifically: In the feature enhancement and selection unit, first the spatial features , frequency characteristics After the convolution layer processing, the convolution feature map is obtained , and then the spatial attention mechanism is used in the convolution feature map Apply deep convolution on it and then perform sigmoid activation to generate a spatial attention map , and its calculation formula is expressed as: ; in, Represents the sigmoid activation function; represents a deep convolutional layer with learnable weights, , Represent the weight and bias of the deep convolutional layer respectively; Performing Spatial Attention Map With convolutional feature map Element-wise multiplication of obtains the spatial participation feature map , and refine it into a refined feature map through a convolutional layer , and then pass through the global average pooling (GAP) layer to obtain the descriptor ; By formula Get channel attention map ,in , represents the weights of the two fully connected layers, represents the real number space, Indicates the number of channels of the image, represents the compression ratio, , represents the deviation of two fully connected layers, ReLU activation function. Execute channel attention map With refined feature map Element-wise multiplication of to achieve scaling and refinement of features , thereby obtaining a channel-by-channel enhanced feature map , and use the convolutional layer to obtain the output feature map of the feature enhancement and selection unit final output , the output feature map It is the result of a composite feature enhancement and selection mechanism of feature enhancement and selection units; Step 4: The dynamic feature fusion encoder is used to fusion the spatial features obtained in step 2 and frequency characteristics Processed via separate but interconnected paths: Frequency characteristics Fast Fourier Transform Transform to frequency domain to obtain frequency encoding features , and its calculation formula is expressed as ; The frequency encoding feature Substitute into the formula Get a set of learning weights ,in represents the global average pooling operation, represents a fully connected network. Represents the Softmax function; The dynamic feature fusion encoder is equipped with an adaptive convolution block , which consists of complex-valued convolutions to process the real components of complex features and imaginary components , to capture the full spectrum of characteristic dynamics , and its calculation formula is expressed as ,in represents the number of convolution kernels, represents the index of the convolution kernel, Indicates The weights corresponding to the convolution kernels are Indicates complex-valued convolution; At the same time, the spatial characteristics The concise processing is performed through convolution cascade, followed by gating operation and subsequent fusion. The calculation formula is expressed as: ; in, represents the enhanced spatial features; A composite function that represents a simplified expression; Represents the last convolutional layer before the fusion of spatial features and frequency features; Represents two consecutive convolutional layers; represents element-wise multiplication; Represents the spatial features processed by a single convolution branch; By formula Get fusion enhancement features To reintegrate spatial information, represents inverse fast Fourier transform; Finally, the complete spectrum Enhanced features with fusion Merge to get feature set , and refine it through a convolutional layer with ReLU activation to obtain the super-resolution image output by the dynamic feature fusion encoder .

[0007] Preferably, in step 4, the ComplexFractiReLU activation function and strategic feature fusion are used in the architecture of the dynamic feature fusion encoder, and the overall operation of ComplexFractiReLU is expressed by the following calculation formula: ; ; in, Indicates the radius used With unique box quantity A function to calculate the fractal dimension by fitting a polynomial on a double logarithmic plot of to cover the data points in the complex plane; represents the complex input of the function, , Respectively represent the real and imaginary parts of the complex input; represents a polynomial fit; represents the radius; Indicated by radius The number of boxes covering the data points on the complex plane when is the scale; Represents the CFReLU activation function.

[0008] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor to perform the dual-domain learning method for single image super-resolution based on a dynamic spectrum mixing network.

[0009] A computer-readable storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the dual-domain learning method for single image super-resolution based on a dynamic spectrum mixing network.

[0010] After adopting the above technical solution, the present invention has the following technical effects: The present invention can construct a dynamic hybrid spectral synthesis network (DHS2Net), which can cope with the challenge of reconstructing low-resolution images into high-resolution images by utilizing the spectral characteristics of images; the present invention can dynamically adjust the processing strategies of spatial domain and frequency domain features according to the complexity of image content by introducing a dynamic feature fusion encoder (DFFE) and a feature enhancement selection unit (FESU), so as to reconstruct image details more finely; wherein, when performing complex value processing, complex valued convolution and complex valued activation functions are used, allowing the network to simultaneously process the amplitude and phase information of the image, which is particularly important for maintaining high-frequency details, and can significantly improve the visual texture and realism of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a system architecture diagram of a specific embodiment of the present invention; Figure 2 It is a dynamic feature fusion encoder architecture diagram of a specific embodiment of the present invention; Figure 3 This is a diagram of the CFReLU activation function architecture of a specific embodiment of the present invention; Figure 4 It is a visualization diagram of the correlation between signal frequency and fractal dimension and the slope of CFReLU at different fractal dimensions according to a specific embodiment of the present invention; Figure 5 It is a comparison diagram of experimental performance of a specific embodiment of the present invention; Figure 6 Outputting detailed area comparison graphs for different models of specific embodiments of the present invention; Figure 7 A high-frequency region comparison diagram is output for different models of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0013] refer to Figure 1-7As shown, the present invention discloses a dual-domain learning method for single image super-resolution based on a dynamic spectrum mixing network, comprising: Step 1: In the decomposition stage, for a given input image , respectively, by step length for , The convolutional layer , To extract low frequency features , high frequency features , and its calculation formula is expressed as: ; Step 2: After decomposition, low-frequency features pass Cascaded residual dense blocks Then obtain the spatial features , high frequency features pass Cascaded complex valued blocks Then get the frequency characteristics , and its calculation formula is expressed as: ; Step 3: Spatial features after processing in step 2 , frequency characteristics The feature enhancement and selection unit (FESU) performs enhancement and fusion respectively, retaining important features while discarding noise or less useful information to produce clearer and more accurate high-resolution images. The feature enhancement and selection unit acts as an adaptive filter to dynamically adjust the spatial features. , frequency characteristics Prioritize key information paths, specifically: In the feature enhancement and selection unit, first the spatial features , frequency characteristics After the convolution layer processing, the convolution feature map is obtained , and then the spatial attention mechanism is used in the convolution feature map Apply deep convolution on it and then perform sigmoid activation to generate a spatial attention map , and its calculation formula is expressed as: ; in, Represents the sigmoid activation function; represents a deep convolutional layer with learnable weights, , Represent the weight and bias of the deep convolutional layer respectively; Performing Spatial Attention Map With convolutional feature map Element-wise multiplication of obtains the spatial participation feature map , and refine it into a refined feature map through a convolutional layer , and then pass through the global average pooling (GAP) layer to obtain the descriptor ; Then use the following formula to get the channel attention map : ; in, , represents the weights of two fully connected (FC) layers, represents the real number space, Indicates the number of channels of the image, Indicates the compression ratio (or scaling relationship); , represents the deviation of two fully connected layers, Represents the ReLU activation function, which is applied after the first fully connected layer; Execute channel attention map With refined feature map Element-wise multiplication of to achieve scaling and refinement of features , thereby obtaining a channel-by-channel enhanced feature map , and use the convolutional layer to obtain the output feature map of the feature enhancement and selection unit final output , the output feature map It is the result of a composite feature enhancement and selection mechanism of feature enhancement and selection units; Step 4: The spatial features obtained in step 2 are obtained by the dynamic feature fusion encoder (DFFE) and frequency characteristics processed through separate but interconnected pathways; Among them, the frequency characteristics Fast Fourier Transform Transform to frequency domain to obtain frequency encoding features , and its calculation formula is expressed as: ; The frequency encoding feature Substitute the following calculation formula to obtain a set of learning weights : ; in, represents the global average pooling operation, represents a fully connected network (FullyConnected Network), Represents the Softmax function; The dynamic feature fusion encoder is equipped with an adaptive convolution block , which consists of complex-valued convolutions to process the real components of complex features and imaginary components , to capture the full spectrum of characteristic dynamics , and its calculation formula is expressed as: ; in, represents the number of convolution kernels, Represents the index of the convolution kernel; Indicates The weights corresponding to the convolution kernels; Indicates complex-valued convolution; At the same time, the spatial characteristics The concise processing is performed through convolution cascade, followed by gating operation and subsequent fusion. The calculation formula is expressed as: ; in, represents the enhanced spatial features; A composite function that represents a simplified expression; Represents the last convolutional layer before the fusion of spatial features and frequency features; Represents two consecutive convolutional layers; represents element-wise multiplication; Represents the spatial features processed by a single convolution branch; The fusion enhancement feature is obtained by the following calculation formula To reintegrate spatial information: ; in, represents inverse fast Fourier transform; Finally, the complete spectrum Enhanced features with fusion Merge to get feature set , and refine it through a convolutional layer with ReLU activation to obtain the super-resolution image output by the dynamic feature fusion encoder .

[0014] Through the above scheme, the present invention can construct a dynamic hybrid spectral synthesis network (DHS2Net), which can cope with the challenge of reconstructing low-resolution images into high-resolution images by utilizing the spectral characteristics of the image; the present invention can dynamically adjust the processing strategies of spatial domain and frequency domain features according to the complexity of the image content by introducing a dynamic feature fusion encoder (DFFE) and a feature enhancement selection unit (FESU), so as to reconstruct the image details more finely; wherein, when performing complex value processing, complex valued convolution and complex valued activation functions are used, allowing the network to simultaneously process the amplitude and phase information of the image, which is particularly important for maintaining high-frequency details, and can significantly improve the visual texture and realism of the image.

[0015] Specific embodiments of the present invention are shown below.

[0016] In step 4 above, the proposed dynamic feature fusion encoder (DFFE) architecture innovatively uses the ComplexFractiReLU activation function and strategic feature fusion, which is expected to establish a general foundation for complex feature representation in deep learning architectures. Figure 3 The ComplexFractiReLU (CFReLU) activation function is a novel component in the field of super-resolution neural networks, providing an innovative approach by integrating fractal theory into complex-valued neural computation; the function dynamically adjusts the activation slope according to the estimated fractal dimension of the input data, a concept that has not been applied in activation functions before. The overall operation of CFReLU can be expressed by the following calculation formula: ; ; in, Indicates the radius used With unique box quantity A function to calculate the fractal dimension by fitting a polynomial on a double logarithmic plot of to cover the data points in the complex plane; represents the complex input of the function, , Respectively represent the real and imaginary parts of the complex input; represents a polynomial fit; represents the radius; Indicated by radius The number of boxes covering the data points on the complex plane when is the scale; Represents the CFReLU activation function. The above CFReLU activation function can adjust its behavior according to the complexity of the input, thereby achieving a more detailed and flexible activation environment, which is particularly beneficial for processing features in super-resolution tasks. Figure 4 , the experimental data provided provides a convincing visualization of the CFReLU activation function: a. Figure 4 The left half of the figure shows the relationship between fractal dimension and frequency, where different frequencies exhibit different fractal dimensions, indicating that the complexity of the feature varies with frequency.

[0017] b. Figure 4 The right half of illustrates the effect of fractal dimension on the slope of the activation function; it is observed that higher fractal dimensions correspond to steeper slopes, meaning that activations of complex features are more pronounced. This adaptability is expected to be particularly beneficial for super-resolution models, as it allows features to be differentially amplified based on their inherent complexity, leading to more detailed and accurate reconstruction of high-resolution images from low-resolution inputs.

[0018] The technical effect of the present invention is demonstrated by experimental data as follows.

[0019] Here, the DHS2Net model of the present invention is compared with existing SISR methods, including: VDSR, DRCN, FSRCNN, EDSR, IMDN, CARN, RCAN, HAN, and SwinIR.

[0020] The results of objective evaluation using magnifications of ×2, ×3, and ×4 are recorded in Figure 5 In , the data presented represent a comprehensive evaluation of the super-resolution capabilities of the DHS2Net model. Figure 6 , 7 The ability of DHS2Net to reconstruct fine details in patches selected from the BSD100 dataset is intuitively demonstrated. The model’s performance is especially evident in complex textures and patterns, which are very close to the ground truth, indicating its nuanced enhancement capability. Figure 5 Quantitative correlations are provided, where DHS2Net shows excellent performance, especially in terms of ×2 and ×3 magnification factors; it is worth noting that at ×3 scale, DHS2Net performs well on the Urban100 dataset, outperforming other methods and highlighting its ability to reconstruct complex urban scenes. These indicators not only indicate the resolution enhancement quality of DHS2Net, but also its consistency across various image contents and scales.

[0021] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0022] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0023] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A dual-domain learning method for single image super-resolution based on dynamic spectral mixture network, characterized in that include: Step 1: In the decomposition stage, for a given input image , respectively, by step size for , The convolutional layer , To extract low frequency features , high frequency features , and its calculation formula is expressed as ； Step 2: After decomposition, low-frequency features pass Cascaded residual dense blocks Then obtain the spatial features , high frequency features pass Cascaded complex valued blocks Then get the frequency characteristics , and its calculation formula is expressed as: ； Step 3: Spatial features after processing in step 2 , frequency characteristics The enhancement and fusion are performed by feature enhancement and selection units respectively, specifically: In the feature enhancement and selection unit, first the spatial features , frequency characteristics After the convolution layer processing, the convolution feature map is obtained , and then the spatial attention mechanism is used in the convolution feature map Apply deep convolution on it and then perform sigmoid activation to generate a spatial attention map , and its calculation formula is expressed as: ； in, Represents the sigmoid activation function; represents a deep convolutional layer with learnable weights, , Represent the weight and bias of the deep convolutional layer respectively; Performing Spatial Attention Map With convolutional feature map Element-wise multiplication of obtains the spatial participation feature map , and refine it into a refined feature map through a convolutional layer , and then pass through the global average pooling (GAP) layer to obtain the descriptor ; By formula Get channel attention map ,in , represents the weights of the two fully connected layers, represents the real number space, Indicates the number of channels of the image, represents the compression ratio, , represents the deviation of two fully connected layers, ReLU activation function. Execute channel attention map With refined feature map Element-wise multiplication of to achieve scaling and refinement of features , thereby obtaining a channel-by-channel enhanced feature map , and use the convolutional layer to obtain the output feature map of the feature enhancement and selection unit final output , the output feature map It is the result of a composite feature enhancement and selection mechanism of feature enhancement and selection units; Step 4: The dynamic feature fusion encoder is used to fusion the spatial features obtained in step 2 and frequency characteristics Processed via separate but interconnected paths: Frequency characteristics Fast Fourier Transform Transform to frequency domain to obtain frequency encoding features , and its calculation formula is expressed as ; The frequency encoding feature Substitute into the formula Get a set of learning weights ,in represents the global average pooling operation, represents a fully connected network. Represents the Softmax function; The dynamic feature fusion encoder is equipped with an adaptive convolution block , which consists of complex-valued convolutions to process the real components of complex features and imaginary components , to capture the full spectrum of characteristic dynamics , and its calculation formula is expressed as ,in represents the number of convolution kernels, represents the index of the convolution kernel, Indicates The weights corresponding to the convolution kernels are Indicates complex-valued convolution; At the same time, the spatial characteristics The concise processing is performed through convolution cascade, followed by gating operation and subsequent fusion. The calculation formula is expressed as: ； in, represents the enhanced spatial features; A composite function that represents a simplified expression; Represents the last convolutional layer before the fusion of spatial features and frequency features; Represents two consecutive convolutional layers; represents element-wise multiplication; Represents the spatial features processed by a single convolution branch; By formula Get fusion enhancement features To reintegrate spatial information, represents inverse fast Fourier transform; Finally, the complete spectrum Enhanced features with fusion Merge to get feature set , and refine it through a convolutional layer with ReLU activation to obtain the super-resolution image output by the dynamic feature fusion encoder .

2. The dual-domain learning method for single image super-resolution based on dynamic spectral mixing network as claimed in claim 1, characterized in that In step 4, the ComplexFractiReLU activation function and strategic feature fusion are used in the architecture of the dynamic feature fusion encoder, and the overall operation of ComplexFractiReLU is expressed by the following calculation formula: ； ； in, Indicates the radius used With unique box quantity A function to calculate the fractal dimension by fitting a polynomial on a double logarithmic plot of to cover the data points in the complex plane; represents the complex input of the function, , Respectively represent the real and imaginary parts of the complex input; represents a polynomial fit; represents the radius; Indicated by radius The number of boxes covering the data points on the complex plane when the scale is Represents the CFReLU activation function.

3. An electronic device, characterized in that It includes a processor, a memory and an application; the application is stored in the memory and is configured to be executed by the processor as claimed in claim 1 or 2, the dual-domain learning method for single image super-resolution based on a dynamic spectral mixing network.

4. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the dual-domain learning method for single image super-resolution based on a dynamic spectral mixing network as described in claim 1 or 2.

Images