Image super-resolution method based on frequency domain and spatial domain

Through the image super-resolution method based on frequency domain and airspace, the enhanced super-resolution generation adversarial network and wavelet transformation are used, combined with the attention mechanism, the problem of insufficient visual effects and accuracy of image super-resolution methods in the prior art is solved. , achieving higher resolution and better practical application effects, avoiding high costs and technical bottlenecks in hardware upgrades.

CN114463183BActive Publication Date: 2024-12-03CENT SOUTH UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210134149.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-14
Publication Date
2024-12-03
Estimated Expiration
2042-02-14

AI Technical Summary

Technical Problem

The existing deep learning-based image super-resolution methods have shortcomings in visual effects and accuracy, and are poor in actual application, and the method of obtaining higher resolution images through hardware upgrades is expensive and has technical bottlenecks.

Method used

Using the image super-resolution method based on frequency domain and airspace, the image super-resolution model is constructed by enhancing the super-resolution generation adversarial network combined with wavelet transformation and attention mechanism, and the image super-resolution model is constructed, using the generative adversarial network, attention mechanism and wavelet transformation from the frequency domain. Image features are extracted and fused at two angles and airspace, and the image super-resolution model is obtained.

Benefits of technology

It realizes higher image resolution, adds high-frequency information, structure information and texture information, improves the accuracy and visual effect of image super resolution, and has high reliability, good practical application effect, and stable and accurate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114463183B_ABST
    Figure CN114463183B_ABST
Patent Text Reader

Abstract

The present invention discloses an image super-resolution method based on frequency domain and spatial domain, including obtaining an original data set and processing it to obtain a training data set; constructing an image super-resolution original model from the frequency domain and spatial domain perspectives based on an enhanced super-resolution generative adversarial network, combined with wavelet transform and attention mechanism; using the training data set for training to obtain an image super-resolution model; and using the image super-resolution model to perform super-resolution processing on actual images. According to the different features of spatial domain images and frequency domain images, the present invention uses generative adversarial networks, attention mechanisms, wavelet transforms and other technologies to train an image super-resolution model, and performs deep feature extraction and feature reconstruction on spatial domain and frequency domain images respectively; therefore, the method of the present invention can make the super-resolution image have more high-frequency information, structural information and texture information, and has both accuracy and visual effects, high reliability, good practical application effects, and stable and precise.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of digital image processing, and in particular relates to an image super-resolution method based on frequency domain and spatial domain. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, electronic images have been widely used in people's production and life, bringing endless convenience to people's production and life. Today, when electronic images are widely used, how to obtain electronic images with higher resolution has become the focus of people's research.

[0003] At present, the traditional way to obtain electronic images with higher resolution is to upgrade the hardware, that is, to use more advanced sensor technology and optical manufacturing technology to obtain electronic images with higher resolution. However, due to the limitations of sensor technology and optical manufacturing technology, the method of obtaining higher quality and higher resolution images through hardware upgrades is not only costly, but also has technical bottlenecks.

[0004] Image super-resolution technology can overcome the limitation of the original image resolution under certain conditions and improve the image resolution. It has low implementation cost and good implementation effect. Therefore, image super-resolution technology has become the focus of current research.

[0005] In recent years, with the rapid development of deep learning, deep learning-based super-resolution methods have gradually replaced traditional interpolation-based and reconstruction-based super-resolution methods due to their excellent performance; there are now a large number of excellent image super-resolution models and methods.

[0006] However, current deep learning-based image super-resolution methods still have shortcomings in terms of visual effects and accuracy when applied in specific applications; moreover, current deep learning-based image super-resolution methods have only achieved certain results in test tasks, but the results are relatively poor in actual applications. Summary of the invention

[0007] The object of the present invention is to provide an image super-resolution method based on frequency domain and spatial domain which has high reliability, good practical application effect and is stable and accurate.

[0008] The image super-resolution method based on frequency domain and spatial domain provided by the present invention comprises the following steps:

[0009] S1. Obtain the original data set;

[0010] S2. Processing the original data set obtained in step S1 to construct a training data set;

[0011] S3. Based on the enhanced super-resolution generative adversarial network, combined with wavelet transform and attention mechanism, the original model of image super-resolution is constructed from the perspectives of frequency domain and spatial domain;

[0012] S4. Using the training data set obtained in step S2, the original image super-resolution model constructed in step S3 is trained to obtain an image super-resolution model;

[0013] S5. Using the image super-resolution model obtained in step S5, super-resolution processing is performed on the actual image to complete the super-resolution of the actual image.

[0014] Step S2 processes the original data set obtained in step S1 to construct a training data set, which specifically includes the following steps:

[0015] A. Set the resolution of high-resolution images and low-resolution images in the training dataset;

[0016] B. Segment the image in the original data set obtained in step S1 to obtain several sub-images as high-resolution images;

[0017] C. Downsampling the high-resolution image obtained in step B to obtain a corresponding low-resolution image;

[0018] D. Match the high-resolution image obtained in step B with the low-resolution image obtained in step C one by one to obtain a training data set.

[0019] The enhanced super-resolution generative adversarial network described in step S3 is combined with wavelet transform and attention mechanism to construct an image super-resolution original model from the perspectives of frequency domain and spatial domain, which specifically includes the following steps:

[0020] a. Construct a spatial super-resolution network to extract features, upsample and reconstruct the input low-resolution image, thereby obtaining the reconstructed spatial image features;

[0021] b. Construct a frequency domain super-resolution network to perform frequency domain decomposition, feature extraction, upsampling and reconstruction on the input low-resolution image, thereby obtaining reconstructed frequency domain image features;

[0022] c. Construct an attention fusion network to fuse the reconstructed spatial domain image features obtained in step a and the reconstructed frequency domain image features obtained in step b, and reconstruct a high-resolution image.

[0023] The step a of constructing a spatial super-resolution network specifically includes the following steps:

[0024] The spatial super-resolution network includes a spatial shallow feature extraction module, a spatial deep feature extraction module, a spatial upsampling module and a spatial feature reconstruction module which are connected in series in sequence;

[0025] The spatial shallow feature extraction module includes a convolution layer with a convolution kernel size of 3*3. The spatial shallow feature extraction module is used to extract shallow features from the input low-resolution image and upload the extracted shallow image features to the spatial deep feature extraction module.

[0026] The spatial deep feature extraction module includes 23 dense residual blocks and a convolution layer, and the convolution kernel size of the convolution layer is 3*3; the input shallow image features are sequentially passed through the dense residual block and the convolution layer to obtain the deep image features, and the intermediate features are retained; the spatial deep feature extraction module is used to extract the deep features of the image and upload the extracted deep image features to the spatial upsampling module;

[0027] The spatial upsampling module includes an upsampling layer, a convolution layer and a relu activation function layer connected in series. The upsampling layer adopts the nearest neighbor sampling mode, and the convolution kernel size of the convolution layer is 3*3. The input deep image features are passed through the upsampling layer to obtain the spatial image features of the target resolution, and then the convolution operation is performed through the convolution layer. Finally, the relu function is activated to complete the upsampling process, and the final spatial image features are uploaded to the spatial feature reconstruction module.

[0028] The spatial feature reconstruction module includes a first convolution layer, a relu activation function layer and a second convolution layer connected in series. The convolution kernel size of the first convolution layer is 3*3, and the convolution kernel size of the second convolution layer is 3*3. The input spatial image features are convolved through the first convolution layer, activated through the relu function, and then convolved through the second convolution layer to obtain the final reconstructed spatial image features.

[0029] The step b of constructing a frequency domain super-resolution network specifically includes the following steps:

[0030] The frequency domain super-resolution network includes a frequency domain decomposition module, a frequency domain shallow feature extraction module, a frequency domain deep feature extraction module, a frequency domain upsampling module and a frequency domain feature reconstruction module which are connected in series in sequence;

[0031] The frequency domain decomposition module uses a stable wavelet transform algorithm to convert the input low-resolution image into four frequency domain images: LL image (horizontal low-frequency image), LH image (horizontal high-frequency image), HL image (vertical high-frequency image) and HH image (diagonal high-frequency image); and uploads the obtained four frequency domain images to the frequency domain shallow feature extraction module;

[0032] The frequency domain shallow feature extraction module includes a convolution layer with a convolution kernel size of 3*3; the frequency domain shallow feature extraction module is used to extract shallow features of the input frequency domain image and upload the extracted frequency domain shallow image features to the frequency domain deep feature extraction module;

[0033] The frequency domain deep feature extraction module includes the first convolution layer, the first dense residual block layer, the second convolution layer, the second dense residual block layer, the third convolution layer, the third dense residual block layer, the fourth convolution layer, the fourth dense residual block layer and the fifth convolution layer, which are connected in series in sequence; the convolution kernel size of each convolution layer is 3*3; the input frequency domain shallow image features are connected with the intermediate features of the spatial domain super-resolution network and input into the first convolution layer together, and the output of the first convolution layer passes through the first dense residual block layer; the output of the first dense residual block layer is connected with the intermediate features of the spatial domain super-resolution network and input into the first convolution layer together. The output of the second convolutional layer is then passed through the second dense residual block layer; the output of the second dense residual block layer is connected with the intermediate features of the spatial super-resolution network and then input into the third convolutional layer together, and the output of the third convolutional layer is then passed through the third dense residual block layer; the output of the third dense residual block layer is connected with the intermediate features of the spatial super-resolution network and then input into the fourth convolutional layer together, and the output of the fourth convolutional layer is then passed through the fourth dense residual block layer; the output of the fourth dense residual block layer is then processed through the fifth convolutional layer to obtain the deep features in the frequency domain and upload it to the frequency domain upsampling module;

[0034] The frequency domain upsampling module includes an upsampling layer, a convolution layer and a relu activation function layer connected in series. The upsampling layer adopts the nearest neighbor sampling mode, and the convolution kernel size of the convolution layer is 3*3. The input frequency domain deep image features are passed through the upsampling layer to obtain the frequency domain image features of the target resolution, and then the convolution operation is performed through the convolution layer, and finally the relu function is activated to complete the upsampling process, and the final frequency domain image features are uploaded to the frequency domain feature reconstruction module.

[0035] The frequency domain feature reconstruction module includes a first convolution layer, a relu activation function layer and a second convolution layer connected in series. The convolution kernel size of the first convolution layer is 3*3, and the convolution kernel size of the second convolution layer is 3*3. The input frequency domain image features are convolved through the first convolution layer, activated through the relu function, and then convolved through the second convolution layer to obtain the final reconstructed frequency domain image features.

[0036] The construction of the attention fusion network described in step c specifically includes the following steps:

[0037] (1) The initial fusion feature F is calculated using the following formula shallow :

[0038] F shallow =HRRDB (Attention(F s ,F freq )×F freq +F s )

[0039] Where F s To reconstruct the spatial domain image feature matrix; F freq is the reconstructed frequency domain image feature matrix; H RRDB () is the processing process corresponding to the dense residual module; Attention (F s ,F freq ) is the similarity calculation matrix, and Attention(F s ,F freq )=softmax(F s ×(F freq ) T ), softmax() is the softmax function operation;

[0040] (2) The deep fusion feature F is calculated using the following formula deep :

[0041] F deep =H RRDB (Attention(F shallow ,F freq )×F freq +F shallow )

[0042] Where F shallow is the preliminary fusion feature calculated in step (1);

[0043] (3) The deep fusion feature F obtained in step (2) deep The input is sent to two convolutional layers to obtain a high-resolution output image; the convolution kernels of the two convolutional layers are both 3*3.

[0044] The training described in step S4 is specifically to use the following formula as the loss function L during training: G :

[0045] L G =L percep +λL GAN +ηL1

[0046] Where L percep is the perceived loss; L GAN is the adversarial loss; L1 is the content loss; λ and η are weight parameters.

[0047] The image super-resolution method based on frequency domain and spatial domain provided by the present invention adopts generative adversarial network, attention mechanism, wavelet transform and other technologies according to the different characteristics of spatial domain images and frequency domain images, trains an image super-resolution model, and performs deep feature extraction and feature reconstruction on spatial domain and frequency domain images respectively; therefore, the method of the present invention can make the super-resolution image have more high-frequency information, structural information and texture information, and has both accuracy and visual effect, high reliability, good practical application effect and stable and precise. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0049] Figure 2 The figure is a schematic diagram of the structure of the original model of image super-resolution of the method of the present invention. DETAILED DESCRIPTION

[0050] like Figure 1 The method flow chart of the method of the present invention is shown as follows: the image super-resolution method based on frequency domain and spatial domain provided by the present invention comprises the following steps:

[0051] S1. Obtain the original data set;

[0052] S2. Process the original data set obtained in step S1 to construct a training data set; specifically, the steps include:

[0053] A. Set the resolution of high-resolution images and low-resolution images in the training dataset;

[0054] B. Segment the image in the original data set obtained in step S1 to obtain several sub-images as high-resolution images;

[0055] C. Downsampling the high-resolution image obtained in step B to obtain a corresponding low-resolution image;

[0056] D. Match the high-resolution image obtained in step B with the low-resolution image obtained in step C one by one to obtain a training data set;

[0057] S3. Based on the enhanced super-resolution generative adversarial network, combined with wavelet transform and attention mechanism, the original model of image super-resolution is constructed from the perspectives of frequency domain and spatial domain;

[0058] The constructed image super-resolution original model is based on the ESRGAN (Enhanced Super-ResolutionGenerative Adversarial Networks) network, which introduces wavelet transform and attention mechanism, performs super-resolution training from the perspectives of frequency domain and spatial domain, and extracts and selects features through the attention fusion module, thereby improving the accuracy and quality of image super-resolution; the constructed image super-resolution original model is as follows Figure 2 As shown;

[0059] The specific implementation includes the following steps:

[0060] a. Construct a spatial domain super-resolution network to extract features, upsample and reconstruct the input low-resolution image, so as to obtain the reconstructed spatial domain image features; specifically, the following steps are included:

[0061] The spatial super-resolution network includes a spatial shallow feature extraction module, a spatial deep feature extraction module, a spatial upsampling module and a spatial feature reconstruction module which are connected in series in sequence;

[0062] The spatial shallow feature extraction module includes a convolution layer with a convolution kernel size of 3*3. The spatial shallow feature extraction module is used to extract shallow features from the input low-resolution image and upload the extracted shallow image features to the spatial deep feature extraction module.

[0063] The spatial deep feature extraction module includes 23 dense residual blocks and a convolution layer, and the convolution kernel size of the convolution layer is 3*3; the input shallow image features are sequentially passed through the dense residual block and the convolution layer to obtain the deep image features, and the intermediate features are retained; the spatial deep feature extraction module is used to extract the deep features of the image and upload the extracted deep image features to the spatial upsampling module;

[0064] The spatial upsampling module includes an upsampling layer, a convolution layer and a relu activation function layer connected in series. The upsampling layer adopts the nearest neighbor sampling mode, and the convolution kernel size of the convolution layer is 3*3. The input deep image features are passed through the upsampling layer to obtain the spatial image features of the target resolution, and then the convolution operation is performed through the convolution layer. Finally, the relu function is activated to complete the upsampling process, and the final spatial image features are uploaded to the spatial feature reconstruction module.

[0065] The spatial feature reconstruction module includes a first convolution layer, a relu activation function layer and a second convolution layer connected in series. The convolution kernel size of the first convolution layer is 3*3, and the convolution kernel size of the second convolution layer is 3*3. The input spatial image features are convolved through the first convolution layer, activated through the relu function, and then convolved through the second convolution layer to obtain the final reconstructed spatial image features.

[0066] b. Construct a frequency domain super-resolution network to perform frequency domain decomposition, feature extraction, upsampling and reconstruction on the input low-resolution image, thereby obtaining reconstructed frequency domain image features; specifically, the following steps are included:

[0067] The frequency domain super-resolution network includes a frequency domain decomposition module, a frequency domain shallow feature extraction module, a frequency domain deep feature extraction module, a frequency domain upsampling module and a frequency domain feature reconstruction module which are connected in series in sequence;

[0068] The frequency domain decomposition module uses the stable wavelet transform algorithm (SWT transform) to convert the input low-resolution image into four frequency domain images: LL image (horizontal low-frequency image), LH image (horizontal high-frequency image), HL image (vertical high-frequency image) and HH image (diagonal high-frequency image); and uploads the obtained four frequency domain images to the frequency domain shallow feature extraction module;

[0069] The frequency domain shallow feature extraction module includes a convolution layer with a convolution kernel size of 3*3; the frequency domain shallow feature extraction module is used to extract shallow features of the input frequency domain image and upload the extracted frequency domain shallow image features to the frequency domain deep feature extraction module;

[0070] The frequency domain deep feature extraction module uses the dense residual block in the ESRGAN model as the basic module, which specifically includes the first convolution layer, the first dense residual block layer, the second convolution layer, the second dense residual block layer, the third convolution layer, the third dense residual block layer, the fourth convolution layer, the fourth dense residual block layer and the fifth convolution layer connected in series in sequence; the convolution kernel size of each convolution layer is 3*3; the input frequency domain shallow image features are connected with the intermediate features of the spatial domain super-resolution network, and then input into the first convolution layer together, and the output of the first convolution layer passes through the first dense residual block layer; the output of the first dense residual block layer is connected with the spatial domain super-resolution network. After the intermediate features of the spatial super-resolution network are connected, they are input into the second convolutional layer together, and the output of the second convolutional layer passes through the second dense residual block layer; after the output of the second dense residual block layer is connected with the intermediate features of the spatial super-resolution network, they are input into the third convolutional layer together, and the output of the third convolutional layer passes through the third dense residual block layer; after the output of the third dense residual block layer is connected with the intermediate features of the spatial super-resolution network, they are input into the fourth convolutional layer together, and the output of the fourth convolutional layer passes through the fourth dense residual block layer; the output of the fourth dense residual block layer is processed by the fifth convolutional layer to obtain the deep features in the frequency domain, and uploaded to the frequency domain upsampling module;

[0071] The frequency domain upsampling module includes an upsampling layer, a convolution layer and a relu activation function layer connected in series. The upsampling layer adopts the nearest neighbor sampling mode, and the convolution kernel size of the convolution layer is 3*3. The input frequency domain deep image features are passed through the upsampling layer to obtain the frequency domain image features of the target resolution, and then the convolution operation is performed through the convolution layer, and finally the relu function is activated to complete the upsampling process, and the final frequency domain image features are uploaded to the frequency domain feature reconstruction module.

[0072] The frequency domain feature reconstruction module includes a first convolution layer, a relu activation function layer and a second convolution layer connected in series. The convolution kernel size of the first convolution layer is 3*3, and the convolution kernel size of the second convolution layer is 3*3. The input frequency domain image features are convolved by the first convolution layer, activated by the relu function, and then convolved by the second convolution layer to obtain the final reconstructed frequency domain image features.

[0073] c. Construct an attention fusion network to fuse the reconstructed spatial domain image features obtained in step a and the reconstructed frequency domain image features obtained in step b, and reconstruct a high-resolution image; specifically, the following steps are included:

[0074] (1) The initial fusion feature F is calculated using the following formula shallow :

[0075] F shallow =H RRDB (Attention(F s ,Ffreq )×F freq +F s )

[0076] Where F s To reconstruct the spatial domain image feature matrix; F freq is the reconstructed frequency domain image feature matrix; H RRDB () is the processing process corresponding to the dense residual module; Attention (F s ,F freq ) is the similarity calculation matrix, and Attention(F s ,F freq )=softmax(F s ×(F freq ) T ), softmax() is the softmax function operation;

[0077] (2) The deep fusion feature F is calculated using the following formula deep :

[0078] F deep =H RRDB (Attention(F shallow ,F freq )×F freq +F shallow )

[0079] Where F shallow is the preliminary fusion feature calculated in step (1);

[0080] (3) The deep fusion feature F obtained in step (2) deep Input into two convolutional layers to obtain the output high-resolution image; the convolution kernels of the two convolutional layers are both 3*3;

[0081] S4. Using the training data set obtained in step S2, the original image super-resolution model constructed in step S3 is trained to obtain an image super-resolution model;

[0082] During training, the following formula is used as the loss function L G :

[0083] L G =L percep +λL GAN +ηL1

[0084] Where L percep is the perceived loss; L GAN is the adversarial loss; L1 is the content loss; λ and η are both weight parameters;

[0085] S5. Using the image super-resolution model obtained in step S5, super-resolution processing is performed on the actual image to complete the super-resolution of the actual image.

Claims

1. An image super-resolution method based on frequency domain and spatial domain comprises the following steps: S1. Obtain the original data set; S2. Processing the original data set obtained in step S1 to construct a training data set; S3. Based on the enhanced super-resolution generative adversarial network, combined with wavelet transform and attention mechanism, the original model of image super-resolution is constructed from the perspectives of frequency domain and spatial domain; specifically, the steps are as follows: a. Construct a spatial super-resolution network to extract features, upsample and reconstruct the input low-resolution image, thereby obtaining the reconstructed spatial image features; b. Construct a frequency domain super-resolution network to perform frequency domain decomposition, feature extraction, upsampling and reconstruction on the input low-resolution image, thereby obtaining reconstructed frequency domain image features; c. Construct an attention fusion network to fuse the reconstructed spatial domain image features obtained in step a and the reconstructed frequency domain image features obtained in step b, and reconstruct a high-resolution image; The construction of the attention fusion network specifically includes the following steps: (1) The initial fusion feature F is calculated using the following formula shallow : F shallow =H RRDB (Attention(F s ,F freq )×F freq +F s ) Where F s To reconstruct the spatial domain image feature matrix; F freq is the reconstructed frequency domain image feature matrix; H RRDB () is the processing process corresponding to the dense residual module; Attention (F s ,F freq ) is the similarity calculation matrix, and Attention(F s ,F freq )=softmax(F s ×(F freq ) T ), softmax() is the softmax function operation; (2) The deep fusion feature F is calculated using the following formula deep : F deep =H RRDB (Attention(F shallow ,F freq )×F freq +F shallow ) Where F shallow is the preliminary fusion feature calculated in step (1); (3) The deep fusion feature F obtained in step (2) deep Input into two convolutional layers to obtain the output high-resolution image; the convolution kernels of the two convolutional layers are both 3*3; S4. Using the training data set obtained in step S2, the original image super-resolution model constructed in step S3 is trained to obtain an image super-resolution model; S5. Using the image super-resolution model obtained in step S5, super-resolution processing is performed on the actual image to complete the super-resolution of the actual image.

2. The image super-resolution method based on frequency domain and spatial domain according to claim 1, characterized in that Step S2 processes the original data set obtained in step S1 to construct a training data set, which specifically includes the following steps: A. Set the resolution of high-resolution images and low-resolution images in the training dataset; B. Segment the image in the original data set obtained in step S1 to obtain several sub-images as high-resolution images; C. Downsampling the high-resolution image obtained in step B to obtain a corresponding low-resolution image; D. Match the high-resolution image obtained in step B with the low-resolution image obtained in step C one by one to obtain a training data set.

3. The image super-resolution method based on frequency domain and spatial domain according to claim 2 is characterized in that The step a of constructing a spatial super-resolution network specifically includes the following steps: The spatial super-resolution network includes a spatial shallow feature extraction module, a spatial deep feature extraction module, a spatial upsampling module and a spatial feature reconstruction module which are connected in series in sequence; The spatial shallow feature extraction module includes a convolution layer with a convolution kernel size of 3*3. The spatial shallow feature extraction module is used to extract shallow features from the input low-resolution image and upload the extracted shallow image features to the spatial deep feature extraction module. The spatial deep feature extraction module includes 23 dense residual blocks and a convolution layer, and the convolution kernel size of the convolution layer is 3*3; the input shallow image features are sequentially passed through the dense residual block and the convolution layer to obtain the deep image features, and the intermediate features are retained; the spatial deep feature extraction module is used to extract the deep features of the image and upload the extracted deep image features to the spatial upsampling module; The spatial upsampling module includes an upsampling layer, a convolution layer and a relu activation function layer connected in series. The upsampling layer adopts the nearest neighbor sampling mode, and the convolution kernel size of the convolution layer is 3*3. The input deep image features are passed through the upsampling layer to obtain the spatial image features of the target resolution, and then the convolution operation is performed through the convolution layer. Finally, the relu function is activated to complete the upsampling process, and the final spatial image features are uploaded to the spatial feature reconstruction module. The spatial feature reconstruction module includes a first convolution layer, a relu activation function layer and a second convolution layer connected in series. The convolution kernel size of the first convolution layer is 3*3, and the convolution kernel size of the second convolution layer is 3*3. The input spatial image features are convolved through the first convolution layer, activated through the relu function, and then convolved through the second convolution layer to obtain the final reconstructed spatial image features.

4. The image super-resolution method based on frequency domain and spatial domain according to claim 3 is characterized in that The step b of constructing a frequency domain super-resolution network specifically includes the following steps: The frequency domain super-resolution network includes a frequency domain decomposition module, a frequency domain shallow feature extraction module, a frequency domain deep feature extraction module, a frequency domain upsampling module and a frequency domain feature reconstruction module which are connected in series in sequence; The frequency domain decomposition module uses a stable wavelet transform algorithm to convert the input low-resolution image into four frequency domain images: horizontal low-frequency image, horizontal high-frequency image, vertical high-frequency image, and diagonal high-frequency image; and uploads the obtained four frequency domain images to the frequency domain shallow feature extraction module; The frequency domain shallow feature extraction module includes a convolution layer with a convolution kernel size of 3*3; the frequency domain shallow feature extraction module is used to extract shallow features of the input frequency domain image and upload the extracted frequency domain shallow image features to the frequency domain deep feature extraction module; The frequency domain deep feature extraction module includes the first convolution layer, the first dense residual block layer, the second convolution layer, the second dense residual block layer, the third convolution layer, the third dense residual block layer, the fourth convolution layer, the fourth dense residual block layer and the fifth convolution layer, which are connected in series in sequence; the convolution kernel size of each convolution layer is 3*3; the input frequency domain shallow image features are connected with the intermediate features of the spatial domain super-resolution network and input into the first convolution layer together, and the output of the first convolution layer passes through the first dense residual block layer; the output of the first dense residual block layer is connected with the intermediate features of the spatial domain super-resolution network and input into the first convolution layer together. The output of the second convolutional layer is then passed through the second dense residual block layer; the output of the second dense residual block layer is connected with the intermediate features of the spatial super-resolution network and then input into the third convolutional layer together, and the output of the third convolutional layer is then passed through the third dense residual block layer; the output of the third dense residual block layer is connected with the intermediate features of the spatial super-resolution network and then input into the fourth convolutional layer together, and the output of the fourth convolutional layer is then passed through the fourth dense residual block layer; the output of the fourth dense residual block layer is then processed through the fifth convolutional layer to obtain the deep features in the frequency domain and upload it to the frequency domain upsampling module; The frequency domain upsampling module includes an upsampling layer, a convolution layer and a relu activation function layer connected in series. The upsampling layer adopts the nearest neighbor sampling mode, and the convolution kernel size of the convolution layer is 3*3. The input frequency domain deep image features are passed through the upsampling layer to obtain the frequency domain image features of the target resolution, and then the convolution operation is performed through the convolution layer, and finally the relu function is activated to complete the upsampling process, and the final frequency domain image features are uploaded to the frequency domain feature reconstruction module. The frequency domain feature reconstruction module includes a first convolution layer, a relu activation function layer and a second convolution layer connected in series. The convolution kernel size of the first convolution layer is 3*3, and the convolution kernel size of the second convolution layer is 3*3. The input frequency domain image features are convolved through the first convolution layer, activated through the relu function, and then convolved through the second convolution layer to obtain the final reconstructed frequency domain image features.

5. The image super-resolution method based on frequency domain and spatial domain according to any one of claims 1 to 4, characterized in that The training described in step S4 is specifically to use the following formula as the loss function L during training: G : L G =L percep +λL GAN +ηL1 Where L percep is the perceived loss; L GAN is the adversarial loss; L1 is the content loss; λ and η are weight parameters.

Citation Information

Patent Citations

  • High-speed acquisition MRI reconstruction method based on residual self-attention image enhancement

    CN111696168A

  • Image blind super-resolution method and system

    CN113139904A