Face super-resolution network based on transform domain and content guidance channel attention fusion
Through the face super-resolution network that integrates attention in the transform domain and content-guided channel, the problem of insufficient detail recovery and robustness of face super-resolution reconstruction in the prior art is solved, and efficient and accurate face image reconstruction is achieved , suitable for real-world applications.
Patent Information
- Application Number
- CN202411324968.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-27
AI Technical Summary
Existing face super-resolution reconstruction techniques have limitations in capturing high-resolution image details and processing real-world low-quality face images, especially under light conditions and pose changes, and require a large amount of computing resources and model parameters. , and it is impossible to accurately restore key facial features, and lacks generalization ability and robustness.
Adopt a face super-resolution network based on transformation domain and content-guided channel attention fusion, through a three-stage U-shaped structure and local-global feature collaboration module, content-guided channel attention fusion module, transformation domain and reflective convolutional features Enhanced modules to capture multi-scale features and generate high-quality face images to avoid face alignment steps.
It realizes efficient face super-resolution reconstruction under real-world conditions, improves image detail clarity and robustness, reduces computing resource requirements, and maintains good performance under different lighting and posture changes.
Smart Images
Figure CN119205508A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of face super-resolution reconstruction, and in particular to a face super-resolution network based on transform domain and content-guided channel attention fusion. Background Art
[0002] The purpose of face super-resolution reconstruction (FSR) is to reconstruct a high-resolution (HR) face image from a low-resolution (LR) input image. It plays a very important role in security monitoring, social media and entertainment, historical photo restoration, forensics and other fields. With the rapid development of society, various downstream tasks such as face recognition have increasingly higher requirements for face clarity, which brings huge challenges to face super-resolution reconstruction.
[0003] At present, most of the existing face super-resolution reconstructions have certain limitations, such as:
[0004] Convolutional neural network (CNN)-based methods have limitations in capturing long-range dependencies in images because the locality of the convolution operation makes it difficult to effectively recover details in high-resolution images;
[0005] Secondly, although multi-task learning methods enhance the performance of face super-resolution reconstruction by jointly training additional tasks (such as face parsing and landmark prediction), they require additional manually annotated data, and it is difficult to predict face prior information from low-resolution input.
[0006] In addition, most existing methods can only generate lower resolution images, which limits their widespread use in practical applications. Real-world face images are often affected by unknown degradation processes such as blur, noise, etc., which make the face super-resolution reconstruction task more complicated. Although deep learning models can theoretically improve the performance of face super-resolution reconstruction, they may require a lot of computing resources and model parameters, which may become a problem in resource-constrained environments.
[0007] In terms of preserving facial identity information, some methods cannot accurately restore the key features of the face when generating high-resolution images, which is crucial for subsequent tasks such as face recognition. In addition, the generalization ability and robustness of existing methods in processing low-quality real-world face images still need to be improved, and methods using generative adversarial networks also face challenges in balancing the realism and detail clarity of generated images. On the other hand, existing methods usually rely on synthetic datasets for evaluation and testing, which may not fully represent the diversity and complexity of real-world face images.
[0008] Therefore, how to make the model perform well in real-world applications, especially under different lighting conditions, posture changes, and expression changes, is an important research issue. With the continuous development of technology, designing an efficient and accurate face super-resolution reconstruction method to meet the growing real-time processing needs is also an urgent problem to be solved in this field. Summary of the invention
[0009] The purpose of the present invention is to solve the problems existing in the prior art and provide a face super-resolution network based on transform domain and content-guided channel attention fusion, which can clearly reconstruct blurred faces without face alignment.
[0010] In order to achieve the above object, the present invention is implemented through the following technical solutions:
[0011] The face super-resolution network based on transform domain and content-guided channel attention fusion includes the following steps:
[0012] S1. Collect a face image dataset and divide the dataset into a training set, a validation set, and a test set;
[0013] S2, crop the images according to the center point and resize the images to uniform pixels, use them as high-resolution images, then use bicubic interpolation operations to downsample these high-resolution images to uniform pixels and treat them as low-resolution inputs;
[0014] S3. Construct a face super-resolution network architecture based on transform domain and content-guided channel attention fusion, which adopts a three-stage U-shaped structure including encoding, transition and decoding; in the encoding stage, the network focuses on capturing multi-scale local and global features; in the decoding stage, the focus shifts to feature integration and image reconstruction;
[0015] S4. Using the training set, according to the face super-resolution network architecture based on transform domain and content-guided channel attention fusion, a face super-resolution model based on transform domain and content-guided channel attention fusion is obtained through training, and the model parameters are adjusted through the back propagation algorithm. At the same time, the validation set is used to verify the training effect and save the best model;
[0016] S5. The best saved model is tested using a test set to obtain a super-resolution reconstruction result of the face of the test image.
[0017] Preferably, in step S2, the images are cropped according to the center point and resized to 128×128 pixels to be used as high-resolution images; these high-resolution images are then downsampled to 16×16 pixels using a bicubic interpolation operation and are treated as low-resolution inputs.
[0018] Preferably, in step S3: in the encoding stage, the input is a low-resolution image I LR ;
[0019] First, a 3×3 convolution kernel is used for preliminary feature extraction;
[0020] Subsequently, these features will undergo three consecutive feature extraction stages, in each of which a local-global feature collaboration module is deployed, which consists of a facial structure perception unit and a self-attention module;
[0021] Next, the features go through a downsampling process, which consists of a 3×3 convolutional layer, an activation function, and another 3×3 convolutional layer, where the first convolutional layer has a stride of 2.
[0022] Preferably, in step S3: a feature refinement module is used in the transition phase.
[0023] Preferably, in step S3: in the decoding stage, the decoder receives deep features of the low-resolution image and gradually integrates these features through a content-guided channel attention fusion module to construct a super-resolution image;
[0024] The decoder consists of an upsampling module, a content-guided channel attention fusion module, and a local-global feature coordination module; the upsampling module contains a 6×6 transposed convolution layer with a stride of 2, followed by an activation function and a 3×3 convolution layer, and the transposed convolution layer is used to expand the feature map size and extract information; the content-guided channel attention fusion module fuses the features extracted in the encoding stage in this process, ensuring that the local and global features in the encoding and decoding stages are fully utilized to generate high-quality facial images; at the end of the decoding stage, a 3×3 convolution layer is used to map the integrated features to the final super-resolution output I out ;
[0025] Finally, by combining the low-resolution image I LR And super-resolution output I out Get an enhanced facial image.
[0026] Preferably, in step S4: during the training process, a data set is given The model parameters of the face super-resolution network based on transform domain and content-guided channel attention fusion are adjusted by minimizing the pixel-level loss function:
[0027]
[0028] Where N represents the number of training images; is the low-resolution image of the i-th image; is the true high-resolution image corresponding to the low-resolution image of the i-th image; F TCCTNet (·) represents the face super-resolution network based on transform domain and content-guided channel attention fusion; Θ represents the parameters of the face super-resolution network based on transform domain and content-guided channel attention fusion.
[0029] Preferably, in the content-guided channel attention fusion module, the channel attention and spatial attention are first calculated by global average pooling and global maximum pooling operations, as well as 1x1 and 7x7 convolutional layers; then the features F obtained by the channel attention and spatial attention are added by an addition operation. C and F S Fusion, get a rough spatial importance map, denoted as F COS =F C +F S ; Then, the content of the input features is used to guide the generation of the final channel-specific spatial importance map, which is achieved by converting F COS Each channel of is alternately arranged with the channel of the input feature X through a channel shuffle operation, and then completed through group convolution and activation function.
[0030] Preferably, the encoding and the features of the same level in the encoding process are weighted and summed, and the specific formula is as follows:
[0031] X=F en +F de ,
[0032]
[0033] F COS =F C +F S ,
[0034] F=σ(GC 7×7 (CS([X,F coa ]))),
[0035] Among them, max(0,x) represents the activation function; C K×K (·) represents a convolution with a kernel size of k×k; [·] represents a channel-level connection operation; F C , GAP, as well as denotes the features processed by the global average pooling operation in the spatial dimension, the features processed by the global average pooling operation in the channel dimension, and the features processed by the global maximum pooling operation in the channel dimension; σ denotes the activation function; CS(·) denotes the channel shuffle operation, GC k×k(·) represents a grouped convolutional layer with a kernel size of k×k. In the implementation, the number of groups is set to C. Then, the input features are directly added to the fusion process by adding skip connections. A 1x1 convolution is then used to match the channels to the subsequent network.
[0036] Preferably, the two feature maps obtained in the encoding and decoding stages and the feature maps obtained by weighted summation are spliced, and a channel attention module is added. The channel attention module consists of adaptive average pooling, two 1x1 convolutions, two activation functions and residual connections. The overall mathematical expression of the content-guided channel attention fusion module is as follows:
[0037] F out =CA(C 1×1 (Concat(F en ,F de ,C 1×1 (F p )))),
[0038] F p =F en +F de +F en ×F+F de ×(1-F),
[0039] Among them, CA represents channel attention and F is the spatial importance map mentioned above.
[0040] Preferably, in step S3: the face super-resolution network architecture based on transform domain and content-guided channel attention fusion includes transform domain and reflection convolution feature enhancement modules, the transform domain and reflection convolution feature enhancement modules include frequency domain branches and reparameterized focused convolutions, the frequency domain branches first convert the input features to the frequency domain using a two-dimensional real fast Fourier transform, in the frequency domain, the feature map is multiplied with a set of learnable complex filters, and then the feature map is returned to the spatial domain after an inverse fast Fourier transform; given a pre-trained model, the reparameterized focused convolution applies a trainable refocusing transform to the basic convolution kernel inherited from the pre-trained model to establish a connection between parameters, and in the case of deep convolution, the reparameterized focused convolution uses the frozen pre-trained convolution kernel as the base weight, and applies the refocusing transform to generate a new deep convolution kernel.
[0041] Compared with the prior art, the beneficial effects of the present invention are:
[0042] 1. The present invention proposes a face super-resolution network based on transform domain and content-guided channel attention fusion for face super-resolution reconstruction, which can clearly reconstruct blurred faces without the need for face alignment.
[0043] 2. The present invention proposes a novel transform domain and reflected convolution feature enhancement module, which consists of a frequency domain transform branch and a reparameterized focused convolution, which transforms and enhances the features in the frequency domain to capture more comprehensive facial structure information while refining and enhancing local details.
[0044] 3. The present invention proposes a new content-guided channel attention fusion module to effectively aggregate the rich features of the encoder and decoder to better reconstruct high-quality images.
[0045] 4. This paper proposes a novel convolution-self-attention aggregation network architecture for face super-resolution, which achieves state-of-the-art performance in various indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is the overall framework of face super-resolution network based on transform domain and content-guided channel attention fusion;
[0047] Figure 2 It is a framework diagram of the content-guided channel attention fusion module;
[0048] Figure 3 This is the framework diagram of the channel attention module;
[0049] Figure 4 It is the framework diagram of the transform domain and reflection convolution feature enhancement module. DETAILED DESCRIPTION
[0050] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.
[0051] Embodiment: The present invention is a face super-resolution network based on transform domain and content-guided channel attention fusion, and the specific implementation steps are as follows:
[0052] S1: Collect a face image dataset and divide the dataset into training set, validation set and test set.
[0053] S2: The images are cropped according to the center point and resized to 128 × 128 pixels, which are used as high-resolution (HR) images. These high-resolution (HR) images are then downsampled to 16 × 16 pixels using a bicubic interpolation operation and are considered as low-resolution (LR) inputs.
[0054] S3: Construct a face super-resolution network architecture based on transform domain and content-guided channel attention fusion. It adopts a three-stage U-shaped structure, including three main parts: encoding, transition and decoding. In the encoding stage, the network focuses on capturing multi-scale local and global features; while in the decoding stage, the focus shifts to feature integration and image reconstruction.
[0055] S4: Using the training set, according to the face super-resolution network architecture based on transform domain and content-guided channel attention fusion, a face super-resolution model based on transform domain and content-guided channel attention fusion is obtained through training, and the model parameters are adjusted through the back propagation algorithm. At the same time, the validation set is used to verify the training effect and save the best model;
[0056] S5: Test the best saved model using the test set to obtain the face super-resolution reconstruction result of the test image.
[0057] Next, the face super-resolution network architecture based on transform domain and content-guided channel attention fusion mentioned in the above steps S3 and S4 will be introduced in detail.
[0058] The overall structure of the face super-resolution network (TCCTNet) based on transform domain and content-guided channel attention fusion. Figure 1 As shown in Figure 1, TCCTNet adopts a three-stage U-shaped structure, including three main parts: encoding, transition, and decoding. In the encoding stage, the network focuses on capturing multi-scale local and global features; while in the decoding stage, the focus shifts to feature integration and image reconstruction.
[0059] In addition, to ensure effective integration of features, the present invention introduces connections between the encoding and decoding stages as well as the transform domain and reflection convolution feature enhancement module (TDRFEM). In order to clearly explain the model architecture of the present invention, several key terms are introduced: LR Represents the input low-resolution image, I SR refers to the super-resolution image generated by the model, and I HR represents the actual high-resolution image.
[0060] 1) Feature extraction stage: The goal of this invention is to capture the intrinsic structure of the image through the feature extraction stage. The input is a low-resolution image I LR, the present invention first uses a 3×3 convolution kernel for preliminary feature extraction. Subsequently, the features will undergo three consecutive feature extraction stages. At each stage, the present invention deploys a local-global feature collaboration module (LGCM), which consists of a facial structure perception unit (FSAU) and a self-attention (Transformer) module. Next, the features go through a downsampling process consisting of a 3×3 convolution layer, an activation function (ReLu), and another 3×3 convolution layer, where the first convolution layer has a stride of 2, aiming to retain key information while reducing the size of the feature map. As a result, after each feature extraction stage, the size of the output feature map is halved, while the number of channels is doubled.
[0061] For example, suppose the input feature map is Then in the i-th feature extraction stage, the encoder will generate a feature map
[0062] 2) Transition stage: The key transition stage is between feature extraction and reconstruction. At this stage, all the features collected during the encoding process will be converged. In order to ensure that these features are effectively used in the subsequent reconstruction process, the present invention adopts a feature refinement module (FRM). The purpose of this module is to optimize and strengthen the encoded features to highlight the details of the facial structure. Through the intervention of the feature refinement module (FRM), the model can process facial features more intensively and enhance the information expression of different facial regions.
[0063] 3) Decoding stage: In the decoding stage, the present invention aims to restore high-quality facial images. The present invention introduces an innovative module called content-guided channel attention fusion module (CGCAF). The decoder receives deep features of low-resolution images and gradually integrates these features through this module to construct a super-resolution image. Figure 1 As shown in Figure 1, the decoder consists of an upsampling module, a content-guided channel attention fusion module (CGCAF), and a local-global feature coordination module (LGCM). The upsampling module contains a 6×6 transposed convolution layer with a stride of 2, followed by an activation function (ReLu) and a 3×3 convolution layer. The transposed convolution layer is used to expand the feature map size and extract information. In this way, the decoder reduces the number of output feature channels and doubles the feature map size in each step. The content-guided channel attention fusion module (CGCAF) fuses the features extracted in the encoding stage in this process, ensuring that the local and global features in the encoding and decoding stages are fully utilized to generate high-quality facial images.
[0064] At the end of the decoding stage, the present invention uses a 3×3 convolutional layer to map the integrated features into the final super-resolution output I out .
[0065] Finally, the present invention combines the low-resolution image I LR And super-resolution output I out Get enhanced facial images. During the training process, given a dataset The present invention adjusts the TCCTNet model parameters of the present invention by minimizing the pixel-level loss function:
[0066]
[0067] Where N represents the number of training images. and are the low-resolution (LR) image and the corresponding true high-resolution (HR) image of the i-th image. TCCTNet (·) and Θ represent the TCCTNet network and its network parameters, respectively.
[0068] Content-Guided Channel Attention Fusion Module (CGCAF):
[0069] In order to enhance the feature representation capability and promote the fusion of the encoding area and the decoder features, the present invention introduces a content-guided channel attention fusion scheme in the decoding stage, so that the network can strengthen the representation capability of the convolutional neural network (CNN) for different regions in the features, especially those regions that are critical to the face restoration task. Specifically, the main goal of the present invention is to explore and utilize the features of the encoding stage in the decoding process. However, the traditional convolutional layer has no constraints when searching the solution space during feature extraction, such as spatial attention and channel attention, which are usually calculated independently without information interaction between channels and spatial dimensions, resulting in the facial structure prior, color and texture consistency prior, etc. The face prior knowledge is not fully utilized and the unique meaning of each channel in the feature space and the contextual information in the spatial dimension cannot be fully considered. In order to solve this problem, the present invention designs a content-guided channel attention fusion (CGCAF) module. The detailed information of the CGCAF module is in Figure 2 Given in.
[0070] according to Figure 2 , it can be observed that the content-guided channel attention fusion module (CGCAF) first calculates the channel attention and spatial attention through global average pooling (GAP) and global maximum pooling (GMP) operations, as well as 1x1 and 7x7 convolutional layers, and then simply adds the features F obtained by channel attention and spatial attention. C and F S Fusion, get the rough spatial importance map (RSIM), and record F COS =F C +F SNext, the content of the input features is used to guide the generation of the final channel-specific spatial importance map (SIM), which is achieved by dividing F COS Each channel of is alternately arranged with the channel of the input feature X through a channel shuffle operation, and then completed through group convolution and activation function (Sigmoid).
[0071] In order to better fuse the same-level features in the encoding process, the present invention performs weighted summation on them according to the spatial importance map (SIM). Because features at different levels have different receptive fields, simple addition or concatenation operations cannot solve the problem of mismatched receptive fields before fusion. The specific formula is as follows:
[0072] X=F en +F de ,
[0073]
[0074] F COS =F C +F S ,
[0075] F=σ(GC 7×7 (CS([X,F coa ]))),
[0076] Among them, max(0,x) represents the activation function (ReLU), C K×K (·) represents a convolution with a kernel size of k×k, and [·] represents a channel-level connection operation; F C , GAP, as well as They represent the features processed by the global average pooling operation on the spatial dimension, the features processed by the global average pooling operation on the channel dimension, and the features processed by the global maximum pooling operation on the channel dimension. σ represents the activation function (Sigmoid), CS(·) represents the channel shuffle operation, and GC k×k (·) denotes a grouped convolutional layer with a kernel size of k×k. In the implementation, the number of groups is set to C. Then, by adding skip connections, the input features are directly added to the fusion process, which helps alleviate the gradient vanishing problem and simplifies the learning process.
[0077] Then use a 1x1 convolution to match the channel with the subsequent network. For example, the feature map channels obtained in the encoding stage are 32, 64 and 128 respectively. Taking 128 channels as an example, due to the requirements of the overall network design, the 128-channel feature map obtained in the encoding stage and the 128-channel feature map in the decoding stage must also be 128 channels after passing through the content-guided channel attention fusion (CGCAF) module. The role of the 1x1 convolution after the final splicing (concat operation) is here, and the 1x1 convolution before splicing is to reduce the amount of calculation.
[0078] It can be seen that the present invention concatenates the two feature maps obtained in the encoding and decoding stages and the feature map obtained by weighted summation at the end, in order to reduce the loss of prior knowledge in the above complex calculation process. Due to the concatenation of a large amount of information at the end, some channel importance information will be lost on the channel. The present invention solves this problem by adding a channel attention (CA) solution.
[0079] The specific structure of the channel attention module (CA) is as follows Figure 3 , which are composed of adaptive average pooling, two 1x1 convolutions, two activation functions and residual connections.
[0080] The overall mathematical expression of the content-guided channel attention fusion module (CGCAF) is as follows:
[0081] F out =CA(C 1×1 (Concat(F en ,F de ,C 1×1 (F p )))),
[0082] F p =F en +F de +F en ×F+F de ×(1-F),
[0083] Among them, CA represents channel attention and F is the spatial importance map (SIM) mentioned above.
[0084] Transform Domain and Reflection Convolutional Feature Enhancement Module (TDRFEM):
[0085] As one of the most important modules in TCCTNet, the Transform Domain and Reflection Convolutional Feature Enhancement Module (TDRFEM) is designed to achieve global spatial interaction. Figure 4 As shown in Figure 3, the module consists of a frequency domain branch and a reparameterized focused convolution, which are used for global and local feature extraction, respectively.
[0086] 1) Frequency domain branch: In the face super-resolution reconstruction task, the main challenge is how to extract key facial features (such as eyes, eyebrows and mouth) and make the network pay more attention to these features. To achieve this, the present invention uses Fourier transform to enable the model to extract as much useful information as possible to better restore details. Figure 4 As shown in the figure, the frequency domain branch first converts the input features to the frequency domain using a two-dimensional real fast Fourier transform. In the frequency domain, the feature map is multiplied with a set of learnable complex filters that can adjust the frequency components of the feature map and enhance or suppress specific frequency information. In this way, the model can capture the global frequency characteristics of the input data. Subsequently, the feature map is transformed back to the spatial domain through an inverse fast Fourier transform, so that the model can use these feature maps processed in the frequency domain.
[0087] 2) Reparameterized Focused Convolution: The frequency domain branch is equivalent to using a convolution kernel with a global size and ring padding to perform a convolution operation in the spatial domain, which can capture the spatial information on the entire feature map, making it easier for the network to capture long-distance dependencies. Reparameterized focused convolution is first of all a convolution that can capture local information. At the same time, it is an improvement on the traditional convolution layer. It enhances the representation ability of the model through reparameterization technology without increasing the inference cost.
[0088] Given a pre-trained model, re-parameterized focused convolution applies a trainable refocusing transformation to the base kernels inherited from the pre-trained model to establish connections between parameters. For example, in depthwise convolution, re-parameterized focused convolution can relate the kernel parameters of a specific channel to the parameters of other kernels, enabling them to refocus on other parts of the model rather than just the input features. In the case of depthwise convolution, re-parameterized focused convolution takes the frozen pre-trained kernels as base weights and applies a refocusing transformation to generate new depthwise convolution kernels. This transformation is achieved by introducing trainable refocusing weights that are combined with the base weights to generate transformed weights for operating on the input features. The refocusing transformation of re-parameterized focused convolution aims to learn deltas on the base weights rather than the original mapping, similar to how residual blocks in residual learning learn deltas on the base feature maps. In this way, re-parameterized focused convolution enables each kernel channel to be connected to all other channels, thereby learning new feature representations.
[0089] With the help of frequency domain branches and reparameterized focused convolution, the transform domain and reflective convolution feature enhancement module (TDRFEM) is able to capture both local features and global relations of the face, which is beneficial for high-quality image reconstruction.
Claims
1. A face super-resolution network based on transform domain and content-guided channel attention fusion, characterized by: Includes steps: S1. Collect a face image dataset and divide the dataset into a training set, a validation set, and a test set; S2, crop the images according to the center point and resize the images to uniform pixels, use them as high-resolution images, then use bicubic interpolation operations to downsample these high-resolution images to uniform pixels and treat them as low-resolution inputs; S3. Construct a face super-resolution network architecture based on transform domain and content-guided channel attention fusion, which adopts a three-stage U-shaped structure including encoding, transition and decoding; in the encoding stage, the network focuses on capturing multi-scale local and global features; in the decoding stage, the focus shifts to feature integration and image reconstruction; S4. Using the training set, according to the face super-resolution network architecture based on transform domain and content-guided channel attention fusion, a face super-resolution model based on transform domain and content-guided channel attention fusion is obtained through training, and the model parameters are adjusted through the back propagation algorithm. At the same time, the validation set is used to verify the training effect and save the best model; S5. The best saved model is tested using a test set to obtain a super-resolution reconstruction result of the face of the test image.
2. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 1, characterized in that: In step S2, the images are cropped according to the center point and resized to 128×128 pixels to be used as high-resolution images; these high-resolution images are then downsampled to 16×16 pixels using a bicubic interpolation operation and are considered as low-resolution inputs.
3. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 1, characterized in that: In step S3: In the encoding stage, the input is a low-resolution image I LR ; First, a 3×3 convolution kernel is used for preliminary feature extraction; Subsequently, these features will undergo three consecutive feature extraction stages, in each of which a local-global feature collaboration module is deployed, which consists of a facial structure perception unit and a self-attention module; Next, the features go through a downsampling process, which consists of a 3×3 convolutional layer, an activation function, and another 3×3 convolutional layer, where the first convolutional layer has a stride of 2.
4. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 3, characterized in that: In step S3: a feature refinement module is adopted in the transition stage.
5. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 4, characterized in that: In step S3: in the decoding stage, the decoder receives the deep features of the low-resolution image and gradually integrates these features through the content-guided channel attention fusion module to construct a super-resolution image; The decoder consists of an upsampling module, a content-guided channel attention fusion module, and a local-global feature coordination module; the upsampling module contains a 6×6 transposed convolution layer with a stride of 2, followed by an activation function and a 3×3 convolution layer, and the transposed convolution layer is used to expand the feature map size and extract information; the content-guided channel attention fusion module fuses the features extracted in the encoding stage in this process, ensuring that the local and global features in the encoding and decoding stages are fully utilized to generate high-quality facial images; at the end of the decoding stage, a 3×3 convolution layer is used to map the integrated features to the final super-resolution output I out ; Finally, by combining the low-resolution image I LR And super-resolution output I out Get an enhanced facial image.
6. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 5, characterized in that: In step S4: During the training process, given a data set The model parameters of the face super-resolution network based on transform domain and content-guided channel attention fusion are adjusted by minimizing the pixel-level loss function: Where N represents the number of training images; is the low-resolution image of the i-th image; is the true high-resolution image corresponding to the low-resolution image of the i-th image; F TCCTNet (·) represents the face super-resolution network based on transform domain and content-guided channel attention fusion; Θ represents the parameters of the face super-resolution network based on transform domain and content-guided channel attention fusion.
7. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 5, characterized in that In the content-guided channel attention fusion module, the channel attention and spatial attention are first calculated through global average pooling and global maximum pooling operations, as well as 1x1 and 7x7 convolutional layers; then the features F obtained by the channel attention and spatial attention are added together through the addition operation. C and F S Fusion, get a rough spatial importance map, denoted as F COS =F C +F S ; Then, the content of the input features is used to guide the generation of the final channel-specific spatial importance map, which is achieved by converting F COS Each channel of is alternately arranged with the channel of the input feature X through a channel shuffle operation, and then completed through group convolution and activation function.
8. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 7, characterized in that: The weighted sum of the same-level features in the encoding process is as follows: X=F en +F de , F COS =F C +F S , F=σ(GC 7×7 (CS([X,F coa ]))), Among them, max(0,x) represents the activation function; C K×K (·) represents a convolution with a kernel size of k×k; [·] represents a channel-level connection operation; F C , GAP, as well as denotes the features processed by the global average pooling operation in the spatial dimension, the features processed by the global average pooling operation in the channel dimension, and the features processed by the global maximum pooling operation in the channel dimension; σ denotes the activation function; CS(·) denotes the channel shuffle operation, GC k×k (·) represents a grouped convolutional layer with a kernel size of k×k. In the implementation, the number of groups is set to C. Then, the input features are directly added to the fusion process by adding skip connections. A 1x1 convolution is then used to match the channels to the subsequent network.
9. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 8, characterized in that: The two feature maps obtained in the encoding and decoding stages and the feature maps obtained by weighted summation are concatenated, and a channel attention module is added. The channel attention module consists of adaptive average pooling, two 1x1 convolutions, two activation functions, and residual connections. The overall mathematical expression of the content-guided channel attention fusion module is as follows: F out =CA(C 1×1 (Concat(F en ,F de ,C 1×1 (F p )))), F p =F en +F de +F en ×F+F de ×(1-F), Among them, CA represents channel attention and F is the spatial importance map mentioned above.
10. The face super-resolution network based on transform domain and content-guided channel attention fusion according to claim 1, characterized in that: In step S3: the face super-resolution network architecture based on transform domain and content-guided channel attention fusion includes transform domain and reflection convolution feature enhancement modules, which include frequency domain branches and reparameterized focused convolutions. The frequency domain branches first convert the input features to the frequency domain using a two-dimensional real fast Fourier transform. In the frequency domain, the feature map is multiplied with a set of learnable complex filters. Subsequently, the feature map is returned to the spatial domain through an inverse fast Fourier transform. Given a pre-trained model, the reparameterized focused convolution applies a trainable refocusing transform to the basic convolution kernel inherited from the pre-trained model to establish a connection between parameters. In the case of deep convolution, the reparameterized focused convolution uses the frozen pre-trained convolution kernel as the base weight and applies the refocusing transform to generate a new deep convolution kernel.