A face super-resolution method and system based on frequency domain self-calibration feature enhancement

Abstract

The application belongs to the technical field of image super-resolution reconstruction, and discloses a face super-resolution method and system based on frequency domain self-calibration feature enhancement, which comprises the following steps: firstly, a multi-scale input is constructed through interpolation; then, a deep layer feature is extracted by using an encoder containing a CNN-KAN hybrid module (MCKM), and a face feature of an original input is fused to enhance multi-scale information mining; next, a fast Fourier adjustment module (FFAB) is used to realize optimization of face detail information in a frequency domain; finally, a multi-layer feature is dynamically fused by using a decoder, and a multi-scale reconstructed image is output; meanwhile, a model is jointly optimized in combination with a multi-scale space domain and a frequency domain reconstruction loss. According to the application, the feature representation is enhanced by using the MCKM, the model is optimized in the frequency domain by using the FFAB, and the encoder-decoder feature is cooperatively optimized by using an adaptive fusion module (AFM), so that the detail restoration precision and the structure fidelity of the face image reconstruction are significantly improved while the calculation efficiency is maintained.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution reconstruction technology, specifically to a face super-resolution method and system based on frequency domain self-calibration feature enhancement. Background Technology

[0002] Image super-resolution (SR) is an important research area in computer vision, aiming to recover higher-resolution images from low-resolution images. In practical applications such as face recognition, video surveillance, remote communication, and intelligent security, the acquired face images are often of low resolution due to limitations in shooting equipment, compression transmission, or imaging distance, severely affecting the performance of subsequent face analysis and recognition. Therefore, high-quality face image super-resolution technology has become one of the urgent problems to be solved.

[0003] In recent years, with the continuous evolution of deep learning technology, image super-resolution methods have shifted from Convolutional Neural Networks (CNNs) to the Visual Transformer architecture. Transformers, with their powerful long-range modeling capabilities and global context awareness, have demonstrated superior performance in image restoration tasks. However, because Transformers typically rely on large amounts of training data and computational resources, their complex structure and high training overhead make them difficult to deploy on resource-constrained devices. In contrast, traditional CNN structures offer greater lightweightness and structural controllability, and still possess significant practical application value.

[0004] Despite the progress made in structural design, the above methods still have the following shortcomings: On the one hand, existing methods mainly model and optimize in the spatial domain, making it difficult to fully explore high-frequency detail information in the frequency domain, resulting in poor image edge and texture restoration effects; on the other hand, there is insufficient collaborative modeling and fusion utilization of multi-scale image information, and a lack of effective enhancement mechanisms for semantic consistency between shallow and deep features, which limits the performance of the model in structure-sensitive tasks such as face images.

[0005] Therefore, there is an urgent need for an image super-resolution reconstruction method that is simple in structure, efficient and controllable, and integrates frequency domain enhancement mechanism and multi-scale feature interaction capability, so as to fully explore the potential of CNN in image restoration tasks and significantly improve the detail restoration capability and structural integrity of face images. Summary of the Invention

[0006] To address the problems mentioned in the background art, this invention provides a face super-resolution method and system based on frequency domain self-calibration feature enhancement. This method, with its lightweight structure and high computational efficiency, can accurately restore details of key regions (such as eyes, mouth, and bridge of the nose) in face images, significantly improving the clarity, structural consistency, and perceptual quality of the reconstructed images.

[0007] In a first aspect, the present invention provides a method for removing moiré patterns from high-resolution images, comprising the following steps:

[0008] Bicubic interpolation is performed on the original low-resolution face image to generate images of 2x and 4x the size as multi-scale inputs;

[0009] The multi-scale input is processed by a hierarchical encoder, which includes multiple encoding stages. Each encoding stage contains residual blocks and a CNN-KAN hybrid module for feature processing and downsampling: the image of 4 times the size is used as the initial input of the main path, processed by initial convolution and the first encoding stage to output the first encoder scale features; the image of 2 times the size is processed by the first convolution block to extract features, concatenated with the first encoder scale features, and then processed by the second encoding stage to output the second encoder scale features; the original low-resolution face image is processed by the second convolution block to extract features, concatenated with the second encoder scale features, and then processed by the third encoding stage to obtain the final encoded features;

[0010] The final encoded features are processed using multiple consecutively stacked Fast Fourier Transform adjustment modules and skip connections to obtain intermediate layer output features.

[0011] The intermediate layer output features are decoded using a hierarchical decoder. Each decoding level of the hierarchical decoder includes: deconvolution upsampling the input features to obtain the current decoding layer upsampled features; fusing the encoder-scale features output at the corresponding scale from the encoder with the current decoding layer upsampled features via an adaptive fusion module; and processing the fused features sequentially through a residual block and a CNN-KAN hybrid module. The hierarchical decoder employs a multi-scale output mode, outputting a reconstructed image at the corresponding scale after processing at each decoding level, ultimately generating a high-resolution image.

[0012] As an optional implementation of the first aspect of this application, the processing of the CNN-KAN hybrid module includes: dividing the input features into multiple parallel differential dilated convolution branches after max pooling; merging the outputs of each branch and concatenating them in the channel dimension; inputting the concatenated features into the KAN layer after tokenization, and then through a depthwise separable convolution to adjust the feature contribution using its learnable function; and combining the output of the KAN layer with the input features through skip connections, and then upsampling the output.

[0013] As an optional implementation of the first aspect of this application, both the first convolutional block and the second convolutional block include a series of convolutional operations with kernel sizes of 3×3, 1×1, 3×3 and 1×1, used to gradually increase the number of feature channels.

[0014] As an optional implementation of the first aspect of this application, the processing of the Fast Fourier Transform adjustment module includes: decomposing the input features into amplitude components and phase components by Fast Fourier Transform; optimizing the amplitude components and phase components by convolution respectively; and reconstructing the optimized components into spatial domain features by inverse Fast Fourier Transform.

[0015] As an optional implementation of the first aspect of this application, the adaptive fusion module's processing includes: concatenating the encoder scale features from the hierarchical encoder with the upsampled features of the current decoding layer of the hierarchical decoder; extracting general features from the concatenated features using convolution; generating weights from the general features through global average pooling, convolution, and activation functions; multiplying the encoder scale features and the upsampled features of the current decoding layer with the weights one by one; and finally concatenating the two adjusted features and combining them with the original concatenated features through residual connection before outputting.

[0016] As an optional implementation of the first aspect of this application, the hierarchical encoder and the hierarchical decoder have a symmetrical structure, that is, the number of downsampling operations in the hierarchical encoder is the same as the number of upsampling operations in the hierarchical decoder, and the encoder scale features output by each encoding stage in the hierarchical encoder are fused by an adaptive fusion module of a corresponding level in the hierarchical decoder.

[0017] As an optional implementation of the first aspect of this application, it further includes: training the model by optimizing it through multi-scale reconstruction loss and frequency domain reconstruction loss; wherein the multi-scale reconstruction loss uses the L1 norm to measure the pixel difference between the predicted image and the real image at multiple scales; and the frequency domain reconstruction loss constrains the difference between the real part and the imaginary part of the model output image and the real image after fast Fourier transform at multiple scales.

[0018] Secondly, embodiments of this application provide a face super-resolution system based on frequency domain self-calibration feature enhancement, comprising:

[0019] The multi-scale input generation module is used to perform bicubic interpolation on the original low-resolution face image to generate images of 2x and 4x size as multi-scale inputs.

[0020] A hierarchical encoder module is used to process the multi-scale input through a hierarchical encoder. The hierarchical encoder includes multiple encoding stages, each of which contains a residual block and a CNN-KAN hybrid module for feature processing and downsampling: the image of 4 times the size is used as the initial input of the main path, processed by initial convolution and the first encoding stage, and the first encoder scale features are output; the image of 2 times the size is processed by the first convolution block to extract features, concatenated with the first encoder scale features, and then processed by the second encoding stage to output the second encoder scale features; the original low-resolution face image is processed by the second convolution block to extract features, concatenated with the second encoder scale features, and then processed by the third encoding stage to obtain the final encoded features;

[0021] An intermediate processing module is used to process the final encoded features using multiple consecutively stacked Fast Fourier Transform adjustment modules and skip connections to obtain intermediate layer output features.

[0022] The hierarchical decoder module is used to decode the output features of the intermediate layer using the hierarchical decoder. The hierarchical decoder includes the following steps at each decoding level: performing deconvolution upsampling on the input features to obtain the upsampled features of the current decoding layer; fusing the encoder-scale features output by the encoder at the corresponding scale with the upsampled features of the current decoding layer through an adaptive fusion module; and processing the fused features sequentially through a residual block and a CNN-KAN hybrid module. The hierarchical decoder adopts a multi-scale output mode, outputting a reconstructed image of the corresponding scale after processing at each decoding level, and finally generating a high-resolution image.

[0023] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0024] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0025] Compared with existing technologies, the beneficial effects of this invention on face super-resolution based on frequency domain self-calibration feature enhancement are mainly reflected in:

[0026] 1. Improved Detail Reconstruction Accuracy: By introducing an encoder-decoder structure based on a CNN-KAN hybrid module, particularly the parallel dilated convolution and KAN dynamic weight allocation mechanism within the MCKM, the model's ability to represent and adaptively adjust multi-scale features is enhanced. Combined with a Fast Fourier Transform (FFT) adjustment module to progressively optimize facial detail information in the frequency domain, it overcomes the limitations of traditional spatial domain processing, enabling more effective recovery of high-frequency details in images, such as eye and lip contours, and hair texture.

[0027] 2. Enhanced structural fidelity: The adaptive fusion module employs a residual weighted fusion strategy, achieving collaborative optimization of shallow features from the encoder and deep features from the decoder. This dynamic fusion mechanism helps maintain the overall structural consistency of the face image and the geometric accuracy of key components, avoiding artifacts and unnatural deformations.

[0028] 3. Dual-domain joint optimization: By combining multi-scale spatial domain reconstruction loss and frequency domain reconstruction loss for joint optimization, model training not only focuses on pixel-level accuracy but also on frequency domain spectrum similarity. This dual constraint enables the model to better learn the intrinsic distribution of face images, thereby improving both global structure and local details, enhancing reconstruction quality and model stability.

[0029] 4. Maintaining computational efficiency: Despite the introduction of new modules and frequency domain processing, this invention was designed with computational efficiency in mind. For example, FFAB operations in the frequency domain are more efficient than depthwise convolutions in the spatial domain, and MCKM strives to control computational overhead while enhancing performance. This allows the method to significantly improve performance while maintaining a relatively reasonable computational complexity. Attached Figure Description

[0030] Figure 1 This is a flowchart of a face super-resolution method based on frequency domain self-calibration feature enhancement according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the overall network architecture of a face super-resolution method based on frequency domain self-calibration feature enhancement provided in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of the internal structure of the CNN-KAN hybrid module (MCKM) used in this embodiment of the invention.

[0033] Figure 4 This is a schematic diagram of the internal structure of the Fast Fourier Transform (FFAB) adjustment module used in this embodiment of the invention.

[0034] Figure 5 This is a schematic diagram of the internal structure of the Adaptive Fusion Module (AFM) used in an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of a face super-resolution system based on frequency domain self-calibration feature enhancement provided in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0038] Example 1

[0039] Please see Figure 1 The flowchart below illustrates a face super-resolution method based on frequency domain self-calibration feature enhancement, as provided in an embodiment of the present invention. Please refer to [link to flowchart illustration]. Figure 2 This is a schematic diagram of the overall network architecture of a face super-resolution method based on frequency domain self-calibration feature enhancement provided by an embodiment of the present invention. The method may include the following steps:

[0040] S1: Perform bicubic interpolation on the original low-resolution face image to generate images of 2x and 4x the size as multi-scale inputs.

[0041] In this embodiment, the present invention first processes the original low-resolution image of the human face ( The bicubic interpolation method is used to perform two upsampling operations on it. First, the bicubic interpolation method is used to perform two upsampling operations on it. Upsampling to twice its original size yields a medium-resolution image. ; then in Based on this, the image is upsampled again to generate a 4x resolution image. This multi-scale image group { As a multi-scale input to the network, it can introduce structural and texture information at different resolutions in the shallow stage, providing richer and more hierarchical prior guidance for subsequent feature extraction modules.

[0042] S2: The multi-scale input is processed by a hierarchical encoder, which includes multiple encoding stages. Each encoding stage contains residual blocks and a CNN-KAN hybrid module for feature processing and downsampling: The image of 4 times the size is used as the initial input of the main path, processed by initial convolution and the first encoding stage to output the first encoder scale features; The image of 2 times the size is processed by the first convolution block to extract features, concatenated with the first encoder scale features, and then processed by the second encoding stage to output the second encoder scale features; The original low-resolution face image is processed by the second convolution block to extract features, concatenated with the second encoder scale features, and then processed by the third encoding stage to obtain the final encoded features.

[0043] Specifically, images Entering the main encoder path, feature extraction and downsampling are performed through a convolutional layer (Conv), residual blocks (RBs), and a CNN-KAN hybrid module (Mixed CNN and Kolmogorov–Arnold Module, MCKM) to enhance global information representation. Simultaneously, and Features are extracted through convolution blocks (ConvB) and then fused into the main path via feature concatenation (Cat) to enhance the interaction of information at different scales.

[0044] The encoder described in this embodiment is a layered encoder, with each encoding layer consisting of convolution (Conv), RBs, and MCKM, and the features are downsampled twice during encoding.

[0045] First, the image Perform convolution operations to increase the number of channels and obtain shallow features. Then, RBs are used for feature extraction, and MCKM is used for feature refinement and downsampling to obtain the features. The formula is expressed as:

[0046]

[0047] Where RBs(•) represents a residual convolutional block of 8 residual units stacked together; MCKM(•) represents a CNN-KAN hybrid module; (•) indicates that the convolution operation is used for downsampling.

[0048] Shallow features By using RBs feature extraction, low-level features (such as edges and textures) are preserved while high-level semantic features are gradually extracted. The skip connection method avoids gradient decay during deep network training, providing richer feature inputs for the subsequent MCKM module.

[0049] In this embodiment, as Figure 3 As shown, the CNN-KAN hybrid module processes the following: after max pooling the input features, it splits them into multiple parallel differentially dilated convolution (Conv) branches, with convolution kernels d of 1, 3, 5, and 7 respectively; the outputs of each branch are merged and concatenated along the channel dimension; the concatenated features are then tokenized and input into the KAN layer, followed by a depthwise separable convolution (DwConv), using its learnable function (ReLU) to adjust the feature contribution; and the output of the KAN layer is combined with the input features through skip connections, then upsampled before output. In the figure, © represents channel concatenation, and ⊕ represents element-wise addition.

[0050] Specifically, in the CNN-KAN hybrid module, given the extracted features The process is divided into four independent processing branches, each of which performs differential dilated convolution. The convolutional kernel weights are trained independently to avoid information interference; then the four outputs are merged along the channel dimension to obtain the features. The features after splicing will be used in the next step. The features are converted into serialized tokens to provide multi-granularity inputs for the KAN layer. Subsequently, the learnable function basis of the KAN layer automatically adjusts the contribution of features at different scales, and jump connections are combined to form a multi-level feedback loop, finally outputting the features. The process can be formalized as follows:

[0051]

[0052] in, This represents a dilated convolution with a dilation rate of d. Indicates feature splicing, This refers to the Kolmogorov–Arnold network layer. This represents depthwise separable convolution; multiple dilated convolutions produce receptive fields of different sizes, thereby facilitating the smooth extraction of multi-scale information.

[0053] Then, the image ConvB gradually increases the number of channels and extracts features. ConvB is a multi-convolution operation on convolutional blocks (first and second convolutional blocks), including a series of convolutions with kernel sizes of 3×3, 1×1, 3×3, and 1×1. The number of channels is gradually increased from 3 to the target number. For a given extracted features... By using four convolutional blocks, the number of channels is gradually increased and features are extracted to obtain... The specific formula can be expressed as follows:

[0054]

[0055] in, and These represent convolution operations with kernel sizes of 1×1 and 3×3, respectively.

[0056] Subsequently, the extracted features are compared with those from the main path. The features are connected and then subjected to feature extraction and downsampling through RBs and MCKM to obtain the features. Similarly, images The number of channels is gradually increased and features are extracted using ConvB; subsequently, the extracted features are compared with features from the main path. The features are obtained by connecting the links, extracting features through RBs and MCKM, and downsampling. The process can be formalized as follows:

[0057]

[0058] Where RBs(•) represents residual convolutional blocks; MCKM(•) represents CNN-KAN hybrid modules. (•) indicates a convolution operation. This indicates feature splicing.

[0059] Through the above steps, the multi-scale features have been obtained by the encoder in this embodiment. .

[0060] S3: The final encoded features are processed using multiple consecutively stacked Fast Fourier Transform adjustment modules and skip connections to obtain intermediate layer output features.

[0061] Specifically, after feature encoding, six consecutively stacked Fast Fourier Adaptive Blocks (FFABs) can be used, operating via skip connections, as an intermediate layer to connect the encoder and decoder.

[0062] After feature encoding by the encoder, the features are... The input intermediate layer contains six consecutively stacked FFABs, which are operated through skip connections to achieve layer-by-layer refinement from low-frequency global structure to high-frequency details, thereby enhancing the model's robustness. The specific formula can be expressed as follows:

[0063]

[0064] in, This indicates the Fast Fourier Transform adjustment module. Indicates feature splicing, Indicates the first i The output of the Fast Fourier Transform adjustment module This is the final output of the intermediate layer.

[0065] In this embodiment, as Figure 4 As shown, the processing of the Fast Fourier Transform (FFT) adjustment module includes: decomposing the input features into amplitude components (Amp) and phase components (Pha) using FFT; optimizing the amplitude components and phase components through convolution; and reconstructing the optimized components into spatial domain features using inverse Fast Fourier Transform (iFFT).

[0066] Specifically, for a given feature input It is decomposed into amplitude components by fast Fourier transform. and phase components The optimization is performed through convolution, and the process can be formalized as follows:

[0067]

[0068] in, This represents the convolution operation. This represents the activation function. Represents the Fast Fourier Transform. This represents the inverse fast Fourier transform.

[0069] The Fast Fourier Transform (FFT) adjustment module converts features to the frequency domain through Fourier transform, decoupling features to optimize global and local features separately. It utilizes low-complexity operations in the frequency domain to avoid complex calculations in the spatial domain, thus achieving high efficiency in feature optimization.

[0070] S4: The hierarchical decoder decodes the output features of the intermediate layer. The hierarchical decoder at each decoding level includes: performing deconvolution upsampling on the input features to obtain the upsampled features of the current decoding layer; fusing the encoder-scale features output by the encoder at the corresponding scale with the upsampled features of the current decoding layer through an adaptive fusion module; and processing the fused features sequentially through a residual block and a CNN-KAN hybrid module. The hierarchical decoder adopts a multi-scale output mode, outputting a reconstructed image at the corresponding scale after processing at each decoding level, and finally generating a high-resolution image.

[0071] Specifically, the decoder is used to perform decoding operations, and the corresponding scale features (shallow features) in the encoder are adaptively fused with the reconstructed features (deep features) after decoding through the Adaptive Fusion Module (AFM). Deconvolution upsampling, RBs and MCKM are combined for decoding, and details are further optimized. Multi-scale output is adopted to generate high-resolution images.

[0072] The decoder described in this embodiment is a hierarchical decoder. Each decoding layer consists of convolution, AFM, RBs, and MCKM, and adopts a multi-output mode. Each decoding layer outputs an image of the corresponding scale. Subsequently, deconvolution upsampling is performed, and AFM is used to adaptively fuse the corresponding scale features in the encoder with the reconstructed features after decoding. This process involves two upsampling and AFM feature adaptive fusion operations. Finally, the reconstructed features are connected to the image via skip connections. Connect them and output the images separately. The decoding process can be formalized as follows:

[0073]

[0074] in, This indicates an adaptive fusion module that adjusts feature weights in real time based on the input content, achieving collaborative optimization of global semantics and local details, and improving the model's ability to model complex image degradation problems.

[0075] The multi-output image reconstruction process can be formalized as follows:

[0076]

[0077] in, These represent the reconstructed images at the corresponding scales.

[0078] It should be noted that the hierarchical encoder and the hierarchical decoder have a symmetrical structure, that is, the number of downsampling operations in the hierarchical encoder is the same as the number of upsampling operations in the hierarchical decoder, and the encoder scale features output by each encoding stage in the hierarchical encoder are fused by the corresponding level of the adaptive fusion module in the hierarchical decoder.

[0079] In this embodiment, the adaptive fusion module (AFM) processes the following steps: concatenating the encoder scale features from the hierarchical encoder with the upsampled features of the current decoding layer of the hierarchical decoder; extracting general features from the concatenated features using convolution; generating weights from the general features through global average pooling, convolution, and activation functions; multiplying the encoder scale features and the upsampled features of the current decoding layer with the weights one by one; and finally concatenating the two adjusted features and combining them with the original concatenated features through residual connection before outputting the result.

[0080] like Figure 5 As shown, the adaptive fusion module can adaptively fuse the corresponding scale features in the encoder with the reconstructed features after decoding. For the input corresponding scale features (shallow features) I in the encoder and the reconstructed features (deep features) after decoding. First, the data is concatenated, and then 1×1 and 3×3 convolutions are used to extract general feature information. .for Global average pooling (GAP) is used to extract global semantic information, and convolutional operations are used to mine features. Then, through... The function generates weights W, which guide the global importance allocation during fusion. The specific process can be expressed as follows:

[0081]

[0082] in, express The function generates weights. This represents the convolution operation. Represents the ReLU activation function. This represents the global average pooling function. This represents the Sigmoid function. Indicates feature splicing, This indicates that the terms are multiplied one by one. Representing shallow features Adaptive results Representing deep features Adaptive results.

[0083] Finally, the fusion result is output using residual connections. The specific process can be expressed as follows:

[0084]

[0085] in, This indicates the result after fusion. This approach represents feature concatenation, and retains the original features and enhances the ability to fuse details through residual connections, thereby improving model stability.

[0086] Furthermore, in some implementations, step S5 is also included: training the model by optimizing it through multi-scale reconstruction loss and frequency domain reconstruction loss; the multi-scale reconstruction loss uses the L1 norm to measure the pixel difference between the predicted image and the real image at multiple scales; the frequency domain reconstruction loss constrains the difference between the real part and the imaginary part of the model output image and the real image after fast Fourier transform at multiple scales.

[0087] Specifically, to optimize the model, a multi-scale spatial domain reconstruction loss is used, the L1 norm is used to measure the pixel difference between the predicted image and the real image, pixel-level error constraints are used to improve reconstruction accuracy, and multi-scale partitioning normalization is used to balance global structure and local details. The specific formula is shown below:

[0088]

[0089] in, Indicates multi-scale reconstruction loss. Indicates the normalized weights. For the first The total number of pixels at each scale is used to balance the contributions of different scales. Indicates the first Reconstructed images at various scales Indicates the first Original image at each scale.

[0090] The frequency domain difference between the output of the multi-scale frequency domain reconstruction loss constraint model and the original image is used to supervise low-frequency structure and high-frequency details at different scales. The specific formula is as follows:

[0091]

[0092] in, This represents the multi-scale frequency domain reconstruction loss. Indicates the normalized weights. For the first The total number of pixels at each scale and These represent the real and imaginary parts of the complex features after the Fast Fourier Transform, respectively. This indicates the concatenation operator.

[0093] The final loss function is as follows:

[0094]

[0095] in, The hyperparameters are used to balance dual-domain training. By leveraging the complementary advantages of multi-scale spatial domain reconstruction loss and frequency domain reconstruction loss, dual fine constraints on the details and structure of face images can be achieved, effectively improving reconstruction quality and model stability.

[0096] In summary, through the synergistic effect of the above steps, the face super-resolution method based on frequency domain self-calibration feature enhancement proposed in this invention can effectively recover high-resolution images with rich details and accurate structure from low-resolution face images.

[0097] Example 2

[0098] Please see Figure 6 The diagram shown is a schematic representation of a face super-resolution system based on frequency domain self-calibration feature enhancement, as proposed in the second embodiment of this application. The system includes the following key modules:

[0099] The multi-scale input generation module 100 is used to perform bicubic interpolation on the original low-resolution face image to generate images of 2x and 4x size as multi-scale inputs.

[0100] The hierarchical encoder module 200 is used to process the multi-scale input through a hierarchical encoder. The hierarchical encoder includes multiple encoding stages, each of which contains residual blocks and a CNN-KAN hybrid module for feature processing and downsampling: the image of 4 times the size is used as the initial input of the main path, processed by initial convolution and the first encoding stage, and the first encoder scale features are output; the image of 2 times the size is processed by the first convolution block to extract features, concatenated with the first encoder scale features, and then processed by the second encoding stage to output the second encoder scale features; the original low-resolution face image is processed by the second convolution block to extract features, concatenated with the second encoder scale features, and then processed by the third encoding stage to obtain the final encoded features;

[0101] The intermediate processing module 300 is used to process the final encoded features using multiple consecutively stacked Fast Fourier Transform adjustment modules and skip connections to obtain intermediate layer output features.

[0102] The hierarchical decoder module 400 is used to decode the output features of the intermediate layer using the hierarchical decoder. The hierarchical decoder includes the following steps at each decoding level: performing deconvolution upsampling on the input features to obtain the upsampled features of the current decoding layer; fusing the encoder-scale features output by the encoder at the corresponding scale with the upsampled features of the current decoding layer through an adaptive fusion module; and processing the fused features sequentially through a residual block and a CNN-KAN hybrid module. The hierarchical decoder adopts a multi-scale output mode, outputting a reconstructed image at the corresponding scale after processing at each decoding level, and finally generating a high-resolution image.

[0103] The face super-resolution system based on frequency domain self-calibration feature enhancement in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not impose specific limitations.

[0104] The face super-resolution system based on frequency domain self-calibration feature enhancement in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0105] The face super-resolution system based on frequency domain self-calibration feature enhancement provided in this application embodiment can realize the various processes implemented by the face super-resolution method based on frequency domain self-calibration feature enhancement in the method embodiment. To avoid repetition, these processes will not be described again here.

[0106] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a face super-resolution method based on frequency domain self-calibration feature enhancement, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0107] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a face super-resolution method based on frequency domain self-calibration feature enhancement, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0108] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0109] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0111] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A face super-resolution method based on frequency domain self-calibration feature enhancement, characterized in that, Includes the following steps: Bicubic interpolation is performed on the original low-resolution face image to generate images of 2x and 4x the size as multi-scale inputs; The multi-scale input is processed by a hierarchical encoder, which includes multiple encoding stages. Each encoding stage contains residual blocks and a CNN-KAN hybrid module for feature processing and downsampling: the image of 4 times the size is used as the initial input of the main path, processed by initial convolution and the first encoding stage to output the first encoder scale features; the image of 2 times the size is processed by the first convolution block to extract features, concatenated with the first encoder scale features, and then processed by the second encoding stage to output the second encoder scale features; the original low-resolution face image is processed by the second convolution block to extract features, concatenated with the second encoder scale features, and then processed by the third encoding stage to obtain the final encoded features; The processing of the CNN-KAN hybrid module includes: dividing the input features into multiple parallel differential dilated convolution branches after max pooling; merging the outputs of each branch and concatenating them along the channel dimension; inputting the concatenated features into the KAN layer after tokenization, and then passing them through a depthwise separable convolution to adjust the feature contribution using its learnable function; and combining the output of the KAN layer with the input features through skip connections, and then upsampling the output. The final encoded features are processed using multiple consecutively stacked Fast Fourier Transform adjustment modules and skip connections to obtain intermediate layer output features. The intermediate layer output features are decoded using a hierarchical decoder. The hierarchical decoder includes the following steps at each decoding level: performing deconvolution upsampling on the input features to obtain the current decoding layer upsampled features; fusing the encoder scale features output by the encoder at the corresponding scale with the current decoding layer upsampled features through an adaptive fusion module; and processing the fused features sequentially through a residual block and a CNN-KAN hybrid module. The adaptive fusion module's processing includes: concatenating the encoder scale features from the hierarchical encoder with the upsampled features of the current decoding layer of the hierarchical decoder; extracting general features from the concatenated features using convolution; generating weights from the general features through global average pooling, convolution, and activation functions; multiplying the encoder scale features and the upsampled features of the current decoding layer with the weights one by one; and finally concatenating the two adjusted features and combining them with the original concatenated features through residual connection before outputting the result. The layered decoder adopts a multi-scale output mode, outputting a reconstructed image of the corresponding scale after processing at each decoding level, and finally generating a high-resolution image.

2. The method according to claim 1, characterized in that, Both the first convolutional block and the second convolutional block include a series of convolutional operations with kernel sizes of 3×3, 1×1, 3×3 and 1×1, used to gradually increase the number of feature channels.

3. The method according to claim 1, characterized in that, The processing of the Fast Fourier Transform adjustment module includes: The input features are decomposed into amplitude and phase components by performing a Fast Fourier Transform. The amplitude component and the phase component are optimized separately by convolution; The optimized components are then reconstructed into spatial domain features using inverse fast Fourier transform.

4. The method according to claim 1, characterized in that, The hierarchical encoder and the hierarchical decoder have a symmetrical structure, that is, the number of downsampling operations in the hierarchical encoder is the same as the number of upsampling operations in the hierarchical decoder, and the encoder scale features output by each encoding stage in the hierarchical encoder are fused by the corresponding level of the adaptive fusion module in the hierarchical decoder.

5. The method according to claim 1, characterized in that, Also includes: The model is trained by optimizing the multi-scale reconstruction loss and the frequency domain reconstruction loss. The multi-scale reconstruction loss uses the L1 norm to measure the pixel differences between the predicted image and the real image at multiple scales. The frequency domain reconstruction loss constrains the differences between the real and imaginary parts of the model output image and the real image after Fast Fourier Transform at multiple scales.

6. A face super-resolution system based on frequency domain self-calibration feature enhancement, characterized in that, include: The multi-scale input generation module is used to perform bicubic interpolation on the original low-resolution face image to generate images of 2x and 4x size as multi-scale inputs. A hierarchical encoder module is used to process the multi-scale input through a hierarchical encoder. The hierarchical encoder includes multiple encoding stages, each of which contains a residual block and a CNN-KAN hybrid module for feature processing and downsampling: the image of 4 times the size is used as the initial input of the main path, processed by initial convolution and the first encoding stage, and the first encoder scale features are output; the image of 2 times the size is processed by the first convolution block to extract features, concatenated with the first encoder scale features, and then processed by the second encoding stage to output the second encoder scale features; the original low-resolution face image is processed by the second convolution block to extract features, concatenated with the second encoder scale features, and then processed by the third encoding stage to obtain the final encoded features; The CNN-KAN hybrid module's processing includes: max-pooling the input features and then dividing them into multiple parallel differentially dilated convolutional branches; merging the outputs of each branch and concatenating them along the channel dimension; inputting the concatenated features into the KAN layer after tokenization, followed by a depthwise separable convolution, using its learnable function to adjust the feature contribution; and combining the KAN layer output with the input features through skip connections, followed by upsampling and output. An intermediate processing module is used to process the final encoded features using multiple consecutively stacked Fast Fourier Transform adjustment modules and skip connections to obtain intermediate layer output features. A hierarchical decoder module is used to decode the output features of the intermediate layers. At each decoding level, the hierarchical decoder includes: performing deconvolution upsampling on the input features to obtain the upsampled features of the current decoding layer; fusing the encoder-scale features output at the corresponding scale from the encoder with the upsampled features of the current decoding layer through an adaptive fusion module; and processing the fused features sequentially through residual blocks and a CNN-KAN hybrid module. The hierarchical decoder adopts a multi-scale output mode, outputting a reconstructed image at the corresponding scale after processing at each decoding level, ultimately generating a high-resolution image. The adaptive fusion module's processing includes: concatenating the encoder-scale features from the hierarchical encoder with the upsampled features of the current decoding layer of the hierarchical decoder; extracting general features from the concatenated features using convolution; generating weights from the general features through global average pooling, convolution, and activation functions; multiplying the encoder-scale features and the upsampled features of the current decoding layer by the weights item by item; and finally concatenating the adjusted two features and combining them with the original concatenated features through residual connections before outputting the result.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of a face super-resolution method based on frequency domain self-calibration feature enhancement as described in any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of a face super-resolution method based on frequency domain self-calibration feature enhancement as described in any one of claims 1-5.

Images