High-resolution face super-resolution method based on enhanced mamba and adversarial learning

Abstract

The application provides a high-magnification face super-resolution method based on enhanced Mamba and adversarial learning, and belongs to the technical field of image processing. The method comprises the following steps: acquiring and constructing a training data set; performing a horizontal flip operation on low-resolution-high-resolution paired image samples and randomly performing 90-degree, 180-degree and 270-degree rotation transformation; pre-processing the input low-resolution image; acquiring and constructing the training data set; performing shallow feature extraction on the input by using a convolution layer; performing deep feature extraction on the image by using a global semantic modeling and local region focusing strategy; fusing the deep features and the shallow features by using a residual method; performing up-sampling processing on the features, performing an inverse normalization operation on the generated face image, and outputting a final high-resolution face reconstruction result; constructing a generative adversarial learning training structure; and gradually optimizing the training model parameters to obtain a good training result. The application effectively improves the reconstruction quality of the face super-resolution.

Description

Technical Field

[0001] This invention provides a high-magnification face super-resolution method based on enhanced Mamba and adversarial learning, belonging to the field of image processing technology. Background Technology

[0002] Face super-resolution, also known as face illusion, is an important branch of single-image super-resolution. Face super-resolution technology aims to recover high-resolution face images from low-resolution face images, thereby improving the visual quality and recognizability of face images. In practical scenarios such as public security monitoring, virtual reality, and biometrics, high-quality face images have significant application value. However, due to limitations in physical equipment, image transmission compression, and shooting environment under actual acquisition conditions, face images often suffer from low resolution, lack of detail, and identity distortion. Therefore, improving the quality of face images through super-resolution algorithms to meet downstream tasks such as identity recognition and attribute analysis is of great significance.

[0003] The human face possesses a unique anatomical structure and highly semantic features, making people extremely sensitive to facial regions. Subtle texture differences can significantly influence subjective judgments about identity, age, emotion, and even authenticity. This characteristic makes face image super-resolution more challenging than natural image super-resolution. However, low-resolution images often lack key texture structures, leading to the loss of high-frequency information. Existing methods struggle to accurately infer true textures from blurred and incomplete original information, easily resulting in over-smoothing, rampant pseudo-textures, or a lack of realistic texture in the super-resolution image. Furthermore, the textures of facial regions typically exhibit significant non-linear relationships. Features such as hair and wrinkles present continuous but complex spatial variation patterns. If the model fails to capture these non-linear dependencies, visual credibility will be significantly reduced. Therefore, maintaining spatial consistency of texture distribution while improving texture detail quality is a crucial breakthrough in the field of face super-resolution, placing higher demands on feature representation methods. Summary of the Invention

[0004] This invention provides a high-magnification face super-resolution method based on enhanced Mamba and adversarial learning to solve the following technical problems:

[0005] (1) To address the problem of limited receptive field in super-resolution models based on convolutional neural networks, this invention utilizes the selective scanning mechanism of state-space models to design a super-resolution method that can effectively model long-range dependencies in face images, thereby improving the overall structural consistency and the accuracy of facial contour layout.

[0006] (2) To address the problem that convolutional neural networks have difficulty in uniformly modeling global semantics and local structure, this invention uses a combination strategy of global dependency modeling and local key region focusing to model the face region, thereby enhancing the model's ability to express the overall semantic information and structural relationships of the face.

[0007] (3) To address the problem of a large number of irrelevant regions interfering with effective modeling during global modeling, this invention constructs a sparse attention mechanism based on semantic similarity, which enables the model to prioritize semantic regions that are more critical to the super-resolution of face images, thereby improving modeling efficiency and reconstruction effect.

[0008] (4) To address the problem that semantically related regions in face images are spatially dispersed and difficult to jointly model, this invention adopts a feature sequence reordering strategy based on semantic similarity to aggregate features that are semantically similar but spatially dispersed, thereby enhancing the global modeling's ability to perceive key structures.

[0009] (5) To address the problem of weakened spatial location information in the serialization modeling process of two-dimensional images, this invention extends rotation coding to the two-dimensional level and directly encodes spatial location information into vectors, thereby enhancing the model's utilization of image spatial location information.

[0010] (6) To address the problem of missing local detail information in the global modeling stage, this invention designs a space-frequency joint adaptive feature enhancement module to compensate and enhance the features of the global modeling stage in order to restore key texture information.

[0011] (7) To address the problem of unstable training of super-resolution models based on generative adversarial networks, this invention uses a relative judgment loss function for optimization to achieve more stable adversarial learning and reduce the phenomena of pattern collapse and reconstruction quality fluctuations during adversarial training.

[0012] (8) In view of the shortcomings of super-resolution models based on generative adversarial networks, which are prone to introducing artifacts and affecting identity consistency, this invention constructs a symmetric discriminator structure that integrates multi-scale information, and integrates shallow structural information and deep semantic information to maintain the authenticity and reliability of facial identity features.

[0013] (9) To address the problem of high computational complexity and high resource consumption of Transformer-based super-resolution models, this invention utilizes Mamaba to perform global semantic modeling, thereby achieving a global receptive field comparable to that of Transformer while maintaining linear complexity.

[0014] To address the aforementioned issues, a generative adversarial network based on acquiring and constructing a training dataset using Mamba and a state-space model is proposed. Through a collaborative strategy of global semantic modeling and key region focusing, this approach effectively improves the reconstruction quality of face super-resolution while ensuring computational efficiency, taking into account both local detail restoration and overall structural consistency.

[0015] The specific technical solution is as follows:

[0016] 1. A high-magnification face super-resolution method based on enhanced Mamba and adversarial learning, characterized by the following steps:

[0017] S1. Obtain and construct the training dataset;

[0018] S2. Perform a horizontal flip operation on the low-resolution–high-resolution paired image samples, and randomly perform rotation transformations of 90°, 180° and 270° to expand the number of training samples and improve the generalization ability of the model.

[0019] S3. For the input low-resolution image, preprocess its pixel values ​​to stabilize the training process;

[0020] S4. Use a 3×3 convolutional layer to perform shallow feature extraction on the input, map the original pixel information into a high-dimensional feature representation, and obtain shallow features containing basic structural information and preliminary texture information.

[0021] S5. Use global semantic modeling and local region focusing strategies to extract deep features from images;

[0022] S6. Perform residual fusion between deep and shallow features to supplement basic structural information and prevent loss of details;

[0023] S7. Upsample the fused features to generate the final high-resolution face image;

[0024] S8. Perform inverse normalization on the generated face image and output the final high-resolution face reconstruction result.

[0025] S9. Construct a generative adversarial learning training structure to improve the perception effect of generated face images;

[0026] S10. Gradually fine-tune the training model parameters to obtain good training results.

[0027] Furthermore, S1 includes the following sub-steps:

[0028] S11. Obtain the publicly available experimental dataset CELEBA for human face images, which contains human face images with variations in pose, expression, and lighting.

[0029] S12. Crop the central region of the acquired high-resolution face image, and crop each image into a face image with a size of 128×128 to ensure the integrity and consistency of the main face region.

[0030] S13. Filter the cropped CELEBA dataset and select 168,854 images to form the training dataset, 100 images to form the validation dataset, and 1,000 images to form the test dataset.

[0031] S14. Use MATLAB software to perform cubic interpolation downsampling on the selected high-resolution face images to produce low-resolution face images with a size of 16×16.

[0032] S15. The downsampled low-resolution image I with a size of 16×16 LR Its corresponding high-resolution face image I with dimensions of 128×128 HR Training sample pairs As input, n is the total number of samples.

[0033] Furthermore, S5 includes the following sub-steps:

[0034] S51. Based on acquiring and constructing the training dataset, global semantic modeling is performed using the Mamba state-space model, specifically as follows:

[0035] S511. Adaptive feature enhancement is performed on the input features in both the spatial and frequency domains; in the spatial domain, the 8-neighbor Laplacian operator is combined. Multi-directional nonlinear texture capture is performed using a gated modulation mechanism, and a Fast Fourier Transform (FFT) mechanism is used in the frequency domain. Image features are converted to the frequency domain for display modeling and modulation;

[0036] S512. Using the Semantic Guided Neighborhood (SGN) method, the enhanced features are semantically similar and rearranged to allow pixels that are semantically similar but spatially discontinuous to be rearranged to obtain information about similar pixels.

[0037] S513. Use state-space equations combined with a cue pooling mechanism for global semantic modeling. The cue pooling mechanism injects cue information into the state-space output equation. The update aims to mitigate the limitation of causal modeling in achieving full-map perception. Among these, x... i For input information, h i Let C and D be the state representations, and let C and D be the state update matrices, respectively.

[0038] S514. Unfold the sequence of the semantically guided neighborhood and perform subsequent processing.

[0039] S515. Fuse the features before and after global semantic modeling using residual connections;

[0040] S52. Use the moving window attention mechanism to focus on local areas, specifically:

[0041] S521. Modify the position encoding embedding method of attention, extending the rotation position encoding from 1D sequence to two-dimensional space. ;in The phase rotation rate is then calculated using the above formula to determine the rotation phase angle, followed by a linear combination.

[0042] S522, For phase angles that simultaneously contain horizontal and vertical coordinate information Constructing the rotation matrix of rotation encoding ;

[0043] S523. Apply phase rotation to the attention Query vector (Q) and Key vector (K) respectively to directly encode the position information into the vectors, thereby obtaining the Q' vector and K' vector containing the position information; Attention scores are calculated using query vectors and key vectors with spatial location information to enhance the model's utilization of pixel spatial information.

[0044]

[0045] S524. Perform semantic relevance filtering on the calculated attention scores, and select attention scores based on semantic relevance. The first K vectors are retained proportionally, where Ratio is the retention ratio and N is the total number of vectors calculated. ;

[0046] S525. For parts with low semantic relevance, they are directly masked to enhance the model's ability to focus on key regions. ;

[0047] S526. Use residual connections to fuse the features before and after local key focusing.

[0048] Furthermore, S9 includes the following sub-steps:

[0049] S91. Using the face super-resolution network of S1-S8 as a generator, the low-resolution face image is mapped to generate the corresponding high-resolution face image.

[0050] S92. Construct a discriminator using a symmetric encoder-decoder structure.

[0051] S921: The encoder downsamples layer by layer through the CSL module, which halves the resolution layer by layer and doubles the number of channels accordingly.

[0052] S922, the decoder recovers the resolution and number of channels by upsampling layer by layer through CSLU;

[0053] S923. The features of the decoder and encoder are spliced ​​layer by layer through skip connections to simultaneously fuse shallow texture and deep semantic information.

[0054] S93. Use the following relative discriminant loss function to optimize the loss and stabilize the training process:

[0055]

[0056] in, For real samples, To generate samples, D(·) is the relative truth score output by the discriminator.

[0057] The beneficial effects of the technical solution of this invention are as follows:

[0058] (1) By introducing an efficient global sequence modeling mechanism, the problem of limited receptive field of traditional convolutional neural networks can be effectively overcome, and the consistency of the overall structure of the face and the accuracy of the layout of facial features can be improved.

[0059] (2) Using Mamaba for global modeling has low computational complexity and can maintain good computational efficiency when processing high-resolution features, avoiding the problem of excessive computation and storage overhead caused by traditional self-attention mechanisms.

[0060] (3) The introduction of the prompt pool mechanism and semantic similarity reordering effectively alleviates the defects of "sequential scanning and inability to perceive the whole graph" caused by Mamba causal modeling.

[0061] (4) By designing a spatial-frequency parallel adaptive feature enhancement module, features are jointly enhanced in the spatial domain and frequency domain to make up for the lack of local details and high-frequency textures in the global modeling process.

[0062] (5) By expanding the pixel encoding method during the serialization modeling process, a two-dimensional spatial rotation position encoding mechanism is formed, which effectively supplements the problem of insufficient spatial position information in serial modeling and enhances the model's ability to express the spatial structure and geometric relationship of the face.

[0063] (6) By introducing semantic sparse modeling and feature reordering strategies, the model can prioritize the regions that are more critical to face reconstruction during the modeling process, reduce interference from irrelevant regions, and improve the utilization efficiency of limited modeling resources.

[0064] (7) A discriminator based on the U-NET symmetric structure was designed, which effectively integrates shallow features and deep semantic information, stabilizes the training process, and optimizes the perception effect of the generated image.

[0065] (8) By combining reconstruction loss, perception loss and adversarial loss for optimization, the perceptual quality and visual realism of the generated results are effectively improved while ensuring pixel-level reconstruction accuracy. Attached Figure Description

[0066] Figure 1 This is the overall architecture of the MambaSF-GAN of the present invention;

[0067] Figure 2 This is the adaptive feature-enhanced state space module of the present invention;

[0068] Figure 3 This is the space-frequency adaptive feature enhancement module of the present invention. Detailed Implementation

[0069] The specific technical solutions of the present invention will be described with reference to the embodiments.

[0070] This invention addresses the critical issue of face image super-resolution in extremely low-resolution and high-magnification scenarios, proposing a face image super-resolution method that combines the Mamba state-space model with generative adversarial networks. This scheme effectively captures the contour structure and semantic information of the face through global semantic modeling and local key region focusing strategies, and further optimizes the perceptual effect of the super-resolution image through adversarial learning. Its overall architecture is as follows: Figure 1 As shown.

[0071] This invention provides a high-magnification face super-resolution method based on enhanced Mamba and adversarial learning, the technical solution of which includes the following steps:

[0072] S1. Obtain and construct the training dataset;

[0073] S11. Obtain the publicly available experimental dataset CELEBA for human face images. The dataset contains a large number of human face images with variations in pose, expression, and lighting.

[0074] S12. Crop the central region of the acquired high-resolution face images, cropping each image into a face image of size 128×128 to ensure the integrity and consistency of the main face region.

[0075] S13. Filter the cropped CELEBA dataset, selecting 168,854 images to form the training dataset, 100 images to form the validation dataset, and 1,000 images to form the test dataset.

[0076] S14. Use MATLAB software to perform cubic interpolation downsampling on the selected high-resolution face images to create low-resolution face images with a size of 16×16.

[0077] S15. The downsampled low-resolution image I with a size of 16×16 LR Its corresponding high-resolution face image I with dimensions of 128×128 HR Training sample pairs As input, n is the total number of samples.

[0078] S2. Perform a horizontal flip operation on the low-resolution–high-resolution paired image samples, and randomly perform rotation transformations of 90°, 180°, and 270° to expand the number of training samples and improve the generalization ability of the model.

[0079] S3. For the input low-resolution image, perform preprocessing such as normalization on its pixel values ​​to stabilize the training process.

[0080] S4. Use a 3×3 convolutional layer to perform shallow feature extraction on the input, mapping the original pixel information into a high-dimensional feature representation, and obtaining shallow features containing basic structural information and preliminary texture information.

[0081] S5. Use global semantic modeling and local region focusing strategies to extract deep features from images.

[0082] S51, such as Figure 2 As shown, global semantic modeling is performed based on the Mamba state space model.

[0083] S511, Input features such as Figure 3 As shown, adaptive feature enhancement is performed in both the spatial and frequency domains. In the spatial domain, the Laplacian operator of the 8-neighborhood is incorporated. Multi-directional nonlinear texture capture is performed using a gated modulation mechanism, and a Fast Fourier Transform (FFT) mechanism is used in the frequency domain. Image features are converted to the frequency domain for display modeling and modulation.

[0084] S512. Using the Semantic Guided Neighborhood (SGN) method, the enhanced features are semantically similar and rearranged to allow pixels that are semantically similar but spatially discontinuous to be rearranged to obtain information about similar pixels.

[0085] S513. Use state-space equations combined with a cue pooling mechanism for global semantic modeling. The cue pooling mechanism injects cue information into the state-space output equation. The update aims to mitigate the limitation of causal modeling in achieving full-map perception. Among these, x... i For input information, h iLet C and D be the state representations, and let C and D be the state update matrices, respectively.

[0086] S514. Unfold the sequence of the semantically guided neighborhood and perform subsequent processing.

[0087] S515. Use residual connections to fuse the features before and after global semantic modeling.

[0088] S52. Use the moving window attention mechanism to focus on local areas.

[0089] S521. Modify the position encoding embedding method of attention, extending the rotation position encoding from 1D sequence to two-dimensional space. ;in The phase rotation rate is then calculated using the above formula to determine the rotation phase angle, followed by a linear combination.

[0090] S522, For phase angles that simultaneously contain horizontal and vertical coordinate information Constructing the rotation matrix of rotation encoding ;

[0091] S523. Apply phase rotation to the query vector Q and key vector K of the attention respectively to directly encode the position information into the vectors, thereby obtaining the Q' vector and K' vector containing the position information; Attention scores are calculated using query vectors and key vectors with spatial location information to enhance the model's utilization of pixel spatial information.

[0092]

[0093] S524. Perform semantic relevance filtering on the calculated attention scores, and select attention scores based on semantic relevance. The first K vectors are retained proportionally, where Ratio is the retention ratio and N is the total number of vectors calculated.

[0094] S525. For parts with low semantic relevance, they are directly masked to enhance the model's ability to focus on key regions. ;

[0095] S526. Use residual connections to fuse the features before and after local key focusing.

[0096] S6. Perform residual fusion between deep and shallow features to supplement basic structural information and prevent loss of details.

[0097] S7. Upsample the fused features to generate the final high-resolution face image.

[0098] S8. Perform inverse normalization and other operations on the generated face image to output the final high-resolution face reconstruction result.

[0099] S9. Construct a generative adversarial learning training structure to further improve the perception effect of generated face images.

[0100] S91. Using the face super-resolution network described in S1-S8 as a generator, the low-resolution face image is mapped to generate the corresponding high-resolution face image.

[0101] S92, Construction as follows Figure 1 The discriminator shown below uses a symmetrical encoder-decoder structure.

[0102] S921: The encoder downsamples layer by layer through the CSL module, which halves the resolution layer by layer and doubles the number of channels accordingly.

[0103] S922, the decoder recovers the resolution and number of channels by upsampling layer by layer through CSLU.

[0104] S923. The features of the decoder and encoder are spliced ​​layer by layer through skip connections to simultaneously fuse shallow texture and deep semantic information.

[0105] S93. Use the following relative discriminant loss function to optimize the loss and stabilize the training process:

[0106]

[0107] in, For real samples, To generate samples, D(·) is the relative truth score output by the discriminator.

[0108] S10. Gradually fine-tune the training model parameters to obtain good training results.

Claims

1. A high-magnification face super-resolution method based on enhanced Mamba and adversarial learning, characterized in that, Includes the following steps: S1. Obtain and construct the training dataset; S2. Perform a horizontal flip operation on the low-resolution (LR) – high-resolution (HR) paired image samples, and randomly perform rotation transformations of 90°, 180° and 270° to expand the number of training samples and improve the generalization ability of the model. S3. For the input low-resolution image, preprocess its pixel values ​​to stabilize the training process; S4. Use a 3×3 convolutional layer to perform shallow feature extraction on the input, map the original pixel information into a high-dimensional feature representation, and obtain shallow features containing basic structural information and preliminary texture information. S5. Use global semantic modeling and local region focusing strategies to extract deep features from images; S6. Perform residual fusion between deep and shallow features to supplement basic structural information and prevent loss of details; S7. Upsample the fused features to generate the final high-resolution face image; S8. Perform inverse normalization on the generated face image and output the final high-resolution face reconstruction result. S9. Construct a generative adversarial learning training structure to improve the perception effect of generated face images; S10. Gradually fine-tune the training model parameters to obtain good training results.

2. The high-magnification face super-resolution method based on enhanced Mamba and adversarial learning according to claim 1, characterized in that, S1 includes the following sub-steps: S11. Obtain the publicly available experimental dataset CELEBA for human face images, which contains human face images with variations in pose, expression, and lighting. S12. Crop the central region of the acquired high-resolution face image, and crop each image into a face image with a size of 128×128 to ensure the integrity and consistency of the main face region. S13. Filter the cropped CELEBA dataset and select 168,854 images to form the training dataset, 100 images to form the validation dataset, and 1,000 images to form the test dataset. S14. Use MATLAB software to perform cubic interpolation downsampling on the selected high-resolution face images to produce low-resolution face images with a size of 16×16. S15. The downsampled low-resolution image I with a size of 16×16 LR Its corresponding high-resolution face image I with dimensions of 128×128 HR Training sample pairs As input, n is the total number of samples.

3. The high-magnification face super-resolution method based on enhanced Mamba and adversarial learning according to claim 1, characterized in that, S5 includes the following sub-steps: S51. Based on acquiring and constructing the training dataset, global semantic modeling is performed using the Mamba state-space model, specifically as follows: S511. Adaptive feature enhancement is performed on the input features in both the spatial and frequency domains; in the spatial domain, the 8-neighbor Laplacian operator is combined. Multi-directional nonlinear texture capture is performed using a gated modulation mechanism, and a Fast Fourier Transform (FFT) mechanism is used in the frequency domain. Image features are converted to the frequency domain for display modeling and modulation; S512. Using the Semantic Guided Neighborhood (SGN) method, the enhanced features are semantically similar and rearranged to allow pixels that are semantically similar but spatially discontinuous to be rearranged to obtain information about similar pixels. S513. Use state-space equations combined with a cue pooling mechanism for global semantic modeling. The cue pooling mechanism injects cue information into the state-space output equation. The update aims to mitigate the inability to perceive the entire graph due to causal modeling; among which x i For input information, h i Let C and D be the state representations, and let C and D be the state update matrices, respectively. S514. Unfold the sequence of the semantically guided neighborhood and perform subsequent processing. S515. Fuse the features before and after global semantic modeling using residual connections; S52. Use the moving window attention mechanism to focus on local areas, specifically: S521. Modify the position encoding embedding method of attention, extending the rotation position encoding from 1D sequence to two-dimensional space. ;in The phase rotation rate is then calculated using the above formula to determine the rotation phase angle, followed by a linear combination. S522, For phase angles that simultaneously contain horizontal and vertical coordinate information Constructing the rotation matrix of rotation encoding ; S523. Apply phase rotation to the attention Query vector Q and Key vector K respectively to directly encode the position information into the vectors, thereby obtaining Q' vector and K' vector containing position information; Attention scores are calculated using query vectors and key vectors with spatial location information to enhance the model's utilization of pixel spatial information. ； S524. Perform semantic relevance filtering on the calculated attention scores, and select attention scores based on semantic relevance. The first K vectors are retained proportionally, where Ratio is the retention ratio and N is the total number of vectors calculated. ; S525. For parts with low semantic relevance, they are directly masked to enhance the model's ability to focus on key regions. ; S526. Use residual connections to fuse the features before and after local key focusing.

4. The high-magnification face super-resolution method based on enhanced Mamba and adversarial learning according to claim 1, characterized in that, S9 includes the following sub-steps: S91. Using the face super-resolution network of S1-S8 as a generator, the low-resolution face image is mapped to generate the corresponding high-resolution face image. S92. Construct a discriminator using a symmetric encoder-decoder structure. S921: The encoder downsamples layer by layer through the CSL module, which halves the resolution layer by layer and doubles the number of channels accordingly. S922, the decoder recovers the resolution and number of channels by upsampling layer by layer using CSLU; S923. The features of the decoder and encoder are spliced ​​layer by layer through skip connections to simultaneously fuse shallow texture and deep semantic information. S93. Use the following relative discriminant loss function to optimize the loss and stabilize the training process: ； in, For real samples, To generate samples, D(·) is the relative truth score output by the discriminator.

Images