Real-time blind face restoration method and device based on identity constraint and frequency domain enhancement

Abstract

This invention provides a real-time face blind restoration method and apparatus based on identity constraints and frequency domain enhancement, relating to the field of image processing technology. The method includes: S1, generating an identity code through CLIP and ArcFace fusion, extracting spatial structural features using ControlNet, and obtaining high-frequency components using learnable wavelet decomposition; S2, mapping the input image to a latent space, and constructing a conditional vector by fusing identity, spatial, and high-frequency features; S3, iterative denoising based on a consistency model, and superimposing lightweight wavelet residual correction to enhance high-frequency details; S4, dynamically allocating weights according to high-frequency energy, and reconstructing a multi-scale fused image using inverse wavelet transform. This invention solves the problems of identity distortion, detail loss, and low computational efficiency in the restoration of low-quality face images. Through identity-space-frequency domain ternary constraints, lightweight residual correction, and degradation perception enhancement, it improves the semantic consistency, detail fidelity, and real-time performance of the restoration results.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a real-time face blindness repair method and apparatus based on identity constraints and frequency domain enhancement. Background Technology

[0002] Traditional blind face restoration methods often face three major challenges: identity feature drift (restoration results are inconsistent with the real identity), high-frequency detail blurring (insufficient texture and edge restoration), and low computational efficiency (difficult to meet real-time requirements). Existing technologies, such as GAN-based generative methods, can improve visual quality, but they are easily affected by degradation types (e.g., artifacts are easily generated when blurred or mixed with noise), and rely on prior degradation information (non-blind restoration), resulting in poor generalization in real-world scenarios. Diffusion-based methods can generate high-quality details, but require many iterations (usually 50+ steps) and have high computational costs, making them difficult to process in real time. In addition, most methods lack explicit constraints on identity features, causing the restoration results to deviate from the target identity, especially with low-quality input (such as surveillance videos and compressed images), where identity information is severely lost.

[0003] In the field of security monitoring, blurred faces need to be quickly restored to reveal identifiable features; in the digitization of cultural heritage, old photos need to be restored while preserving the true appearance of the people; and in mobile applications, lightweight models are needed to enhance selfie images in real time. Therefore, there is an urgent need for a restoration solution that does not require degradation priors, has strong identity constraints, allows for controllable enhancement of details, and is computationally efficient. Summary of the Invention

[0004] To address the aforementioned technical problems in related technologies, this invention proposes a real-time face blindness repair method and apparatus based on identity constraints and frequency domain enhancement.

[0005] In a first aspect, the present invention provides a real-time face blindness repair method based on identity constraints and frequency domain enhancement, comprising the following steps:

[0006] S1. Multimodal feature extraction: Extracting low-quality input image I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh N is the number of reference images;

[0007] S2. Mixed Conditional Latent Space Initialization: The low-quality input image I is initialized using the VQGAN encoder E. LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. idSpatial structural features f s and the high-frequency detail component F in the diagonal direction hh Construct a mixed conditional vector c;

[0008] S3. Multi-step generation with frequency domain enhancement: Iterative denoising of the initial latent variable z2 and the mixing condition vector c is performed in the latent space using a consistency model, and wavelet residual correction is superimposed to enhance high-frequency details and generate a multi-scale restored image.

[0009] S4. Wavelet Domain Adaptive Fusion: Image Recovery Based on Multi-Scale High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse .

[0010] Specifically, step S1 includes the following steps:

[0011] S11. Obtain a low-quality input image. and reference image set Where H is the height of the input image, W is the width of the input image, 3 represents the RGB three channels, and N is an integer greater than or equal to 1; For the i-th reference image, H ref For the reference image height, W ref The width is the reference image width;

[0012] S12, Reference image The input is fed into the pre-trained CLIP model to extract global semantic features f CLIP The reference image To obtain from the reference image set The i-th image selected from the list;

[0013] S13, Reference image The input is fed into a pre-trained ArcFace model to extract the identity feature vector f. ArcFace ;

[0014] S14. Transfer global semantic features f CLIP With identity feature vector f ArcFace The joint feature f is generated through channel splicing operations. joint After processing by a multilayer perceptron, the output identity encoding vector c is obtained. id ;

[0015] S15, Input low-quality image I LQ Input a spatial encoder based on the ControlNet architecture, combined with the identity encoding vector c id Generate spatial structure features f s ;

[0016] S16. The low-quality input image I is processed using a learnable wavelet transform module. LQ Decomposition yields the high-frequency detail components F in the diagonal direction. hh .

[0017] Specifically, step S2 includes the following steps:

[0018] S21. Transform the low-quality input image I using the VQGAN encoder E. LQ Mapping to the latent space yields the initial latent variables. Where E() is the encoder E-mapping function; H is the image height, W is the image width, and 1024 is the number of latent variable channels;

[0019] S22, Convert the identity encoding vector c id Flattened spatial structure characteristics s ) and the high-frequency detail components in the diagonal direction after flattening. hh The concatenation is performed along the channel dimension to generate a joint condition vector c. joint Then, it is compressed into a joint condition vector c through a fully connected layer. joint The dimension is used to obtain the mixing condition vector c = Linear(c joint The final output consists of the initial latent variable z2 and the mixing condition vector c; these are then concatenated along the channel dimension using the following formula:

[0020]

[0021] Flatten() is the expansion function; Linear() is the linear mapping function corresponding to the fully connected layer.

[0022] Specifically, step S3 includes:

[0023] Initial latent variables and mixed condition vector As input, a consensus model is used to perform multi-step iterations, and wavelet residual correction terms are incorporated during the iteration process to generate the initial latent variable z for denoising. t-1 ; and in generating the initial latent variable z t-1 During the process, a multi-scale decoder D k From the latent variable z T-k Generate multi-scale restored images

[0024] The multi-scale decoder D kBased on the SIMO architecture, it contains k levels of upsampling blocks, each level containing transposed convolution and LeakyReLU activation function; where T is the total number of iterations, and T takes the value of 4; k = 1, 2, 3; The decoding results correspond to the highest resolution z3, the intermediate resolution z2, and the lowest resolution z1, respectively.

[0025] Specifically, step S4 includes:

[0026] Reconstructing image sets at multiple scales and from the diagonal high-frequency detail components Extracted high-frequency components at various scales As input, the L1 norm of the high-frequency components at each scale is first calculated as the high-frequency energy. Then based on the high-frequency energy The fusion weights λ are dynamically allocated using the Softmax function. k Reconstructed images at each scale Perform a learnable discrete wavelet transform to decompose it into low-frequency subbands. and high-frequency subband Diagonal high-frequency detail components With high-frequency subband The horizontal and vertical components are spliced ​​together to form the reconstructed high-frequency subband. The reconstructed image is then generated using inverse wavelet transform: Finally, according to the fusion weight λ k Weighted fusion reconstruction of images to generate multi-scale fused images

[0027] Specifically, the method further includes:

[0028] S5. Dynamic Post-processing and Output: Multi-scale Fusion Image Based on Gaussian Blur fuse The sharpening intensity is adjusted to generate a low-frequency substrate image I. blur .

[0029] Specifically, step S5 includes:

[0030] Multi-scale fused images When performing Gaussian blurring, a 5×5 Gaussian kernel is first generated. Its weight values ​​are calculated using a two-dimensional Gaussian function, which is shown below:

[0031]

[0032] Where G(x, y) represents the weight values ​​of relative coordinates x and y within the Gaussian kernel; σ is the standard deviation, with a value of 1.0, and the origin is located at the center of the Gaussian kernel; the weights of each point within the Gaussian kernel are normalized and summed to 1, with the specific values ​​as follows:

[0033]

[0034] Multi-scale fused image I fuse Each RGB channel independently performs a zero-padding 2D convolution operation. The padding width of the 2D convolution operation is 2, the stride is 1, and pixel values ​​outside the boundary are treated as 0. The formula is expressed as follows:

[0035]

[0036] Where i and j represent the multi-scale fused image I, respectively. fuse The rows and columns are used to index the position of pixels, i∈[1,H],j∈[1,W]; c is the channel index, c=1,2,3 correspond to the red, green and blue color channels respectively; dx and dy are the offsets used for two-dimensional convolution operations; dx represents the horizontal offset, with a value range of -2 to 2; dy represents the vertical offset, with a value range of -2 to 2.

[0037] Specifically, step S12 includes:

[0038] Reference image The input to the pre-trained CLIP model is first standardized to adjust the resolution to A×A pixels, and then the first input tensor is generated by mean-variance normalization. Subsequently, the image is segmented into a B×B sequence of image blocks by the C-layer Transformer block of the image encoder. Each block is linearly projected and mapped to a C-dimensional vector. Global semantic information is aggregated through a self-attention mechanism. Finally, the feature vector corresponding to the CLS label of the last Transformer block is extracted, mapped to E-dimensional space through a fully connected layer, and the global semantic feature f is generated. CLIP .

[0039] Specifically, step S13 includes:

[0040] Reference image The pre-trained ArcFace model is first input, and then normalization is performed to adjust the image resolution to F×F pixels. Then, mean-variance normalization is performed to generate the second input tensor. Then the second input tensor Through the forward propagation of the pre-trained ArcFace model's backbone network, the feature vector f is extracted sequentially through an initial convolutional layer, a batch normalization (BN) layer, and a ReLU activation layer, followed by four residual blocks, and finally extracted after the last layer of global average pooling. base ; to feature vector Input the fully connected layer corresponding to the ArcFace loss function, apply additive angular boundary calculation to output the normalized identity feature vector. The backbone network is ResNet-100.

[0041] Secondly, the present invention provides a real-time face blindness repair device based on identity constraints and frequency domain enhancement, which, based on the real-time face blindness repair method based on identity constraints and frequency domain enhancement described in the first aspect above, includes the following units:

[0042] A multimodal feature extraction unit is used to extract low-quality input images I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh N is the number of reference images;

[0043] Latent spatial initialization unit, used to process low-quality input image I through VQGAN encoder E LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Construct a mixed conditional vector c;

[0044] The frequency domain enhancement unit is used to perform latent space iterative denoising on the initial latent variable z2 and the mixing condition vector c through a consistency model, and then superimpose wavelet residual correction to enhance high-frequency details and generate a multi-scale restored image.

[0045] Wavelet domain fusion unit for image reconstruction based on multiple scales High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse ;

[0046] The sharpening unit is used for multi-scale fusion image I based on Gaussian blur. fuse The sharpening intensity is adjusted to generate a low-frequency substrate image I. blur .

[0047] This invention provides a real-time face blindness repair method based on identity constraints and frequency domain enhancement, including: S1, multimodal feature extraction: extracting low-quality input image I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh S2, Mixed Conditional Latent Space Initialization: The low-quality input image I is initialized using the VQGAN encoder E. LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Constructing the hybrid conditional vector c; S3, multi-step generation with frequency domain enhancement: using a consistency model to perform latent space iterative denoising on the initial latent variable z2 and the hybrid conditional vector c, and superimposing wavelet residual correction to enhance high-frequency details to generate a multi-scale restored image. S4. Wavelet Domain Adaptive Fusion: Image Recovery Based on Multi-Scale High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse This invention innovatively combines multimodal identity features (CLIP semantics + ArcFace discriminative features) with a frequency domain enhancement mechanism. Specifically, it uses identity-space-frequency domain ternary constraints, accelerates generation through a lightweight consistency model (LCM), and introduces dynamic residual correction to achieve precise control of high-frequency details. Ultimately, it breaks through the traditional method's dilemma of balancing identity consistency, detail fidelity, and real-time performance, providing a highly available solution for practical scenarios.

[0048] Furthermore, this invention also adjusts the sharpening intensity of the multi-scale fused image through Gaussian blurring, thereby suppressing high-frequency components (such as noise and sharp edges) in the image and generating a low-frequency base image. By retaining only global illumination and smoothing region information, the image becomes smoother, which helps to remove unnecessary noise while preserving image details, thereby improving the overall appearance and processing quality of the image. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A schematic diagram of a real-time face blindness repair method based on identity constraints and frequency domain enhancement provided in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a real-time face blindness repair device based on identity constraints and frequency domain enhancement provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a real-time face blindness repair device based on identity constraints and frequency domain enhancement, provided in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0054] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] refer to Figure 1 This embodiment provides a real-time face blindness repair method based on identity constraints and frequency domain enhancement, including the following steps:

[0057] S1. Multimodal feature extraction: Extracting low-quality input image I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh N is the number of reference images;

[0058] S11. Obtain a low-quality input image. and reference image set Where H is the height of the input image, W is the width of the input image, 3 represents the RGB three channels; N is the number of reference images; N is an integer greater than or equal to 1;

[0059] For the i-th reference image, H ref For the reference image height, W ref The width is the reference image width;

[0060] It is understood that this embodiment involves The format indicates that the dimension of variable X is Y; for example... Indicate I LQ The dimensions are H×W×3.

[0061] The reference image set N images selected from different angles or lighting conditions for the same target person; from the reference image set The i-th image is selected as the reference image. The selection rules can be random or selected according to requirements;

[0062] S12, Reference image The input is fed into the pre-trained CLIP model to extract global semantic features f CLIP The reference image To obtain from the reference image set The i-th image selected from the list;

[0063] Step S12 specifically includes:

[0064] Reference image The input to the pre-trained CLIP model is first standardized to adjust the resolution to A×A pixels, and then the first input tensor is generated by mean-variance normalization. Then the first input tensor The image is segmented into a B×B sequence of image blocks by processing the C-layer Transformer block of the pre-trained CLIP model's visual encoder. Each block is linearly projected and mapped to a C-dimensional vector. Global semantic information is aggregated through a self-attention mechanism. Finally, the feature vector corresponding to the CLS label of the last Transformer block is extracted and mapped to E-dimensional space through a fully connected layer to generate global semantic features f. CLIP ;

[0065] Furthermore, L2 normalization is used to constrain the global semantic features f. CLIP The characteristic distribution;

[0066] The dimension of the feature vector is D; the L2 normalization formula is: The mean of the normalized mean and variance is [0.4815, 0.4578, 0.4082], and the variance is [0.2686, 0.2613, 0.2758]; the global semantic feature f CLIP The dimension is 512; where A = 224, B = 16, C = 12, D = 768, and E = 512.

[0067] In this embodiment, the pre-trained CLIP model consists of two branches: a visual encoder and a text encoder. These two branches achieve cross-modal semantic alignment through comparative learning. The visual encoder uses a Vision Transformer (ViT-B / 16) architecture. The input image is first segmented into 16×16 pixel non-overlapping blocks (196 blocks in total). Each block is linearly projected into a 768-dimensional vector and superimposed with a learnable positional encoding. This is then processed through a 12-layer Transformer module to finally extract the 768-dimensional global feature vector corresponding to the [CLS] marker. The text encoder is based on a 12-layer standard Transformer model. The input text is processed by BPE (Byte-Pair)... The word segmentation (Encoding) is a sequence of maximum length 77 (including start symbol [SOS] and end symbol [EOS]). After passing through a 512-dimensional word embedding layer, it is encoded layer by layer through a self-attention mechanism, finally extracting the 512-dimensional text features at the [EOS] marker positions. Visual and text features are mapped to a shared 512-dimensional embedding space through independent linear projection layers, allowing the cosine similarity of cross-modal features to be directly used for contrastive learning. This structure captures local details and global semantics of the image through block encoding and hierarchical attention mechanisms, and combines the contextual modeling capability of the text sequence to achieve efficient alignment of image and text in a unified semantic space. During the training phase, the input image of the model is uniformly scaled to 224×224 resolution, and the text is input after normalization. After independent forward computation in two branches, the feature space distribution is optimized through symmetric cross-entropy loss, ultimately forming a cross-modal semantic representation capability with strong generalization.

[0068] CLIP Model: CLIP is a deep learning model proposed by OpenAI that is trained by pairing images with relevant text descriptions. The core idea of ​​the CLIP model is to learn a visual and linguistic representation such that the representations of the same concept in the visual and text domains are close in feature space. In this way, the CLIP model can be used for a variety of visual recognition tasks without needing to fine-tune for specific tasks.

[0069] Based on the ViT-B / 16 architecture: This refers to the fact that the visual encoder part in the CLIP model uses the base version of VisionTransformer (ViT-B), and the image is segmented into 16x16 pixel blocks ( / 16). ViT-B / 16 is a powerful visual feature extractor capable of handling complex relationships in images.

[0070] Mean-variance normalization, also known as Z-score standardization, is a data preprocessing method used to adjust the numerical range of a dataset so that the mean of each feature is 0 and the standard deviation is 1. This method can give features of different magnitudes the same scale, thereby eliminating the influence of dimensions and improving the performance of certain machine learning algorithms.

[0071] Self-attention, also known as internal attention, is an attention mechanism that associates different positions within a single sequence to compute a representation of the same sequence. This mechanism allows the model to dynamically adjust the level of attention given to each element when processing sequence data, thereby capturing complex dependencies within the sequence.

[0072] The core of the self-attention mechanism lies in the fact that it does not rely on external information, but rather on the interaction and integration of information among elements within the sequence. This means that for each element in the sequence, the self-attention mechanism calculates the relevance of that element to all other elements in the sequence, generating a weighted representation where the weights reflect the relationships between elements.

[0073] L2 normalization refers to scaling the eigenvectors to the unit L2 norm, that is, the L2 norm (Euclidean norm) of the eigenvectors is equal to 1.

[0074] S13, Reference image The input is fed into a pre-trained ArcFace model to extract the identity feature vector f. ArcFace ;

[0075] Step S13 specifically includes:

[0076] Reference image The pre-trained ArcFace model is first input, and then normalization is performed to adjust the image resolution to F×F pixels. Then, mean-variance normalization is performed to generate the second input tensor. Then the second input tensor Through the forward propagation of the pre-trained ArcFace model's backbone network, the feature vector f is extracted sequentially through an initial convolutional layer, a batch normalization (BN) layer, and a ReLU activation layer, followed by four residual blocks, and finally extracted after the last layer of global average pooling. base ; to feature vector Input the fully connected layer corresponding to the ArcFace loss function, apply additive angular boundary calculation to output the normalized identity feature vector. The backbone network is ResNet-100;

[0077] The mean-variance normalization formulas mentioned in steps S12 and S13 are as follows:

[0078]

[0079] Where x, y are the image pixel coordinates, c is the color channel index, and μ c σ is the mean of the color channels. c Let be the variance of the color channels, and ò be the numerical stability constant.

[0080] The mean-variance normalization parameter is processed independently for each channel, that is, the mean and variance are defined separately for the three color channels of the RGB image.

[0081] In this embodiment, the mean of the color channels with mean-variance normalization is [0.5, 0.5, 0.5], and the variance of the color channels is [0.5, 0.5, 0.5]. The initial convolutional layer includes a 7×7 convolutional kernel with a stride of 2 and 64 output channels. Each of the four residual blocks contains a 3-layer Bottleneck structure, with the number of channels increasing progressively to 512. The weight matrix of the fully connected layer... The L2 norm constraint of the eigenvector is 1; the eigenvector f base With identity feature vector The dimension is 512; F = 12.

[0082] The pre-trained ArcFace model is a deep learning model specifically designed for face recognition. It uses ResNet-100 as a feature extractor and is optimized with the ArcFace loss function to generate highly discriminative facial feature representations. Such a model can be used directly for face recognition tasks or as the basis for transfer learning of other models.

[0083] The pre-trained ArcFace model is optimized during the training phase by maximizing inter-class differences and minimizing intra-class differences. It can extract discriminative identity features without fine-tuning and is used for the joint construction of subsequent identity encoding.

[0084] ArcFace is a loss function used for face recognition tasks. It is based on deep learning-based feature embedding and classification techniques. ArcFace increases inter-class differences by introducing additive angular margins, making the distance between different classes in the feature space larger, thereby improving recognition accuracy.

[0085] The ArcFace loss function is an improvement on the traditional softmax loss function. Its purpose is to make features of the same category more compact and features of different categories more dispersed during training.

[0086] ResNet-100 refers to a residual network architecture with 100 convolutional layers. ResNet is a deep neural network architecture that addresses the vanishing gradient problem in deep network training by introducing residual learning.

[0087] ResNet-100 is used as the backbone network to extract high-level features of images, which are then fed into the ArcFace loss function for classification.

[0088] Global Average Pooling (GAP) is a pooling technique used in deep learning networks, especially in Convolutional Neural Networks (CNNs). Its main function is to average the pixel values ​​across all pixels in each feature map, resulting in a single numerical value that can be considered a global representation of that feature map.

[0089] Additive Angular Margin (AAM) is a loss function strategy used in deep learning for face recognition tasks, particularly when using deep feature embeddings and cosine similarity metrics. It aims to improve the model's discriminative power by adding angular boundaries to the feature space, thus bringing embeddings of similar categories closer together and separating embeddings of different categories.

[0090] L2 norm constraint, also known as weight decay or Tikhonov regularization, is a common regularization method used in machine learning and deep learning model training. It reduces overfitting by adding a term to the loss function that is proportional to the L2 norm of the model parameters (i.e., the Euclidean norm of the weight vector) to penalize large weight values.

[0091] S14. Transfer global semantic features f CLIP With identity feature vector f ArcFace The joint feature f is generated through channel splicing operations. joint After processing by a multilayer perceptron, the output identity encoding vector c is obtained. id ;

[0092] The multilayer perceptron includes two fully connected layers and a GELU activation function; the channel splicing operation is as follows: Operations; in this embodiment, the identity encoding vector c idThe dimension is 1024;

[0093] In another possible implementation, the reference image set can be... For each reference image, operations S12 and S13 are performed separately to independently extract global semantic features. With identity feature vector Subsequently, a joint feature f is generated by fusing features from all reference images and averaging them element-wise. joint Then, it is mapped to the final identity encoding vector c through a multilayer perceptron. id The formula for element-wise averaging of the feature fusion is as follows:

[0094]

[0095] in, The global semantic features of the i-th reference image; Let be the identity feature vector of the i-th reference image.

[0096] S12 and S13 process each reference image independently (i.e., for the i-th reference image, S12 extracts...). S13 Extraction Both correspond strictly to the same input image, avoiding misalignment of features across images.

[0097] S15, Input low-quality image I LQ Input a spatial encoder based on the ControlNet architecture, combined with the identity encoding vector c id Generate spatial structure features f s ;

[0098] The spatial encoder based on the ControlNet architecture includes an initial convolutional layer and four downsampling blocks. The initial convolutional layer includes a 3×3 convolutional kernel with a stride of 1, padding of 1, and 64 channels. The downsampling blocks include a 3×3 convolution and a ReLU activation function with a stride of 2 and padding of 1. The spatial structure features...

[0099] Step S15 specifically includes:

[0100] Low-quality input image The input is a spatial encoder based on the ControlNet architecture. First, the input is transformed into an initial feature map through an initial convolutional layer. The initial feature map then Combined with the identity encoding vector c id The spatial structure features are processed sequentially through four downsampling blocks.

[0101] Specifically, the subsequent initial feature map Combined with the identity encoding vector c id The spatial structure feature f is output after being processed by four downsampling blocks in sequence. s Specifically:

[0102] Each downsampling block consists of a 3×3 convolutional kernel and a ReLU activation function. The kernel stride is 2, and the padding is 1. The spatial resolution is progressively reduced while the number of channels is increased: the first downsampling block compresses the feature map size from H×W×64 to... The second to fourth downsampling blocks output feature map sizes of [size missing]. In each downsampling stage, the identity encoding vector c id The feature map is dynamically projected to the current feature channel dimension through a fully connected layer (e.g., 256 dimensions for the second block), then expanded to the same size as the feature map through spatial copying. Finally, the feature map for each level is obtained by element-wise addition of the residual features to the convolution output features. The formula is expressed as:

[0103] F l =ReLU(Conv(F) l-1 ))+Reshape(Linear(c id ))

[0104] Where F l Let f represent the feature map output by the l-th downsampling block; where ReLU() is the ReLU activation function; Conv() is the convolution function; l = 1, 2, 3, 4 represents the block number; Reshape() represents expanding the projected vector into a spatial feature map; in this embodiment, the feature channels are the number of channels in each downsampling block, with 128 for the first downsampling block and 256 for the second, third, and fourth downsampling blocks. The feature map output by the last downsampling block is used as the spatial structure feature f. s In this embodiment

[0105] A fully connected layer, also known as a linear transformation layer, has the core function of linearly mapping the input vector (or feature map) through a weight matrix and a bias term. Linear() is the linear mapping function corresponding to a fully connected layer.

[0106] By employing a conditional injection mechanism, we ensure the deep integration of identity semantics and spatial structure, enabling f s Simultaneously, the low-level geometric information of the input image (such as the position of facial features) and the identity constraints of the reference image are encoded to provide fine-grained guidance for the subsequent generation stage.

[0107] Conditional injection is a technique that uses dynamic adjustments to the intermediate feature representations of a model to deeply integrate external conditional information (such as identity semantic encoding) with the spatial structural features of an image. Its core is to use parametric mapping and feature manipulation to enable global identity semantic constraints to guide the generation of local details during image reconstruction layer by layer, thereby enhancing identity consistency while maintaining spatial coherence.

[0108] Downsampling is a common operation in convolutional neural networks (CNNs), also known as downsampling or pooling.

[0109] A 3×3 convolution represents a convolution operation with a kernel size of 3 rows and 3 columns. The convolution kernel slides across the input data to generate an output feature map.

[0110] A stride of 2 means that the convolutional kernel moves with a stride of 2 across the input feature map. This means that the convolutional kernel skips 1 pixel each time it moves, which results in the output feature map being smaller than the input feature map.

[0111] Padding to 1: Padding adds extra borders to the edges of the input feature map, usually filled with 0s. Here, padding to 1 means adding a 1-pixel-wide border to each dimension of the input feature map. This prevents the feature map size from shrinking excessively after convolution.

[0112] ReLU activation function: ReLU (Rectified Linear Unit) is an activation function that sets all negative pixels to 0, while leaving positive pixels unchanged. This function is often used to increase the non-linearity of a network, enabling it to learn and simulate more complex functions.

[0113] S16. The low-quality input image I is processed using a learnable wavelet transform module. LQ Decomposition yields the low-frequency approximate component F. ll Horizontal high-frequency detail component F lh Vertical high-frequency detail component F hl and the high-frequency detail component F in the diagonal direction hh .

[0114] Step S16 specifically includes:

[0115] Low-quality input image The input is a learnable wavelet transform module, which first expands the image size to an even resolution using zero-padding to generate a preprocessed image. in,

[0116] Subsequently, four sets of trainable wavelet kernels are used to perform depthwise separable convolution operations: each set of wavelet kernels consists of two convolution kernels with dimensions of 2×2×3×256, denoted as The input image is subjected to downsampling convolution with a stride of 2, corresponding to the decomposition directions of low-frequency approximation, horizontal high-frequency, vertical high-frequency, and diagonal high-frequency, respectively.

[0117]

[0118] Where DW-Conv represents a depthwise separable convolution function, and the output size of each convolution group is [size missing]. After processing with the GELU activation function, the four outputs are grouped according to frequency band type: low-frequency approximate component F ll Directly by Generation, horizontal high-frequency components Vertical high frequency components High-frequency detail components in the diagonal direction ψ k Here are the wavelet kernel parameters; k = 1, 2, 3, 4;

[0119] Furthermore, the wavelet kernel parameter ψ can be optimized through backpropagation during the training process of the wavelet transform module. k And introduce the perfect reconstruction constraint loss function: L recon =‖ILWT(LWT(I LQ ))-I LQ ||1;

[0120] LWT() and ILWT() are the learnable wavelet transform and its inverse transform, respectively, ensuring lossless information during the decomposition process. The final output consists of four components. (Croped to half the original input resolution), where the high-frequency component F hh It serves as the core input for subsequent frequency domain enhancement.

[0121] The Learnable Wavelet Network (LWN) module employs a hierarchical adaptive frequency domain decomposition structure. It performs multi-scale frequency band separation of input features through three levels of learnable low-pass and high-pass wavelet kernels, generating four sub-band components at each level: low frequency (LL), horizontal high frequency (LH), vertical high frequency (HL), and diagonal high frequency (HH). Its core utilizes parameterized 5×5 convolutional kernels to perform wavelet transform. Kernel weights are constrained by orthogonal regularization loss to satisfy compact support orthogonality, ensuring decomposition reversibility. Skip connections between levels fuse low-level high-frequency details (HH) with high-level low-frequency basis (LL), and Sigmoid gating is introduced to dynamically allocate channel weights, suppressing noise bands and enhancing identity-related texture features. This module optimizes the frequency band division boundaries of the wavelet kernels through end-to-end training, achieving degradation-aware frequency domain feature decoupling. Combined with a lightweight convolutional structure, it improves computational efficiency, effectively supporting high-frequency detail reconstruction in image inpainting.

[0122] Depthwise separable convolution is an efficient convolution operation that is popular in deep learning models, especially in lightweight neural network designs for mobile and edge devices. It breaks down the traditional convolution operation into two more efficient steps: depthwise convolution and pointwise convolution.

[0123] Depthwise Convolution: In depthwise convolution, each input channel is convolved individually using a different filter. If the input has C channels, then there will be C independent 3x3 (or other size) filters. Each filter operates only on its corresponding input channel, not all channels. This means that if there are C input channels, C feature maps will be generated.

[0124] Pointwise Convolution: Following depthwise convolution, pointwise convolution takes the output of the depthwise convolution as input and uses 1x1 kernels to combine these feature maps. If F output channels are required, pointwise convolution will use F 1x1xC filters, where C is the number of output channels of the depthwise convolution.

[0125] Step S1 first acquires a low-quality input image. (Where H is the image height, W is the image width, and 3 represents the RGB three channels) and reference image set (N≥1 is the number of reference images), at least one reference image Ref i Extracting global semantic features from pre-trained CLIP models Identity features are extracted using a pre-trained ArcFace model. The two are concatenated via a channel splicing operation (denoted as...). Generate joint features The result is then processed by a multilayer perceptron (MLP) containing two fully connected layers and a GELU activation function, outputting an identity encoding vector. At the same time, low-quality image I LQ The input is a spatial encoder based on the ControlNet architecture (containing 4 downsampling blocks, each block containing 3×3 convolutions and ReLU activations), combined with the identity encoding vector c. id Generate spatial structure features Furthermore, I is processed through a learnable wavelet transform module (LWN). LQ Decomposition yields low-frequency approximate components. Horizontal high-frequency detail component F lh Vertical high-frequency detail component F hl and high-frequency detail components in the diagonal direction The LWN module is implemented using a trainable wavelet kernel and satisfies the perfect reconstruction constraint; the final output is the identity encoding vector c. id Spatial structural features f s and high-frequency detail component F hh As a constraint for subsequent processing.

[0126] S2. Mixed Conditional Latent Space Initialization: The low-quality input image I is initialized using the VQGAN encoder E. LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Construct a mixed conditional vector c;

[0127] S21. Transform the low-quality input image I using the VQGAN encoder E. LQ Mapping to the latent space yields the initial latent variables. Where E() is the encoder E-mapping function; H is the image height, W is the image width, and 1024 is the number of latent variable channels;

[0128] The VQGAN encoder E is a downsampling encoder containing 4 convolutional layers, with 4C = 1024 output channels, where C is the feature channel base, with a value of 256. The first convolutional layer includes a 3×3 kernel, a stride of 2, padding of 1, and 64 output channels. The second convolutional layer includes a 3×3 kernel, a stride of 2, padding of 1, and 256 output channels. The third convolutional layer includes a 3×3 kernel, a stride of 2, padding of 1, and 256 output channels. The fourth convolutional layer includes a 3×3 kernel, a stride of 1, padding of 1, and 1024 output channels.

[0129] Step S21 is as follows:

[0130] Low-quality input image The input VQGAN encoder E first performs a first convolutional layer to downsample the input from its original resolution H×W×3, outputting a feature map of size H×W×3. Batch normalization and ReLU activation function are then applied; the second convolutional layer uses the same configuration as the first convolutional layer, increasing the number of channels to 256, and further compressing the output feature map size. The same batch normalization and ReLU activation process is applied; the third layer expands the number of channels to 512 through 3×3 convolutions with a stride of 2, reducing the spatial resolution to... Finally, the fourth convolutional layer increases the channel dimension to 1024 without changing the resolution, generating the initial latent variables. During the encoding process, zero padding is used after each convolutional layer to ensure size alignment, and shallow detail features are fused through residual connections. The formula is expressed as:

[0131] F l =ReLU(Conv(BN(F) l-1 )))+Conv skip (F l-1 )

[0132] Where l = 1, 2, 3, 4 represent the hierarchical index, and BN() is the batch normalization function; Conv skip A 1×1 convolution is used to adjust the number of residual branch channels; F l This is the feature map obtained from the l-th convolutional layer.

[0133] The VQGAN encoder E is jointly optimized during the training phase using adversarial loss and perceptual loss to ensure that the latent variable z2 retains both the structural information of the input image (such as facial contours) and the semantic expression capability that the generative model can decode, thus providing a complete starting point for the generation of the subsequent consistency model (LCM).

[0134] Batch normalization (BatchNorm) is a commonly used technique in deep learning to accelerate the training process of neural networks and improve performance. It normalizes the data in each mini-batch, resulting in a more stable distribution of activation values ​​in the network, which helps alleviate the so-called "internal covariate shift" problem.

[0135] Zero padding is a common technique used in deep learning to process image data, especially in Convolutional Neural Networks (CNNs). Its purpose is to add extra borders around the boundaries of the input image, with all pixel values ​​within these borders set to 0.

[0136] S22, Convert the identity encoding vector c id Flattened spatial structure characteristics s ) and the high-frequency detail components in the diagonal direction after flattening. hh The concatenation is performed along the channel dimension to generate a joint condition vector c. joint Then, it is compressed into a joint condition vector c through a fully connected layer Linear(·). joint The dimension is used to obtain the mixture condition vector. The final output consists of the initial latent variable z2 and the mixing condition vector c; these are concatenated along the channel dimension using the following formula:

[0137]

[0138] Flatten() is the expansion function; the spatial structure features after flattening. The spatial dimension after flattening; high-frequency detail components along the diagonal direction after flattening. 256 represents the flattened spatial dimension; 256 represents the channel dimension; joint condition vector The dimension is

[0139] S3. Multi-step generation with frequency domain enhancement: Iterative denoising of the initial latent variable z2 and the mixing condition vector c is performed in the latent space using a consistency model, and wavelet residual correction is superimposed to enhance high-frequency details and generate a multi-scale restored image.

[0140] Specifically, step S3 includes:

[0141] Initial latent variables and mixed condition vector As input, a consensus model is used to perform multi-step iterations, and wavelet residual correction terms are incorporated during the iteration process to generate the initial latent variable z for denoising. t-1 ; and in generating the initial latent variable z t-1 During the process, a multi-scale decoder D k From the latent variable z T-k Generate multi-scale restored images The latent variable z T-k The initial latent variable z generated during the iteration process t-1 The index relationship is k = T - t + 1;

[0142] The multi-scale decoder D k Based on the SIMO architecture, it contains k levels of upsampling blocks, each level containing transposed convolution and LeakyReLU activation function; where T is the total number of iterations, and T takes the value of 4; k = 1, 2, 3; These correspond to the decoding results for the highest resolution z3, the intermediate resolution z2, and the lowest resolution z1, respectively.

[0143] The initial latent variables and mixed condition vector As input, a consensus model is used to perform multi-step iterations, and wavelet residual correction terms are incorporated during the iteration process to generate the initial latent variable z for denoising. t-1 Specifically:

[0144] The initial latent variables and mixed condition vector As input, a consensus model is used to perform multi-step iterations, and wavelet residual correction terms are incorporated during the iteration process to generate the initial latent variable z for denoising. t-1 Specifically, each iteration calls the LCM single-step generation function LCM_step(z t Generate preliminary latent variable z (t, c) t-1 And at time step t∈{2,3,4}, wavelet residual correction terms are superimposed. Perform latent spatial denoising to enhance high-frequency details: z t The current latent variable is z2; the current latent variable in the first iteration is the initial latent variable z2.

[0145] The call to the LCM single-step generation function LCM_step(z) t Generate preliminary latent variable z (t, c) t-1 Specifically, it includes:

[0146] First, the mixture condition vector c is projected onto the spatial dimension through a fully connected layer. and with the current latent variable z t By adding element by element, we obtain the latent variables that enhance the conditions. Then time step t is converted into sinusoidal position encoding. And mapped to time weights through a multilayer perceptron (MLP). and Perform a Hadamard product along the channel dimension to generate time-conditional latent variables. Time-conditional latent variables Input the improved latent consistency denoising network to calculate the denoising residual Δz denoise Finally, the initial latent variable z is generated through the consistency condition formula. t-1 ;

[0147] The improved latent consistency denoising network consists of four residual blocks, each block containing a 3×3 depthwise convolutional layer, a layer normalization layer, and a multi-head attention mechanism. The multi-head attention mechanism contains four heads, each with a dimension of 256.

[0148] The formula for calculating the denoised residual is as follows:

[0149]

[0150] Here, ResBlock() represents a residual block, and the number in the lower right corner (such as ResBlock1, ResBlock2) represents the processing level number of the residual block, which is used to distinguish the feature extraction modules at different stages.

[0151] The consistency condition formula is as follows:

[0152]

[0153] in The cumulative noise attenuation coefficient (β) s (∈(0,1) represents the preset noise scheduling parameters), denominator term Used for stable gradient updates;

[0154] This process achieves rapid denoising of degenerate perception by explicitly fusing mixed conditions with a multi-level attention mechanism.

[0155] The process involves generating the initial latent variable z. t-1 During the process, a multi-scale decoder D k From the latent variable z T-k Generate multi-scale restored images Specifically:

[0156] Potential variables at different iteration steps (where T = 4, k = 1, 2, 3) Input multi-scale decoder D k Multi-scale image reconstruction based on SIMO architecture:

[0157] For the highest resolution decoding path (k=1), with latent variables As input, the feature map size is first increased to [size missing] through a first-level upsampling block (transposed convolutional kernel size 4×4, stride 2, padding 1, output channels 512). After LeakyReLU activation (negative slope 0.2), and spatial structure features Resolution is aligned using bilinear interpolation, and fused features are generated by stitching together along the channel dimension. The second-level upsampling block (4×4 transposed convolution kernel, stride 2, 256 output channels) further expands the resolution to and low-frequency components Add them together; finally, the third-level upsampling block (transposed convolution kernel 4×4, stride 2, output channels 3) outputs a high-resolution RGB restored image. The intermediate resolution (k=2) and low resolution (k=3) paths take z1 and z2 as inputs, respectively, and generate intermediate resolution restored images through level 2 and level 1 upsampling blocks. Low-resolution image restoration in The resolution is And upsampled to H×W×3 by bicubic interpolation; during the decoding process, a skip connection is introduced after each stage of transposed convolution, and the formula for the skip connection is expressed as:

[0158]

[0159] Where l = 1, 2, 3 represents the upsampling block level, and Interp() is the bilinear interpolation alignment resolution function. These are the corresponding hierarchical features extracted from the encoder;

[0160] ConvTranspose() is a transposed convolution, also known as deconvolution, used to progressively restore the spatial resolution of the feature map in the decoder;

[0161] LearkyReLU() is a Leaky Rectified Linear Unit that allows small gradients for negative inputs. It is used to avoid neuron death in Generative Adversarial Networks (GANs) and enhance the model's ability to model noise and degenerate features.

[0162] Upsampling function, usually implemented by transposed convolution or interpolation (such as bilinear upsampling), is used to progressively enlarge the feature map size in the decoding path and fuse it with shallow features provided by skip connections to reconstruct a high-resolution image;

[0163] The design achieves multi-scale supervision through a lightweight cascaded structure, ensuring the quality of the generated images. It is strictly aligned with the input degradation conditions in terms of identity features, spatial structure, and high-frequency details.

[0164] The Latent Consistency Model (LCM) is a type of model used in machine learning, especially for unsupervised learning tasks. The core idea of ​​LCM is to improve learning efficiency and quality by leveraging the consistent distribution of data within the latent space. The latent space typically refers to the low-dimensional space to which the original data is mapped by the encoder, containing a compressed representation of the original data.

[0165] The Hadamard product, also known as the element-wise product or Schur product, is a mathematical operation used between two matrices or vectors of the same shape. In the Hadamard product, corresponding elements of the two matrices or vectors are multiplied, and the result is the product of those elements at the corresponding positions in the resulting matrix or vector.

[0166] SIMO (Single Input Multi Output) architecture is a system design approach where there is a single input source, which, after processing by the system, can produce multiple output results.

[0167] S4. Wavelet Domain Adaptive Fusion: Image Recovery Based on Multi-Scale High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse ;

[0168] Step S4 specifically includes:

[0169] The multi-scale restored image set output in step S3 and from the diagonal high-frequency detail components Extracted high-frequency components at various scales As input, the L1 norm of the high-frequency components at each scale is first calculated as the high-frequency energy. Then based on the high-frequency energy The fusion weights λ are dynamically allocated using the Softmax function. k Reconstructed images at each scale Perform a learnable discrete wavelet transform to decompose it into low-frequency subbands. and high-frequency subband Diagonal high-frequency detail components With high-frequency subband The horizontal and vertical components are spliced ​​together to form the reconstructed high-frequency subband. The reconstructed image is then generated using inverse wavelet transform: Finally, according to the fusion weight λ k Weighted fusion reconstruction of images to generate multi-scale fused images

[0170] From step S1, we can know the diagonal high-frequency detail component F hh It is generated through multi-scale decomposition. The multi-scale characteristic is reflected in the stepwise extraction of image frequency domain information by wavelet transform at different levels. The multi-scale decomposition includes three levels of decomposition k = 1, 2, 3.

[0171] First-order scaling (k=1): After the original input image undergoes the first-level learnable wavelet transform, high-frequency components with the same resolution as the input are generated. Its frequency band coverage is the widest, mainly capturing pixel-level high-frequency details (such as skin texture, hair edges, and local noise);

[0172] Second-order scale (k=2): The low-frequency components of the first-order decomposition The input is a second-level wavelet transform, which generates high-frequency components with half the resolution. Its frequency band is narrower, focusing on medium-scale structured high-frequency features (such as facial contours and lighting boundaries);

[0173] Third-order scale (k=3): Further refinement of the second-order low-frequency components The high-frequency components are decomposed and then downsampled by half to obtain the resolution. Its frequency band is the narrowest, associated with sparse high-frequency signals in the global low-frequency substrate (such as facial symmetry, large-scale illumination transition).

[0174] Therefore, F hh It is a collection of multi-scale high-frequency components. It is easy to find high-frequency detail components on the diagonal. High-frequency components at various scales were extracted.

[0175] The calculation of the L1 norm of high-frequency components at each scale The formula is:

[0176] in, This represents the feature value of the high-frequency subband in the diagonal direction at spatial location (i,j) and the c-th channel in the k-th level (k=1,2,3) decomposition layer after multi-scale decomposition of the input image by the learnable wavelet transform module. It is the diagonal high-frequency response value of a specific position and channel in the k-th level decomposition;

[0177] The Softmax function is: (satisfy );

[0178] The low-frequency subband and high-frequency subband All include three directions: horizontal, vertical, and diagonal. For channel-dimensional concatenation; IDWT() is the inverse wavelet transform function; i,j represent the row (height direction) and column (width direction) of the high-frequency component feature map, respectively, used to index the position of the pixel, i∈[1,H],j∈[1,W]; c is the channel index, c=1,2,3 correspond to the red, green and blue color channels respectively;

[0179] The L1 norm, commonly mentioned in mathematics, signal processing, and optimization problems, refers to a norm of a vector in a vector space. Specifically, the L1 norm of a vector is defined as the sum of the absolute values ​​of its components.

[0180] S5. Dynamic Post-processing and Output: Multi-scale Fusion Image Based on Gaussian Blur fuse The sharpening intensity is adjusted to generate a low-frequency substrate image I. blur .

[0181] Specifically, this includes: the multi-scale fused image output from step S4. When performing Gaussian blurring, a 5×5 Gaussian kernel is first generated. Its weight values ​​are calculated using a two-dimensional Gaussian function:

[0182]

[0183] Where G(x, y) represents the weight values ​​of relative coordinates x and y within the Gaussian kernel; σ is the standard deviation, with a value of 1.0, and the origin is located at the center of the Gaussian kernel; the weights of each point within the Gaussian kernel are normalized and summed to 1, with the specific values ​​as follows:

[0184]

[0185] Multi-scale fused image I fuse Each RGB channel independently performs a zero-padding 2D convolution operation. The padding width of the 2D convolution operation is 2, the stride is 1, and pixel values ​​outside the boundary are treated as 0. The formula is expressed as follows:

[0186]

[0187] Where i and j represent the multi-scale fused image I, respectively. fuse The rows (height direction) and columns (width direction) are used to index the position of pixels, i∈[1,H],j∈[1,W]; c is the channel index, c=1,2,3 correspond to the red, green and blue color channels respectively; dx and dy are the offsets used for two-dimensional convolution operations; dx represents the horizontal offset, with a value range of -2 to 2; dy represents the vertical offset, with a value range of -2 to 2.

[0188] This operation suppresses high-frequency components (such as noise and sharp edges) in the image, resulting in a low-frequency substrate image. Only global illumination and smoothed region information are retained to provide a benchmark for subsequent high-frequency residual calculations.

[0189] The design of this embodiment stems from real-world industry pain points: in the field of security monitoring, blurred faces need to be quickly restored to reveal identifiable features; in the digitization of cultural heritage, old photos need to be restored while preserving the true appearance of the people; and in mobile applications, lightweight models are needed to enhance selfie images in real time. Therefore, there is an urgent need for a restoration solution that requires no prior degradation, has strong identity constraints, allows for controllable enhancement of details, and is computationally efficient.

[0190] This embodiment provides a real-time face blindness repair method based on identity constraints and frequency domain enhancement, including: S1, multimodal feature extraction: extracting low-quality input image I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh S2, Mixed Conditional Latent Space Initialization: The low-quality input image I is initialized using the VQGAN encoder E. LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Constructing the hybrid conditional vector c; S3, multi-step generation with frequency domain enhancement: using a consistency model to perform latent space iterative denoising on the initial latent variable z2 and the hybrid conditional vector c, and superimposing wavelet residual correction to enhance high-frequency details to generate a multi-scale restored image. S4. Wavelet Domain Adaptive Fusion: Image Recovery Based on Multi-Scale High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse This embodiment innovatively combines multimodal identity features (CLIP semantics + ArcFace discriminative features) with a frequency domain enhancement mechanism. Specifically, it uses identity-space-frequency domain ternary constraints, accelerates generation through a lightweight consistency model (LCM), and introduces dynamic residual correction to achieve precise control of high-frequency details. Ultimately, it breaks through the traditional method's dilemma of balancing identity consistency, detail fidelity, and real-time performance, providing a highly available solution for real-world scenarios.

[0191] Furthermore, this embodiment also adjusts the sharpening intensity of the multi-scale fused image through Gaussian blurring, thereby suppressing high-frequency components (such as noise and sharp edges) in the image and generating a low-frequency base image. By retaining only global illumination and smoothing region information, the image becomes smoother, which helps to remove unnecessary noise while preserving image details, thereby improving the overall appearance and processing quality of the image.

[0192] Example 2

[0193] refer to Figure 2 This embodiment provides a real-time face blindness repair device based on identity constraints and frequency domain enhancement, including the following units:

[0194] A multimodal feature extraction unit is used to extract low-quality input images I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh N is the number of reference images;

[0195] Latent spatial initialization unit, used to process low-quality input image I through VQGAN encoder E LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Construct a mixed conditional vector c;

[0196] The frequency domain enhancement unit is used to perform latent space iterative denoising on the initial latent variable z2 and the mixing condition vector c through a consistency model, and then superimpose wavelet residual correction to enhance high-frequency details and generate a multi-scale restored image.

[0197] Wavelet domain fusion unit for image reconstruction based on multiple scales High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse ;

[0198] The sharpening unit is used for multi-scale fusion image I based on Gaussian blur. fuse The sharpening intensity is adjusted to generate a low-frequency substrate image I. blur .

[0199] Example 3

[0200] refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of the real-time face blindness repair device based on identity constraints and frequency domain enhancement in this embodiment. The real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0201] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement. For example, the computer program can be divided into the modules in Embodiment 2; the specific functions of each module are described in the working process of the device described in the above embodiments, and will not be repeated here.

[0202] The real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement, and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement may also include input / output devices, network access devices, buses, etc.

[0203] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement, connecting all parts of the device through various interfaces and lines.

[0204] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0205] The modules / units integrated in the real-time face blindness repair device 20 based on identity constraints and frequency domain enhancement, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0206] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0207] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 A process, multiple processes, and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0210] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A real-time face blindness repair method based on identity constraints and frequency domain enhancement, characterized in that, The following steps are involved: S1. Multimodal feature extraction: Extracting low-quality input image I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh N is the number of reference images; Step S1 specifically includes the following steps: S11. Obtain a low-quality input image. and reference image set Where H is the height of the input image, W is the width of the input image, 3 represents the RGB three channels, and N is an integer greater than or equal to 1; For the i-th reference image, H ref For the reference image height, W ref The width is the reference image width; S12, Reference image The input is fed into the pre-trained CLIP model to extract global semantic features f CLIP The reference image To obtain from the reference image set The i-th image selected from the list; S13, Reference image The input is fed into a pre-trained ArcFace model to extract the identity feature vector f. ArcFace ; S14. Transfer global semantic features f CLIP With identity feature vector f ArcFace The joint feature f is generated through channel splicing operations. joint After processing by a multilayer perceptron, the output identity encoding vector c is obtained. id ; S15, Input low-quality image I LQ Input a spatial encoder based on the ControlNet architecture, combined with the identity encoding vector c id Generate spatial structure features f s ; S16. The low-quality input image I is processed using a learnable wavelet transform module. LQ Decomposition yields the high-frequency detail components F in the diagonal direction. hh ; S2. Mixed Conditional Latent Space Initialization: The low-quality input image I is initialized using the VQGAN encoder E. LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Construct a mixed conditional vector c; S3. Multi-step generation with frequency domain enhancement: Iterative denoising of the initial latent variable z2 and the mixing condition vector c is performed in the latent space using a consistency model, and wavelet residual correction is superimposed to enhance high-frequency details and generate a multi-scale restored image. S4. Wavelet Domain Adaptive Fusion: Image Recovery Based on Multi-Scale High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse .

2. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Transform the low-quality input image I using the VQGAN encoder E. LQ Mapping to the latent space yields the initial latent variables. Where E() is the encoder E-mapping function; H is the image height, W is the image width, and 1024 is the number of latent variable channels; S22, Transform the identity encoding vector c id Flattened spatial structure characteristics s ) and the high-frequency detail components in the diagonal direction after flattening. hh The concatenation is performed along the channel dimension to generate a joint condition vector c. joint Then, it is compressed into a joint condition vector c through a fully connected layer. joint The dimension is used to obtain the mixing condition vector c = Linear(c joint The final output consists of the initial latent variable z2 and the mixing condition vector c; these are then concatenated along the channel dimension using the following formula: Flatten() is the expansion function; Linear() is the linear mapping function corresponding to the fully connected layer.

3. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 1, characterized in that, Step S3 specifically includes: Initial latent variables and mixed condition vector As input, a consensus model is used to perform multi-step iterations, and wavelet residual correction terms are incorporated during the iteration process to generate the initial latent variable z for denoising. t-1 ; and in generating the initial latent variable z t-1 During the process, a multi-scale decoder D k From the latent variable z T-k Generate multi-scale restored images The multi-scale decoder D k Based on the SIMO architecture, it contains k levels of upsampling blocks, each level containing transposed convolution and LeakyReLU activation function; where T is the total number of iterations, and T takes the value of 4; k = 1, 2, 3; The decoding results correspond to the highest resolution z3, the intermediate resolution z2, and the lowest resolution z1, respectively.

4. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 1, characterized in that, Step S4 specifically includes: Reconstructing image sets at multiple scales and from the diagonal high-frequency detail components Extracted high-frequency components at various scales As input, the L1 norm of the high-frequency components at each scale is first calculated as the high-frequency energy. Then based on the high-frequency energy The fusion weights λ are dynamically allocated using the Softmax function. k Reconstructed images at each scale Perform a learnable discrete wavelet transform to decompose it into low-frequency subbands. and high-frequency subband Diagonal high-frequency detail components With high-frequency subband The horizontal and vertical components are spliced ​​together to form the reconstructed high-frequency subband. The reconstructed image is then generated using inverse wavelet transform: Finally, according to the fusion weight λ k Weighted fusion reconstructed image generates multi-scale fused image I fuse :

5. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 1, characterized in that, The method further includes: S5. Dynamic Post-processing and Output: Multi-scale Fusion Image Based on Gaussian Blur fuse The sharpening intensity is adjusted to generate a low-frequency substrate image I. blur .

6. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 5, characterized in that, Step S5 specifically includes: Multi-scale fused images When performing Gaussian blurring, a 5×5 Gaussian kernel is first generated. Its weight values ​​are calculated using a two-dimensional Gaussian function, which is shown below: Where G(x, y) represents the weight values ​​of relative coordinates x and y within the Gaussian kernel; σ is the standard deviation, with a value of 1.0, and the origin is located at the center of the Gaussian kernel; the weights of each point within the Gaussian kernel are normalized and summed to 1, with the specific values ​​as follows: Multi-scale fused image I fuse Each RGB channel independently performs a zero-padding 2D convolution operation. The padding width of the 2D convolution operation is 2, the stride is 1, and pixel values ​​outside the boundary are treated as 0. The formula is expressed as follows: Where i and j represent the multi-scale fused image I, respectively. fuse The rows and columns are used to index the position of pixels, i∈[1,H],j∈[1,W]; c is the channel index, c=1,2,3 correspond to the red, green and blue color channels respectively; dx and dy are the offsets used for two-dimensional convolution operations; dx represents the horizontal offset, with a value range of -2 to 2; dy represents the vertical offset, with a value range of -2 to 2.

7. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 1, characterized in that, Step S12 specifically includes: Reference image The input to the pre-trained CLIP model is first standardized to adjust the resolution to A×A pixels, and then the first input tensor is generated by mean-variance normalization. Subsequently, the image is segmented into a B×B sequence of image blocks by the C-layer Transformer block of the image encoder. Each block is linearly projected and mapped to a C-dimensional vector. Global semantic information is aggregated through a self-attention mechanism. Finally, the feature vector corresponding to the CLS label of the last Transformer block is extracted, mapped to E-dimensional space through a fully connected layer, and the global semantic feature f is generated. CLIP .

8. The real-time face blindness repair method based on identity constraints and frequency domain enhancement according to claim 1, characterized in that, Step S13 specifically includes: Reference image The pre-trained ArcFace model is first input, and then normalization is performed to adjust the image resolution to F×F pixels. Then, mean-variance normalization is performed to generate the second input tensor. Then the second input tensor Through the forward propagation of the backbone network of the pre-trained ArcFace model, the feature vector f is extracted sequentially through the initial convolutional layer, the batch normalization (BN) layer, and the ReLU activation layer, followed by four residual blocks, and finally extracted after the last layer of global average pooling. base ; to feature vector Input the fully connected layer corresponding to the ArcFace loss function, apply additive angular boundary calculation to output the normalized identity feature vector. The backbone network is ResNet-100.

9. A real-time face blindness repair device based on identity constraints and frequency domain enhancement, characterized in that, Includes the following units: A multimodal feature extraction unit is used to extract low-quality input images I LQ and reference image set An identity encoding vector c is generated by fusing CLIP and ArcFace. id Then, combined with ControlNet, spatial structure features f are extracted. s And learnable wavelet transform is used to obtain the high-frequency detail component F in the diagonal direction. hh N is the number of reference images; The multimodal feature extraction unit specifically includes the following steps: S11. Obtain a low-quality input image. and reference image set Where H is the height of the input image, W is the width of the input image, 3 represents the RGB three channels, and N is an integer greater than or equal to 1; For the i-th reference image, H ref For the reference image height, W ref The width is the reference image width; S12, Reference image The input is fed into the pre-trained CLIP model to extract global semantic features f CLIP The reference image To obtain from the reference image set The i-th image selected from the list; S13, Reference image The input is fed into a pre-trained ArcFace model to extract the identity feature vector f. ArcFace ; S14. Transfer global semantic features f CLIP With identity feature vector f ArcFace The joint feature f is generated through channel splicing operations. joint After processing by a multilayer perceptron, the output identity encoding vector c is obtained. id ; S15, Input low-quality image I LQ Input a spatial encoder based on the ControlNet architecture, combined with the identity encoding vector c id Generate spatial structure features f s ; S16. The low-quality input image I is processed using a learnable wavelet transform module. LQ Decomposition yields the high-frequency detail components F in the diagonal direction. hh ; Latent spatial initialization unit, used to process low-quality input image I through VQGAN encoder E LQ The initial latent variable z2 is obtained by mapping to the latent space, and the identity encoding vector c is fused with it. id Spatial structural features f s and the high-frequency detail component F in the diagonal direction hh Construct a mixed conditional vector c; The frequency domain enhancement unit is used to perform latent space iterative denoising on the initial latent variable z2 and the mixing condition vector c through a consistency model, and then superimpose wavelet residual correction to enhance high-frequency details and generate a multi-scale restored image. Wavelet domain fusion unit for image reconstruction based on multiple scales High-frequency detail components on the diagonal Calculate the dynamic weight allocation of high-frequency energy, and reconstruct the multi-scale fused image using inverse wavelet transform. fuse .

Images