A text and face collaborative restoration method based on cross-modal alignment

Abstract

The application is suitable for the technical field of image processing, and provides a text and face collaborative repairing method based on cross-modal alignment, including the following steps: constructing a text-face multi-modal data set; based on a cross-modal network architecture, realizing image super-resolution in the text-face multi-modal data set and text and image alignment; training the cross-modal network, designing an optimized hybrid loss function; and reasoning to generate high-quality images. Through constructing a deep fusion framework of text semantics and image features, the identity feature reservation capability of the repaired face image is significantly improved, thereby laying a key foundation for improving the face recognition rate.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a text and face collaborative repair method based on cross-modal alignment. BACKGROUND

[0002] In recent years, the application value of face image repair technology in public security, criminal investigation and other fields is increasingly prominent, but factors such as complex lighting conditions and imaging device performance limitations often lead to problems such as detail blur and identity feature weakening in low-quality face images, and traditional repair methods are difficult to effectively restore the available information of the image. For example, the repair scheme based on optimization strategy relies on manually designed face prior knowledge, and such methods are limited by the coverage range of the prior model and the high complexity of iterative calculation. In the case of severe degradation scenes such as occlusion and profile, the repair effect is difficult to meet the actual needs.

[0003] With the development of deep learning technology, repair models based on neural networks have gradually become the mainstream. From the early convolutional neural network (CNN), generative adversarial network (GAN) to the recent Transformer and diffusion model, related methods improve the repair ability by learning a large amount of image data. However, most of these schemes only rely on single-modal visual information and lack the guidance of external semantic clues. In the case of complex degradation (such as shadows, occlusions, and pose changes), it is difficult to accurately establish the semantic association of facial details, resulting in problems such as structural distortion, loss of identity features, and semantic inconsistency in the repair results. For example, traditional methods often generate structures that deviate from the true features when repairing profile faces or faces obscured by accessories due to the lack of semantic constraints. Diffusion models, with their progressive generation capabilities, have shown robustness in handling complex degradation in face repair, but they are still limited to single-modal image input and cannot effectively utilize external semantic information such as text to optimize the repair direction. At the same time, although the text-to-image generation field has achieved high-quality image synthesis through cross-modal models such as CLIP, it mainly focuses on the generation task and has two major bottlenecks when directly applied to face repair: first, it is difficult to preserve the identity features of the input low-quality image, leading to semantic correctness but identity deviation in the repair results; second, there is a lack of a three-tuple dataset of text-low-quality image-high-quality image, making the training of cross-modal models face the challenge of data sparsity. Therefore, how to integrate text semantics and image features, build a cross-modal alignment mechanism, and simultaneously achieve identity preservation and semantic consistent detail reconstruction in the repair process has become a key breakthrough point for the evolution of face repair technology from single-modal to multi-modal. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a text and face collaborative repair method based on cross-modal alignment, which aims to solve the problems raised in the above background technology.

[0005] The embodiment of the present invention is implemented as follows: a method for collaborative restoration of text and face based on cross-modal alignment, comprising the following steps:

[0006] Construct a text-face multimodal dataset;

[0007] Based on a cross-modal network architecture, we achieve image super-resolution and text-image alignment in a text-face multimodal dataset.

[0008] Train cross-modal networks and design and optimize hybrid loss functions;

[0009] Inference generates high-quality images.

[0010] The embodiments of the present invention provide a text and face collaborative restoration method based on cross-modal alignment. By constructing a deep fusion framework of text semantics and image features, it significantly improves the ability to retain identity features of restored facial images, thereby laying a key foundation for improving face recognition rates. Specifically, the CLIP model extracts identity-related semantics (such as age, gender, facial features, etc.) from the text and performs cross-modal alignment with the potential identity features in the low-quality image to generate a joint embedding containing rich identity clues. At the same time, the stacked identity embedding mechanism integrates multi-source identity information and combines the mask diffusion loss to force optimization of the structural consistency of key facial areas (such as eyes, nose, and jawline), effectively avoiding the distortion or loss of identity features caused by insufficient single-modal information in traditional methods. At the same time, this cross-modal collaborative restoration mechanism can generate frontal facial images with consistent identity features even when the facial features are incomplete (such as those blocked by eyes, hats, etc., profile, heavy makeup, etc.), significantly enhancing the identity recognition rate of the restored image, and thus providing more reliable images for subsequent face recognition and other needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of a method for collaborative text and face restoration based on cross-modal alignment is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0013] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0014] like Figure 1 FIG. 1 is a flowchart of a method for collaborative text and face restoration based on cross-modal alignment provided by an embodiment of the present invention, comprising the following steps:

[0015] S1, construct a text-face multi-modal dataset, specifically including:

[0016] S1.1, obtain high-quality (HQ) face images from public face datasets FFHQ, CelebA, randomly take two images with different poses, clothes or expressions for each person identity (ID), and form a large number of image pairs;

[0017] S1.2, for a random HQ image in each image pair, generate a corresponding low-quality (LQ) image through degradation, and form an LQ-HQ image pair;

[0018] S1.3, manually annotate structured text descriptions (Text) for each LQ-HQ image pair, including gender, age, nationality, clothing, expression, and action, thereby forming an "LQ-HQ-Text" triplet data to solve the problem of missing multi-modal dataset;

[0019] S1.4, due to the high requirement for model generalization in face restoration, the dataset constructed from FFHQ is used as the training set, and the dataset constructed from CelebA is used as the test set.

[0020] S2, based on the cross-modal network architecture, realize image super-resolution and text and image alignment in the text-face multi-modal dataset, specifically including:

[0021] S2.1, use the image super-resolution module to realize image super-resolution:

[0022] S2.1.1, use HQ encoder and LQ encoder, adopt U-Net style down-sampling architecture, input 512x512x3 RGB image, through 5 stages of down-sampling and residual block processing, compress the image into 16x16x256 feature map, each stage contains a convolution layer with a step of 2 and 4 residual blocks, the initial convolution layer converts RGB to 64 channels, the subsequent down-sampling layers gradually adjust the channel number to 256, the residual block is composed of two 3x3 convolution layers and LeakyReLU activation, and the jump connection is used to alleviate the problem of gradient disappearance;

[0023] S2.1.2. Use the Transformer module to predict the codebook index sequence corresponding to the low-quality image: First, the 16×16×256 feature map output by the encoder is flattened into a 256×256 sequence, and a sinusoidal position encoding is added to preserve spatial information. The sequence is processed by 9 Transformer blocks, each of which contains a multi-head self-attention layer and a feedforward network. The self-attention layer calculates the correlation between elements in the sequence to capture global dependencies, and the feedforward network enhances the feature expression capability. Each block also contains residual connections and layer normalization to stabilize training. Finally, the features are mapped to 1024 dimensions (codebook size) through a linear layer, and a softmax classifier is used to predict the code index at each position;

[0024] S2.1.3. Use the HQ decoder (symmetric with the encoder) and an upsampling architecture to recover image details. Input 16×16×256 quantized features are processed through five stages of upsampling and residual blocks to gradually restore the image to a 512×512×3 RGB image. Each stage consists of a nearest neighbor interpolation layer (doubling the resolution), a 3×3 convolutional layer (adjusting the number of channels), and four residual blocks. The final convolutional layer maps the 64-channel features to 3-channel RGB, using the Tanh activation function to constrain the range to [-1, 1].

[0025] S2.2. Use the text-image alignment module to achieve text and image alignment:

[0026] S2.2.1. Use an image encoder (the ViT-L / 14 pre-trained model from CLIP) to extract the semantic embedding of the input ID image.

[0027] S2.2.2. Use a text encoder (using the dual text encoder architecture of SDXL (stable-diffusion-xl-base-1.0)) to extract multi-granularity semantic embeddings of textual cues, where the text encoder includes:

[0028] The main encoder (CLIP ViT-L / 14 text encoder) is used to process natural language descriptions;

[0029] Auxiliary encoder (OpenCLIP ViT-bigG / 14) for enhancing long text comprehension capabilities;

[0030] S2.2.3. Construct stacked ID embeddings: Fuse the embeddings of multiple input ID images with the classifier features in the text to generate fused embedding vectors. Concatenate these vectors along the length dimension to form stacked ID embeddings.

[0031] S2.3.4, diffusion model adaptation and cross-attention mechanism: based on the SDXL-based diffusion model (stable-diffusion-xl-base-1.0), the stack ID embedding replaces the category word position in the text embedding, generates an updated text embedding, and through the cross-attention mechanism, the query vector interacts with the updated text embedding, so that the model adaptively fuses ID information, ensuring that the generated image not only retains ID information but also conforms to the text description.

[0032] S3, training the cross-modal network, designing an optimized hybrid loss function:

[0033] The training strategy includes the following processes:

[0034] S3.1, learn the discrete codebook through HQ image self-reconstruction, and train the HQ encoder and HQ decoder at the same time;

[0035] S3.2, freeze the discrete codebook and HQ decoder, train the LQ encoder and Transformer module, predict the codebook index sequence, and model the global face structure;

[0036] S3.3, fix the codebook and the Transformer module, and jointly train the image encoder, the text encoder and the diffusion model, and fine-tune the image encoder, the text encoder and the diffusion model parameters using the joint training total loss;

[0037] The hybrid loss function includes:

[0038] Codebook learning loss: use L1 loss in the process of learning the discrete codebook in step S3.1 , perceptual loss , adversarial loss , code-level loss Joint optimization to align the encoder output with the quantized representation, and the expressions of the several losses are as follows:

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] wherein, is the original high-definition image; is the image obtained after learning HQ image self-reconstruction in step S3.1; represents the VGG19 feature extractor; is the discriminator; compressed features obtained after inputting the original image into the encoder, Each feature vector is replaced by the nearest entry in the discrete codebook in step S3.1 through the Transformer module, thereby obtaining quantized features ; denotes the stop gradient operator; As a balance coefficient, it can be taken as 0.5 in the embodiment of the application;

[0044] Total loss of codebook learning :

[0045] ;

[0046] Total loss of the Transformer module: in the process of predicting the codebook index sequence through the Transformer module in step S3.2, cross-entropy loss and L2 loss are used for supervised training to improve the detail reconstruction capability, and the expressions of the two losses are as follows:

[0047] ;

[0048] ;

[0049] wherein, is the value of the i-th position in the probability distribution of the real codebook index; is the probability of the i-th codebook index predicted by the model; is the compressed feature after encoding the LQ image in step S2.1; , is the spatial size of the feature map obtained after encoding the LQ image in step S2.1, and the height and width are respectively;

[0050] Total loss of the Transformer module :

[0051] ;

[0052] wherein, As a balance coefficient, it can be taken as 0.5 in the embodiment of the application;

[0053] Mask diffusion loss : in the process of joint training with the diffusion model in step S3.3, the mask diffusion loss is introduced, and the binary mask of the identity-related region is generated through Mask2Former , the weighted noise prediction error is calculated, and the expression is as follows:

[0054] ;

[0055] wherein, is the real noise; is the noise predicted by the diffusion model; is the noise image at time step ; is the stacked ID embedding in step S2.2.3; denotes the expectation over time step , subject to uniform distribution , is the total number of diffusion steps, the layer expectation operation can make the model participate in training at different stages of the diffusion process (from strong noise to weak noise in the whole process), avoiding the model only adapting to the noise law of a certain stage; denotes the expectation over high-definition image , reconstructed image , and mask , the layer expectation operation covers data diversity, so that the loss calculation does not depend on a single image / mask, ensuring that the model is generalized to different inputs and different key area masks;

[0056] Joint training total loss: used in joint training in step S3.3, the above losses are combined according to the weights:

[0057] ;

[0058] wherein, , , are dynamically adjustable weight coefficients.

[0059] S4, inferencing to generate high-quality images: input an image, resize the input image to 512x512; input text processing, the text needs to contain the gender keyword "man" or "woman", otherwise an incorrect gender picture may be generated due to factors such as the hairstyle of the person; extract image multi-scale features through the image super-resolution module and the CLIP image encoder, extract semantic features through the text encoder, and then generate a stacked ID embedding through the text-image alignment module; perform 50-step iterative denoising in the diffusion model, and output a high-quality repair result.

[0060] Comparative analysis:

[0061] In a consistent experimental setup, that is, using the same input image and corresponding text prompt, the text and face collaborative restoration method based on cross-modal alignment proposed in the embodiment of the present application is compared with a variety of text-to-image algorithms in the prior art. The method proposed in the embodiment of the present application is closer to the target image in visual effect. In the case of incomplete facial features (obscured by eye hats, side faces, heavy makeup, etc.), the method proposed in the embodiment of the present application can generate a frontal face image with consistent identity features compared with a variety of text-to-image algorithms in the prior art. At the same time, quantitative tests are conducted, and the method proposed in the embodiment of the present application is always superior to the competing methods in key evaluation indicators, including face similarity (Face Sim.), DINO, and LPIPS, demonstrating its strong ability in identity preservation and facial detail reconstruction. In the experiment, more than ten methods are compared, and only the best methods are shown in terms of quantitative indicators and visual effects, as shown in Table 1:

[0062] Table 1

[0063]

[0064] In the comparison of the three core indicators of face similarity (Face Sim.), DINO, and LPIPS, the embodiment of the present application shows advantages. Compared with InstantID, MoMA (ECCV 2024), ConsistentID, and DEADiff (CVPR 2024) methods, the method of the embodiment of the present application reaches 44.80% in the Face Sim. indicator, significantly surpassing the 33.75% of DEADiff (CVPR 2024), indicating better capture and restoration of facial features. The DINO indicator is as high as 62.04%, significantly leading other methods, reflecting excellent visual feature expression performance. In the LPIPS indicator, the value of 0.7152 is lower than other methods, proving that the perceptual difference between the generated content and the real content is minimal, and the visual effect is more realistic. In summary, the data fully prove that in the face restoration task, whether it is feature similarity, expression ability, or visual quality, the embodiment of the present application is at a leading level.

[0065] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A text and face collaborative restoration method based on cross-modal alignment, characterized in that, The method comprises the following steps: constructing a text-face multimodal dataset; realizing image super-resolution and text and image alignment in the text-face multimodal dataset based on a cross-modal network architecture; training the cross-modal network and designing an optimized hybrid loss function; generating high-quality images through reasoning; The step of realizing image super-resolution and text and image alignment in the text-face multimodal dataset based on the cross-modal network architecture specifically comprises: extracting and compressing features of the image by using an HQ encoder and an LQ encoder; predicting a codebook index sequence corresponding to the low-quality image by using a Transformer module; reconstructing a high-quality image based on the compressed quantized features by using an HQ decoder; extracting semantic embeddings of input ID images by using an image encoder; extracting multi-granularity semantic embeddings of text prompts by using a text encoder; fusing the embeddings of the multiple input ID images and the category word features in the text to generate a fused embedding vector, concatenating the fused embedding vector along the length dimension to form a stacked ID embedding; replacing category word positions in the text embedding with the stacked ID embedding based on a diffusion model to generate an updated text embedding, and interacting the query vector with the updated text embedding through a cross-attention mechanism to enable the model to adaptively fuse ID information, so that the generated image retains the ID information and conforms to the text description.

2. The text and face collaborative inpainting method based on cross-modal alignment according to claim 1, characterized in that, The step of constructing the text-face multimodal dataset specifically comprises: obtaining high-quality face images from a public face dataset, randomly taking two different images for each person ID to form a large number of image pairs; generating corresponding low-quality images by degrading a random high-quality image in each image pair to form LQ-HQ image pairs; manually annotating structured text descriptions for each LQ-HQ image pair, including the characteristics of the person, thereby forming LQ-HQ-Text triple data.

3. The text and face collaborative inpainting method based on cross-modal alignment according to claim 2, characterized in that, The step of training the cross-modal network and designing the optimized hybrid loss function specifically comprises: learning a discrete codebook through high-quality image self-reconstruction, and simultaneously training an HQ encoder and an HQ decoder; freezing the discrete codebook and the HQ decoder, training an LQ encoder and a Transformer module to predict a codebook index sequence and model global facial structures; fixing the codebook and the Transformer module, and jointly training the image and text encoders and the diffusion model, and fine-tuning the image encoder, the text encoder, and the diffusion model parameters using a joint training total loss.

4. The text and face collaborative inpainting method based on cross-modal alignment according to claim 3, characterized in that, The joint training total loss is a combination of a codebook learning total loss, a Transformer module total loss, and a mask diffusion loss according to weights; wherein the codebook learning total loss is used for optimization during the learning of the discrete codebook, the Transformer module total loss is used for optimization during the prediction of the codebook index sequence, and the mask diffusion loss is introduced during the joint training of the image, text, and diffusion models.

Images