Text-guided robust sketch-to-face image generation method and system

Abstract

The application discloses a robust sketch-face image generation method and system based on text guidance, and the method comprises the following steps: receiving a sketch image and a text description, the text is divided into dataset-level and feature-level texts according to semantic granularity and is encoded respectively, and the sketch image is also encoded; inputting the two types of text features into a text-guided sparse gating network, fusing the features through a two-stage routing decision and a Top-K sparse activation strategy; and injecting the fused features into a ControlNet architecture conditional generation model to generate a real face image in cooperation with a truncated accelerated sampling strategy. Through the synergistic effect of the innovative technologies such as double-condition input of'sketch + text', ControlNet constraint, pre-training GAN and diffusion model fusion, the three goals of precise control, high-quality output and efficient generation are realized, and the combined effect is significantly better than that of the prior art.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, specifically to a text-guided robust sketch-face image generation method and system. Background Technology

[0002] Facial image reconstruction, specifically referring to the automatic generation of realistic facial images that retain original identity features from input sketch images, is a key task in the fields of computer vision and image processing. This technology has significant application value in public security (such as forensic sketching), digital entertainment (such as character generation and animation production), and human-computer interaction (such as virtual avatar creation). Its core objective is not only to require the generated images to be visually realistic and natural, but also to accurately restore the person's identity features to ensure the usability of the generated images in serious applications such as identity authentication.

[0003] Early traditional image reconstruction methods relied primarily on manually designed prior knowledge and image inpainting techniques, such as image filtering, texture synthesis, and reconstruction based on 3D deformation models. These methods can achieve basic contour-to-image conversion, but their performance is heavily dependent on the completeness and accuracy of the prior model. When faced with different painting styles, complex line structures, and diverse facial features, traditional methods often perform poorly, producing results that generally lack realism, have blurred details, and low identity recognition, failing to meet the accuracy and robustness requirements of practical applications.

[0004] With breakthroughs in deep learning, methods based on Generative Adversarial Networks (GANs) have gradually become the mainstream in this field. GANs learn a complex nonlinear mapping from the sketch domain to the real face image domain through adversarial training between the generator and discriminator, achieving significant progress in image realism. However, GAN methods have inherent limitations: First, their training process is unstable and prone to mode collapse, leading to artifacts such as facial distortion and unnatural textures in the generated images. Second, and more critically, most GAN methods lack effective identity feature preservation mechanisms, resulting in insufficient consistency between the generated face images and the identity information of the original sketch in key facial recognition features. This leads to a significant decrease in recognition rates in face recognition systems, making them unsuitable for high-reliability scenarios such as identity authentication. Furthermore, existing GAN models are typically trained on sketch data of a specific style (e.g., a specific dataset), exhibiting weak generalization ability across different painting styles and dataset distributions, with a significant performance drop during cross-style testing.

[0005] In recent years, diffusion models have demonstrated outstanding performance in image generation. Through progressive denoising forward and backward processes, they can generate images with rich detail and extremely high quality, generally outperforming GANs in terms of realism and detail. However, directly applying diffusion models to the task of converting sketches to faces still faces a series of challenges: 1) Low inference efficiency: Standard diffusion models typically require thousands of sampling iterations for inference, resulting in slow generation speeds that cannot meet the real-time requirements of applications such as public safety; 2) Insufficient conditional control and semantic understanding: Existing methods, when introducing sketches as conditions, often employ simple conditional injection methods (such as concatenating sketch images and noise images along the channel dimension), or use general text encoders (such as CLIP) to encode simple text descriptions and fuse them through attention mechanisms. This approach lacks an understanding of the deeper semantic content behind the sketch lines (such as facial structure and identity features), making it difficult to achieve effective and accurate interaction between the sketch image and the generation process. In particular, it has limited ability to process stylized and abstract sketches. 3) Poor cross-domain generalization ability: When tested across datasets with different styles and distributions, the model is prone to feature confusion and loss of identity information, resulting in unstable generation quality.

[0006] To improve the accuracy of conditional control, researchers have proposed conditional control network architectures such as ControlNet, which guide the generation process by injecting input conditions (such as edge maps and pose maps) into the diffusion model in a trainable manner. However, existing methods typically treat sketches only as low-level edge or contour information, failing to fully explore and utilize their high-level semantic information. For sketches with simplified lines or exaggerated styles, the reconstruction results are poor. Regarding model training strategies, to improve generalization, existing research often simply merges multiple sketch datasets of different styles for training, but ignores the significant differences between datasets in terms of painting style, line density, and level of abstraction. This simplistic data mixing strategy makes it difficult for the model to learn a unified and robust cross-style feature representation, and may even lead to model confusion during the learning process.

[0007] In terms of accelerating inference, existing methods such as DDIM and knowledge distillation can reduce sampling steps, but they usually still require dozens of sampling steps to ensure quality. They cannot generate high-quality images in a very small number of steps (e.g., 2-4 steps), and the acceleration process often comes at the cost of sacrificing image detail and the accuracy of identity features. It is particularly noteworthy that most existing research focuses on improving the visual quality of the generated images (such as FID and IS scores), generally neglecting the performance of the generated results in practical face recognition systems. While the reconstructed images may be visually satisfactory, their recognition rate in identity authentication tasks often falls short of practical application levels, which greatly limits the practical value of this technology.

[0008] In summary, existing sketch-to-face image reconstruction methods still have significant shortcomings in several key aspects, including image realism and naturalness, identity feature preservation, inference generation speed, and cross-style and cross-dataset generalization ability. Particularly in terms of identity feature preservation, the lack of an effective mechanism to ensure consistency between the generated image and the input sketch leads to low face recognition rates. Therefore, there is an urgent need in this field for a novel technical solution that can systematically address the above problems, achieving fast, high-quality, high-identity-fidelity, and highly generalizable sketch-to-face reconstruction to meet the pressing needs of serious application scenarios such as public security and identity authentication. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a text-guided robust sketch-face image generation method and system, aiming to solve the problems in the background technology.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a text-guided robust sketch-face image generation method, comprising the following steps:

[0011] Step S1, Text Description Segmentation and Encoding Processing: Receive the input sketch image and text description, divide the text description into two parts according to semantic granularity: "dataset-level text" and "feature-level text", and encode them separately to obtain dataset-level text features and feature-level text features; encode the sketch image to obtain sketch image encoded features;

[0012] Step S2: Text-guided sparse gating network realizes expert network activation and feature fusion: Dataset-level text features and feature-level text features are input into the text-guided sparse gating network, and the fused features are output; The sparse gating network adopts the Top-K sparse activation strategy and refines the expert selection process through a two-stage routing decision, dynamically calculates the activation weight of each expert network, and thus realizes expert network activation and feature fusion.

[0013] Step S3: Generate real face images using a conditional generation model based on the ControlNet architecture: Inject the fused features as conditional control signals into the conditional generation model based on the ControlNet architecture, and use the accelerated sampling strategy to generate the final real face images.

[0014] Furthermore, in step S1, the dataset-level text comes from the annotations of the overall style or scene of each dataset, used to represent global semantic information; the dataset-level text is processed by the dataset-level text encoder to extract dataset-level text features; the feature-level text comes from the annotations or descriptions of fine-grained facial attributes, and the feature-level text is processed by the feature-level text encoder to extract feature-level text features.

[0015] Furthermore, the specific process of step S2 is as follows:

[0016] Step S2.1, Two-stage routing decision:

[0017] Coarse-grained expert selection: A learnable parameter matrix related to the dataset style is set for each expert network, and the first parameter is calculated via dot product. An expert network analyzes data-level text features. Response score ;

[0018] The response scores of all expert networks to dataset-level text features are combined into a vector. ;

[0019] according to Select the highest score A network of experts was used as a candidate set;

[0020] Fine-grained expert activation: In the candidate set, learnable parameters are set for the fine-grained attributes of each expert network, and the activation of the first expert network is calculated. An expert network analyzes feature-level text features. Response score ;

[0021] All expert networks on feature-level text features The response score is composed of a vector. ;

[0022] "Dual gating" is implemented using element-wise multiplication to score the response. and response score Fusion, to obtain the first The final activation weights of the expert network Combine the final activation weights of all expert networks to obtain the combined weights. ;

[0023] Combine weights After normalization and sorting by numerical value, select the combined weights. The largest front One expert network is activated, and the rights of unselected expert networks are reset to zero;

[0024] Step S2.2, Expert Network Activation and Feature Fusion: The activated expert network receives the encoded features of the sketch image, performs specific transformations, and produces the expert network output features; the final activation weights are then used. The output features of the expert network are weighted and summed to achieve hierarchical feature fusion.

[0025] Step S2.3, Interaction between fused features and sketch features: The fused expert network output features Deeply fuse with the encoded features of the sketch image to form semantically enhanced fused features. .

[0026] Furthermore, the specific process of step S3 is as follows:

[0027] Step S3.1, Conditional Fusion and Control: Construct a conditional generative model based on the ControlNet architecture, and fuse features. Injected as a control condition;

[0028] Step S3.2, Accelerated Sampling Strategy: Using a conditional generation model based on the ControlNet architecture, a coarse-to-fine truncation accelerated sampling strategy is adopted to generate the final image of the real human face.

[0029] Furthermore, the conditional generative model based on the ControlNet architecture is a conditional control model formed by additionally integrating the ControlNet module on top of the pre-trained basic generative model.

[0030] Furthermore, the conditional generative model control flow based on the ControlNet architecture is defined as follows:

[0031] ;

[0032] In the formula, This represents the output features of a conditional generative model based on the ControlNet architecture; This represents the backbone network of the pre-trained basic generative model; Represents the fixed parameters of the pre-trained base generative model; Indicates a zero convolution operation; , Each represents a learnable parameter of a zero convolutional layer, corresponding to the weight parameters in a zero convolution operation; This represents the trainable copy parameters of the encoder part of the pre-trained base generative model; This represents the basic input features of the pre-trained basic generative model.

[0033] Furthermore, the specific process of step S3.2 is as follows: Initial face images are quickly generated using a pre-trained generative adversarial network. ; by setting a preset number of cutoff steps The initial face image The denoising starting point of the diffusion model is mapped; based on the denoising starting point, a conditional generative model based on the ControlNet architecture is used for finite-step reverse denoising and refinement to obtain the final real face image.

[0034] A text-guided robust sketch-face image generation system includes:

[0035] The data processing module is used to receive the input sketch image and text description, divide the text description into two parts according to semantic granularity: "dataset-level text" and "feature-level text", and encode them separately to obtain dataset-level text features and feature-level text features; and encode the sketch image to obtain sketch image encoded features.

[0036] The expert network activation and feature fusion module is used to input dataset-level text features and feature-level text features into a text-guided sparse gating network and output fused features. The sparse gating network adopts a Top-K sparse activation strategy and refines the expert selection process through a two-stage routing decision, dynamically calculates the activation weights of each expert network, and thus realizes expert network activation and feature fusion.

[0037] The image generation module is used to inject the fused features as conditional control signals into the conditional generation model based on the ControlNet architecture, and in conjunction with the accelerated sampling strategy, generate the final realistic face image.

[0038] An electronic device includes a processor, a memory, and a bus, wherein the processor and the memory are connected via the bus, wherein the memory is used to store a set of program code, and the processor is used to call the program code stored in the memory to execute a text-guided robust sketch-face image generation method.

[0039] A non-volatile computer storage medium storing computer-executable instructions that execute a text-guided robust sketch-face image generation method.

[0040] Compared with existing technologies, the present invention has the following advantages:

[0041] (1) This invention overcomes the performance limitations of existing single-technology solutions by synergistically applying a number of innovative technologies, including dual-condition input of "sketch + text", ControlNet conditional constraint architecture, fusion generation of pre-trained GAN and diffusion model, truncation acceleration sampling strategy and loss function adapted to dual conditions. The features of each technology support each other and complement each other in terms of function. It ensures the accuracy and controllability of generation through dual-condition input and ControlNet architecture, improves the generation quality with the fusion architecture of GAN and diffusion model, and optimizes the generation efficiency with truncation sampling. Finally, it achieves the triple goal of "precise control, high-quality output and efficient generation". The combined effect is far greater than the sum of the effects of each single technology and is significantly better than existing generation methods.

[0042] (2) This invention relies on a dual-condition design of “sketching input providing structural constraints + text guidance clarifying style attributes”, combined with ControlNet’s fine-grained intervention in the generation process, to ensure that the generated face image not only strictly conforms to the core structural features of the sketch, such as outline, posture, and facial feature layout, but also accurately reproduces the attribute requirements of the text, such as expression, style, and detail texture, thus solving the problems of structural loss of control and style deviation in traditional generation methods. At the same time, the structural stability of the GAN model is combined with the anti-pattern collapse and detail restoration capabilities of the diffusion model, making the generated image closer to the distribution of real face data in terms of facial feature proportions, skin texture, and detail richness, which greatly improves the realism and visual effect of the generated image.

[0043] (3) This invention skips the redundant forward diffusion steps of the diffusion model by using a preset truncation step acceleration sampling strategy. It directly starts iterating from the denoising starting point of the initial face generated by the adapted GAN. Without losing the generation quality, it significantly reduces the number of sampling iterations, effectively solving the problem of excessive generation time in traditional diffusion models and meeting the high-efficiency generation requirements in practical applications. In addition, for sketch inputs of different qualities (such as blurred lines and missing details) and diverse text descriptions, the model has strong anti-interference ability by fusing feature optimization and conditional feature adaptation mechanisms. It can stably output images that meet the requirements without complex preprocessing, adapting to more diverse practical application scenarios. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0045] like Figure 1 As shown, the present invention provides a technical solution: a text-guided robust sketch-face image generation method, comprising the following steps:

[0046] Step S1: Text description segmentation and encoding processing: Receive the input sketch image and text description, divide the text description into two parts according to semantic granularity: "dataset-level text" and "feature-level text", and encode them separately to obtain dataset-level text features and feature-level text features; encode the sketch image to obtain sketch image encoded features.

[0047] Dataset-level text encoding: The dataset-level text originates from the annotations of the overall style or scene of each dataset during joint training of multiple datasets, used to represent global semantic information (e.g., the AR dataset is labeled "real face photo with white background", the XM2VTS dataset is labeled "real face photo with dark blue background", and the CUHK dataset is labeled "real face photo with blue background"). During the inference phase, the dataset-level text can be provided by the user or extracted from the complete text description according to preset rules (e.g., identifying background environment or overall style terms based on keywords). The dataset-level text is processed by a dataset-level text encoder to extract dataset-level text features representing the overall style semantics. The encoding process is represented as follows:

[0048] ;

[0049] In the formula, Represents dataset-level text features; Represents dataset-level text; This represents a dataset-level text encoder; This represents the learnable parameter matrix corresponding to the dataset-level text encoding. This represents a non-linear activation function.

[0050] Feature-level text encoding: The feature-level text originates from the annotation or description of fine-grained facial attributes, used to reflect local detailed semantic information; in practical applications, the feature-level text can be provided by user-provided descriptions of facial attributes; the feature-level text is processed by a feature-level text encoder to extract feature-level text features representing fine-grained semantics. The encoding process is represented as follows:

[0051] ;

[0052] In the formula, Representing feature-level text features; Represents feature-level text; This indicates a feature-level text encoder; This represents the learnable mapping matrix corresponding to the feature-level text encoding.

[0053] Step S2: Text-guided sparse gating network realizes expert network activation and feature fusion: Dataset-level text features and feature-level text features are input into the text-guided sparse gating network, and the fused features are output. The sparse gating network adopts the Top-K sparse activation strategy and refines the expert selection process through a two-stage routing decision, dynamically calculates the activation weights of each expert network, and thus realizes expert network activation and feature fusion.

[0054] The specific process of step S2 is as follows:

[0055] Step S2.1, Two-stage routing decision:

[0056] Coarse-grained expert selection (dataset-level guidance): A learnable parameter matrix related to the dataset style is set for each expert network, and the first parameter is calculated via dot product. An expert network analyzes data-level text features. Response score :

[0057] ;

[0058] In the formula, Indicates the first A gated learnable parameter matrix corresponding to the dataset style of each expert network; This represents the activation function, used to normalize the output to a non-negative score;

[0059] The response scores of all expert networks to dataset-level text features are combined into a vector. , Indicates the first The response scores of an expert network to dataset-level text features;

[0060] according to Select the highest score Using a set of expert networks as candidates, we first screen out expert networks that are obviously not a good match for the style of the current scene, and then retain only the expert networks that are relevant to the text features of the dataset to proceed to the next step.

[0061] Fine-grained expert activation (feature-level guidance): In the candidate set, learnable parameters are set for the fine-grained attributes of each expert network, and the first learnable parameter is calculated. An expert network analyzes feature-level text features. Response score :

[0062] ;

[0063] In the formula, Indicates the first The gating learnable parameter matrix corresponding to each expert network and related to fine-grained attributes;

[0064] All expert networks on feature-level text features The response score is composed of a vector. , Indicates the first The response scores of an expert network to feature-level text features;

[0065] "Dual gating" is implemented using element-wise multiplication to score the response. and response score Fusion, to obtain the first The final activation weights of the expert network Combine the final activation weights of all expert networks to obtain the combined weights. ;

[0066] When an expert network simultaneously meets the data set-level text features and feature-level text features At that time, its final activation weight The higher the product, the greater the activation level of the expert network; conversely, if there is a mismatch in one aspect, the product result will reduce the activation level of the expert network.

[0067] Combine weights After normalization and sorting by numerical value, select the combined weights. The largest front A network of experts was activated (Top-K selection). The value range is 1 to 4), and the unselected expert network weights are reset to zero to achieve sparse activation.

[0068] Step S2.2, Expert Network Activation and Feature Fusion: The activated expert network receives the encoded features of the sketch image, performs specific transformations, and produces expert network output features (each expert network is trained with different dataset styles and attribute combinations, and can learn the feature extraction and transformation capabilities for a certain type of semantics); using The weighted summation of the expert network output features achieves hierarchical feature fusion, as shown below:

[0069] ;

[0070] In the formula, Indicates the first The output characteristics of an activated expert network; This represents the output characteristics of the fused expert network.

[0071] Step S2.3, Interaction between fused features and sketch features: The fused expert network output features Deeply fuse with the encoded features of the sketch image (i.e., connect at the channel dimension) to form semantically enhanced fused features. The fusion feature It simultaneously contains structural information from the input sketch and high-level semantic information from the text description, achieving a combination of visual features and semantic guidance.

[0072] Step S3: Generate real face images using a conditional generation model based on the ControlNet architecture: Inject the fused features as conditional control signals into the conditional generation model based on the ControlNet architecture, and use the accelerated sampling strategy to generate the final real face images.

[0073] The specific process of step S3 is as follows:

[0074] Step S3.1, Conditional Fusion and Control: Construct a conditional generative model based on the ControlNet architecture (this model is a conditional control model formed by additionally integrating the ControlNet module on top of the pre-trained basic generative model), and fuse features. As a control condition injection; the conditional generative model control flow based on the ControlNet architecture is defined as follows:

[0075] ;

[0076] In the formula, This represents the output features of the conditional generative model based on the ControlNet architecture. It is the result of the conditional generative model processing and is used to generate realistic human face images. This represents the backbone network of the pre-trained basic generative model; This represents the fixed parameters of the pre-trained base generative model (these parameters are frozen in the ControlNet architecture to preserve pre-trained knowledge). Indicates a zero convolution operation; , Each represents a learnable parameter of a zero convolutional layer, corresponding to the weight parameters in a zero convolution operation; This represents the trainable copy parameters of the encoder part of the pre-trained base generative model; This represents the basic input features of the pre-trained basic generative model.

[0077] To improve the model's practicality, this invention also designs a truncation-based accelerated sampling strategy. Traditional diffusion models require starting with pure Gaussian noise and undergoing hundreds or even thousands of iterative denoising steps to generate the final image. This process can be represented as:

[0078] ;

[0079] ;

[0080] In the formula, This indicates the final step (the first step) of the forward diffusion process. The initial noisy image corresponding to step (step) follows a standard multivariate normal distribution. , Represents the unit covariance matrix; , These represent the first and second stages of the diffusion process, respectively. , The noisy image of the step; Indicates the first The diffusion coefficient of the step; Indicates the preceding The cumulative product of the diffusion coefficients; This represents a noise prediction network under an accelerated sampling strategy; Indicates the first The standard deviation of the random noise added during the noise reduction process; This represents random noise that follows a standard normal distribution.

[0081] Step S3.2, Accelerated Sampling Strategy: Using a conditional generation model based on the ControlNet architecture, a coarse-to-fine truncation accelerated sampling strategy is employed to generate the final, realistic face image. Specifically:

[0082] Use a pre-trained generative adversarial network to quickly generate initial face images. :

[0083] ;

[0084] In the formula, This represents the generator of a pre-trained generative adversarial network. This represents random noise input into a pre-trained generative adversarial network;

[0085] By a preset number of cutoff steps The initial face image Mapped to the denoising starting point of the diffusion model:

[0086] ;

[0087] In the formula, The starting point for denoising in the diffusion model is the initial face image. The initial input to the diffusion model denoising process after mixing with noise; Indicates the corresponding cutoff step number The cumulative product of the diffusion coefficients is used to control the mixing weights of the initial face image and noise; This represents random noise sampled from a standard multivariate normal distribution;

[0088] by Starting from this point, a conditional generative model based on the ControlNet architecture is used to perform a finite number of steps (2-4 steps) of reverse denoising and refinement to obtain the final realistic face image;

[0089] Among them, the optimization objective of the conditional generative model based on the ControlNet architecture is determined by the loss function. definition:

[0090] ;

[0091] In the formula; This represents the original, real-life human face image, which serves as the target reference image. Expressing expectations; Representation and diffusion steps The purpose of adapting the conditional control features is to enable the conditional signals to adapt to the noise images at different steps of the diffusion model, ensuring that the conditional control is effective in each step of denoising. Represents the encoded features of a sketch image; Indicates the first step in the diffusion process A noisy image of the step.

[0092] A text-guided robust sketch-face image generation system includes:

[0093] The data processing module is used to receive the input sketch image and text description, divide the text description into two parts according to semantic granularity: "dataset-level text" and "feature-level text", and encode them separately to obtain dataset-level text features and feature-level text features; and encode the sketch image to obtain sketch image encoded features.

[0094] The expert network activation and feature fusion module is used to input dataset-level text features and feature-level text features into a text-guided sparse gating network and output fused features. The sparse gating network adopts a Top-K sparse activation strategy and refines the expert selection process through a two-stage routing decision, dynamically calculates the activation weights of each expert network, and thus realizes expert network activation and feature fusion.

[0095] The image generation module is used to inject the fused features as conditional control signals into the conditional generation model based on the ControlNet architecture, and in conjunction with the accelerated sampling strategy, generate the final realistic face image.

[0096] An electronic device includes a processor, a memory, and a bus, wherein the processor and the memory are connected via the bus, wherein the memory is used to store a set of program code, and the processor is used to call the program code stored in the memory to execute a text-guided robust sketch-face image generation method.

[0097] A non-volatile computer storage medium storing computer-executable instructions that execute a text-guided robust sketch-face image generation method.

[0098] After image generation is completed, the generation quality can be verified through quantitative indicators and qualitative visual effects: quantitatively, indicators such as LPIPS, FID, and CKDN are calculated to ensure the perceptual similarity and distribution consistency between the generated image and the real image; qualitatively, the performance is further verified by manually observing the facial naturalness, texture details, identity features and matching degree between the generated image and the input sketch.

[0099] In terms of quantitative evaluation, this invention demonstrates superior performance on multiple authoritative metrics. As shown in Table 1, in the reference image quality assessment, this method achieves significant advantages on key metrics such as LPIPS (Learned Perceptual Patch Similarity), CKDN, and FID on the AR dataset. Specifically, the excellent performance on the LPIPS metric proves that the generated images are highly close to the real images in perceptual quality; the outstanding performance on the CKDN metric reflects the model's ability in detail preservation and noise suppression; and the significant advantage on the FID metric indicates that the distribution of the generated images is highly consistent with the distribution of the real data. In addition, it also achieves state-of-the-art levels on other parametric metrics such as FSIM, PSNR, SSIM, CW-SSIM, NLPD, and VIF.

[0100] Table 1 Comparison of reference image quality assessment metrics

[0101]

[0102] As shown in Table 2, in terms of no-reference image quality assessment, this invention demonstrates excellent performance across multiple metrics, including CLIPIQA+, CNNIQA, DBCNN, MUSIQ, PAQ2PIQ, BRISQUE, PI, NIQE, and TRES. Particularly noteworthy is the outstanding performance on deep learning-based no-reference metrics such as CLIPIQA+ and CNNIQA, which verifies that the generated images conform to human visual perception characteristics; while the stable performance on traditional no-reference metrics proves the statistical authenticity of the generated images.

[0103] Table 2 Comparison of Quality Assessment Indicators for No-Reference Images

[0104]

[0105] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A text-guided robust sketch-face image generation method, characterized in that, Includes the following steps: Step S1: Text description segmentation and encoding processing: Receive the input sketch image and text description, divide the text description into two parts according to semantic granularity: dataset-level text and feature-level text, and encode them separately to obtain dataset-level text features and feature-level text features. The sketch image is encoded to obtain the coded features of the sketch image; Step S2: Text-guided sparse gating network enables expert network activation and feature fusion: Input dataset-level text features and feature-level text features into the text-guided sparse gating network, and output fused features; The sparse gating network adopts a Top-K sparse activation strategy and refines the expert selection process through a two-stage routing decision, dynamically calculating the activation weights of each expert network, thereby realizing expert network activation and feature fusion. Step S3: Generate real face images using a conditional generation model based on the ControlNet architecture: Inject the fused features as conditional control signals into the conditional generation model based on the ControlNet architecture, and use the accelerated sampling strategy to generate the final real face images. In step S1, the dataset-level text comes from the annotation of the overall style or scene of each dataset, which is used to represent global semantic information; the dataset-level text is processed by the dataset-level text encoder to extract dataset-level text features; Feature-level text comes from annotations or descriptions of fine-grained facial attributes. The feature-level text is processed by a feature-level text encoder to extract feature-level text features. The specific process of step S2 is as follows: Step S2.1, Two-stage routing decision: Coarse-grained expert selection: A learnable parameter matrix related to the dataset style is set for each expert network, and the first parameter is calculated via dot product. An expert network analyzes data-level text features. Response score ; The response scores of all expert networks to dataset-level text features are combined into a vector. ; according to Select the highest score A network of experts was used as a candidate set; Fine-grained expert activation: In the candidate set, learnable parameters are set for the fine-grained attributes of each expert network, and the activation of the first expert network is calculated. An expert network analyzes feature-level text features. Response score ; All expert networks on feature-level text features The response score is composed of a vector. ; "Dual gating" is implemented using element-wise multiplication to score the response. and response score Fusion, to obtain the first The final activation weights of the expert network ; The final activation weights of all expert networks are combined to obtain the combined weights. ; Combine weights After normalization and sorting by numerical value, select the combined weights. The largest front One expert network is activated, and the rights of unselected expert networks are reset to zero; Step S2.2, Expert Network Activation and Feature Fusion: The activated expert network receives the encoded features of the sketch image, performs specific transformations, and produces the expert network output features; Using final activation weights The output features of the expert network are weighted and summed to achieve hierarchical feature fusion. Step S2.3, Interaction between fused features and sketch features: The fused expert network output features Deeply fuse with the encoded features of the sketch image to form semantically enhanced fused features. ; The specific process of generating the final realistic face image using the accelerated sampling strategy is as follows: An initial face image is quickly generated using a pre-trained generative adversarial network. ; by setting a preset number of cutoff steps The initial face image This is mapped to the starting point for denoising in the diffusion model; Starting from the denoising point, a conditional generative model based on the ControlNet architecture is used for finite-step reverse denoising refinement to obtain the final realistic face image.

2. The text-guided robust sketch-face image generation method according to claim 1, characterized in that: Conditional generative models based on the ControlNet architecture are conditional control models formed by integrating ControlNet modules on top of pre-trained basic generative models.

3. The text-guided robust sketch-face image generation method according to claim 2, characterized in that: The conditional generative model control flow based on the ControlNet architecture is defined as follows: ； In the formula, This represents the output features of a conditional generative model based on the ControlNet architecture; This represents the backbone network of the pre-trained basic generative model; Represents the fixed parameters of the pre-trained base generative model; Indicates a zero convolution operation; , Each represents a learnable parameter of a zero convolutional layer, corresponding to the weight parameters in a zero convolution operation; This represents the trainable copy parameters of the encoder part of the pre-trained base generative model; This represents the basic input features of the pre-trained basic generative model.

4. A text-guided robust sketch-face image generation system, used to execute the text-guided robust sketch-face image generation method according to any one of claims 1-3, characterized in that: The data processing module is used to receive the input sketch image and text description, divide the text description into two parts according to semantic granularity: dataset-level text and feature-level text, and encode them separately to obtain dataset-level text features and feature-level text features. The sketch image is encoded to obtain the coded features of the sketch image; The expert network activation and feature fusion module is used to input dataset-level text features and feature-level text features into a text-guided sparse gating network and output fused features. The sparse gating network adopts a Top-K sparse activation strategy and refines the expert selection process through a two-stage routing decision, dynamically calculating the activation weights of each expert network, thereby realizing expert network activation and feature fusion. The image generation module is used to inject the fused features as conditional control signals into the conditional generation model based on the ControlNet architecture, and in conjunction with the accelerated sampling strategy, generate the final realistic face image.

5. An electronic device, characterized in that, The method includes a processor, a memory, and a bus, wherein the processor and the memory are connected via the bus, the memory is used to store a set of program code, and the processor is used to call the program code stored in the memory to execute the text-guided robust sketch-face image generation method as described in any one of claims 1-3.

6. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer can execute instructions to perform the text-guided robust sketch-face image generation method according to any one of claims 1-3.

Images