A vq priori resampling face editing method based on inductive moment matching and decoupled guidance

Abstract

This invention discloses a VQ prior resampling face editing method based on inductive moment matching and decoupled guidance. It constructs a multimodal guided fusion network, a VQ dictionary prior module, and an inductive moment matching generation model, addressing the issues of missing details, identity distortion, and low naturalness indices in face forgery. This method employs an inductive moment matching generation framework to achieve efficient single-step or few-step inference. During training, a decoupled pose-identity attention module is introduced, using source image identity features as keys and values, and target image pose as a query, resolving the coupling problem between identity and pose. A gated AdaIN mechanism is used to balance text and pose guidance, followed by style feature injection, resolving the conflict between content and style guidance. Furthermore, a VQ dictionary face prior module is introduced during training, constraining the model output through additional perceptual loss to make its feature distribution approximate real face components, significantly improving the realism of the generated images.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance. Background Technology

[0002] With the increasing demand for entertainment and personalized content, highly realistic face editing technology has become a key research area. Utilizing deep learning techniques, especially generative models (such as GANs and diffusion models), to automatically and accurately edit target attributes from input images is currently the mainstream research direction in this field.

[0003] However, existing state-of-the-art (SOTA) face editing technologies still face several unresolved problems and technical shortcomings in practical applications: First, there is the problem of strong coupling between identity and pose. When performing pose or expression transfer (e.g., generating an image using A's identity and B's pose), existing methods struggle to accurately transfer the target pose while fully preserving the identity (ID) information of the source image. Source ID features (such as face shape and facial features) are easily "contaminated" or "covered" by the target pose, resulting in a final result that does not resemble the original person.

[0004] Second, there is the issue of conflicting multimodal guidance. Modern editing tasks typically require multiple guidance modalities, such as text ("put on glasses"), pose (target image), and style (painting style). Existing methods often encounter conflicts when fusing these different modalities of guidance. For example, if the style guidance is too strong, the model may ignore text instructions; and vice versa.

[0005] Third, there is a loss of detail and low natural sensory index. Human faces are highly detailed objects. Existing methods often result in the loss of detail (such as skin texture and pores) and blurred edges in the faked areas after editing, or artifacts in the unedited areas. The generated images perform poorly in terms of natural sensory index, appearing "fake" or "plastic".

[0006] Fourth, the three dilemmas of training frameworks. Achieving high-quality generation, efficient inference, and stable training is a core challenge. Traditional diffusion models are slow inference; knowledge distillation methods (such as VSD) require complex two-stage "teacher-student" training and depend on the quality of the teacher model; other low-step models (such as Consistency Models) often face problems of training instability and hyperparameter sensitivity.

[0007] Therefore, there is an urgent need in this field for a new technical solution that can stably train an efficient, low-step model from scratch, which can logically decouple the control of identity, pose, content and style, and ensure the high realism and detail of the generated faces. Summary of the Invention

[0008] The purpose of this invention is to address the problems of low identity preservation, multimodal guidance conflict, detail distortion and training instability in current face editing technologies, and to propose a VQ prior resampling face editing method and system based on inductive moment matching and decoupling guidance.

[0009] This invention addresses the aforementioned technical problems primarily through the following approaches: On one hand, it employs inductive moment matching as the core generative framework, training an efficient, low-step model from scratch using a single-stage inductive bootstrapping method, thus resolving the dependence of VSD distillation on the teacher model and the training instability of Consistency Models; on the other hand, it integrates decoupled multimodal guidance and VQ face priors in the training and inference of inductive moment matching to ensure the accuracy of editing and the high realism of the generated data.

[0010] The objective of this invention is achieved through the following technical solution: a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance, comprising: The multimodal dataset is preprocessed, and the VQ-VAE model is pre-trained. Multimodal encoders are used to extract features from multimodal data, which are then input into a U-Net network whose attention module is replaced by a decoupled pose-identity attention and gating-AdaIN style fusion computation. The VQ prior loss obtained from the VQ-VAE model is combined with the inductive moment matching core loss for single-stage training. The trained network model is used for staged model inference: in the high-step stage, only content and structure guidance are used to generate semantic skeletons; in the intermediate steps, noise is re-injected; and in the low-step stage, style guidance is activated to render details and style; finally, a high-fidelity face image is output.

[0011] Furthermore, in the preprocessing of the multimodal dataset, the dataset includes: source images, target images, edited text, and style images. The dataset is normalized in terms of image size and pixel value, and different encoders are selected for feature extraction.

[0012] Furthermore, the pre-training of the VQ-VAE model specifically includes: The Vector Quantization Autoencoder (VQ-VAE) model consists of an encoder, a decoder, and a learnable VQ dictionary. A pre-training dataset is constructed using large-scale, high-resolution real face images containing local facial components and texture details. Training and convergence: The vector quantization model is trained on the pre-training dataset. By minimizing the reconstruction loss and commitment loss, the VQ dictionary is converged into a codebook containing K discrete quantization vectors, where each quantization vector represents a local component feature of a real face. Parameter freezing: After the VQ model training converges, all parameters of the encoder, decoder and VQ dictionary D are set to a frozen state, as a fixed face prior knowledge module, which is used to calculate the VQ prior loss in step three.

[0013] Furthermore, the decoupled pose-identity attention and gating-AdaIN style fusion computation includes: Decoupled Pose-Identity Attention: The pose / expression features of the target image f_target are used as the query Q, and the identity details of the source image f_source are used as the key K and value V; calculate out_qkv = Attention(Q=f_target, K=f_source, V=f_source) to obtain the structural features that fuse the identity of the source image and the pose of the target image. The gating system incorporates AdaIN's style, specifically: First, the text editing guidance feature out_text and the style guidance feature out_style are calculated. Then, a learnable gating weight g is used to balance content and structure; f_content_structure = (g * out_text) + ((1- g) * out_qkv); Use cross-modal adaptive instance normalization to inject style into the generated image; f_final_guidance = AdaIN(content=f_content_structure, style=out_style); Finally, f_final_guidance is added back to the image features of U-Net.

[0014] Furthermore, the inductive moment matching core loss specifically includes: Three random time points s, r, and t are sampled, satisfying s < r < t, and real data x and noise ε are sampled from the dataset; Calculate the latent variable x_t at time t and the latent variable x_r at time r; Calculate the predicted distribution P: y_s,t = (x_t, s, t, All_Guides), where The model with the current training parameters; Calculate the target distribution Q: y_s,r = -(x_r, s, r, All_Guides), where - Model parameters for stopping gradients; Calculate L_IMM: L_IMM = (Average similarity within P + Average similarity within Q) - 2 * (Average similarity between P and Q). Where the similarity within P is k(y_s,t, y_s,t'), meaning the similarity is calculated by taking two samples from the predicted distribution P; the similarity within Q is k(y_s,r, y_s,r'), meaning the similarity is calculated by taking two samples from the target distribution Q; and the similarity between P and Q is k(y_s,t, y_s,r), meaning the similarity is calculated by taking one sample from P and one sample from Q respectively.

[0015] Furthermore, the VQ prior loss specifically includes: Obtain the student model output: match the inductive moment generator. Latent variables generated in the current training iteration _student is fed into the main decoder VAE Decoder, which decodes it into a pixel-level face image x_student; Feature extraction: Input the x_student into the VQ encoder E_vq pre-trained and frozen in step 2 to extract its continuous intermediate feature maps z_e; Feature quantization: In the VQ dictionary, for each feature vector in the z_e feature map, the nearest neighbor query is used to find and replace it with the most matching discrete quantization vector in the dictionary to obtain the quantized feature map z_q; Quantization Reconstruction: The quantized feature map z_q is input into the frozen VQ decoder G_vq to reconstruct an image x_recon that is entirely composed of real face component features; Loss calculation: The perceptual loss between the image generated by the x_student student model and the image composed of the x_recon discrete vector dictionary is calculated to obtain the VQ prior loss L_VQ_Prior. L_VQ_Prior is used for backpropagation to update the generator. The parameters are used to train the generated image features to actively approximate the feature distribution of a real human face.

[0016] Furthermore, the total loss in single-stage training using the inductive moment matching core loss combined with the VQ prior loss includes: a weighted fusion of the VQ prior loss, the inductive moment matching core loss, the identity preservation loss calculated by the pre-trained face recognition network, the reconstruction loss to ensure the consistency of non-edited regions, and the adversarial loss introduced in the later stages of training.

[0017] Furthermore, the specific method of using only content and structure to guide the generation of semantic skeletons in the high-step stage is as follows: In the high-step denoising stage, the U-Net network only runs under decoupled pose-identity attention guidance and text guidance, generating latent variables z_cutoff containing the correct identity, pose, and content; Noise is re-injected into z_cutoff to obtain z_restart_new; denoising begins from z_restart_new until t=0. The network model runs together under all bootstraps to render style and details on the established structure and perform style transfer.

[0018] According to another aspect of the specification, a VQ prior resampling face editing device based on inductive moment matching and decoupling guidance is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the VQ prior resampling face editing method based on inductive moment matching and decoupling guidance.

[0019] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the described VQ prior resampling face editing method based on inductive moment matching and decoupling guidance.

[0020] The beneficial effects of this invention are: Efficient and stable training is achieved by using an inductive moment matching framework, enabling single-stage, out-of-the-box training without relying on a teacher model. The training process is more stable than Consistency Models, while also enabling efficient single-step or few-step inference.

[0021] The precise decoupling of identity and pose, achieved through the QKV-Attention mechanism, ensures that the identity features of the source image are preserved with high fidelity, while accurately transferring the pose and expression of the target image.

[0022] Conflict-free multimodal guidance, the Gated-AdaIN fusion mechanism, processes guidance according to priority, first merging content and structure, and then injecting style, effectively resolving the conflict between text instructions and style instructions.

[0023] High realism and detail preservation are achieved through the powerful regularization term L_VQ_Prior, which forces the model to generate images that conform to the real human facial anatomy and skin texture, significantly improving the "natural sensory index".

[0024] The phased fine-grained control and the Restart Sampling inference strategy separate semantic generation and style rendering at time steps, enabling fine-grained control over the editing process. Attached Figure Description

[0025] Figure 1 A system architecture block diagram of a face editing method based on inductive moment matching provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the data flow of the decoupled pose-identity attention (QKV-Attention) module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the data flow of the gated-AdaIN style fusion module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the IMM single-stage training process provided in this embodiment of the invention, combined with VQ dictionary prior auxiliary loss; Figure 5 This is a flowchart of the Denoise-Repaint (Restart Sampling) phased inference strategy provided in this embodiment of the invention; Figure 6 This is a schematic diagram of a face editing device based on inductive moment matching provided in an embodiment of the present invention. Detailed Implementation

[0026] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0027] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention; however, the present invention may also be implemented in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0028] This invention provides a VQ prior face editing method based on inductive moment matching and decoupling guidance, such as... Figure 1 As shown, the method includes four core steps: Step 1, data preparation and VQ prior pre-training; Step 2, model architecture and forward propagation; Step 3, loss calculation and model training; Step 4, phased model inference.

[0029] Step 1: Data Preparation and VQ Prior Pre-training 1.1 Multimodal Data Preparation: Training data is prepared, mainly consisting of the following four categories: source face images x_source (providing identity), target face images x_target (providing pose / expression), style reference images x_style (providing style), and edited text c_text (providing content instructions), for example, using FFHQ or ff++ datasets. 1.2 Standardized Preprocessing: All input images (x_source, x_target, x_style) are uniformly normalized to a spatial resolution of [256, 256, 3], and pixel values ​​are normalized to the range of [-1, 1].

[0030] 1.3 VQ Dictionary (Codebook) Pre-training: To inject rich and realistic prior knowledge of faces into the model and address the issues of image detail distortion and low "natural sensory index" after editing, this invention first independently pre-trains a Vector Quantization (VQ-VAE) model. The VQ model is trained on a high-resolution face dataset (such as FFHQ). The VQ-VAE consists of an encoder E_vq, a decoder G_vq, and a VQ dictionary D.

[0031] The specific structure and training steps are as follows: a) VQ-VAE encoder E_vq: E_vq is responsible for mapping the input x (size [B, 3, 256, 256]) to a latent space feature map z_e (size [B, 256, 32, 32]). E_vq consists of one initial convolutional module, three downsampling modules (M=3), and one output convolutional module, as follows: Initial convolutional module: one 7x7 standard convolutional layer (Conv2d), stride 1, input channels 3, output channels C=128.

[0032] Downsampling Module 1 (M=1): One 3x3 convolutional layer with a stride of 2 (channel count 128->256), followed by N=2 residual blocks. Each residual block consists of two 3x3 convolutional layers, batch normalization, and a SiLU activation function, implemented through residual connections. The output feature map size of this module is [B, 256, 128, 128].

[0033] Downsampling Module 2 (M=2): One 3x3 convolutional layer with a stride of 2 (channel count 256->512), followed by N=2 residual blocks. The output feature map size of this module is [B, 512, 64, 64].

[0034] Downsampling Module 3 (M=3): One 3x3 convolutional layer with a stride of 2 (channel count 512->512), followed by N=2 residual blocks. The output feature map size of this module is [B, 512, 32, 32].

[0035] Output convolutional module: one 1x1 convolutional layer, which maps the number of channels from C=512 to the embedding dimension D=256, to obtain the final continuous feature map z_e (size [B, 256, 32, 32]).

[0036] b) VQ dictionary D: The VQ dictionary is a codebook D with a size of [K, D], where K is preferably 8192 (i.e., 8192 "blocks") and D is 256. D performs a nearest neighbor query between z_e ([B*32*32, 256]) and D ([8192, 256]) to obtain the quantized feature z_q.

[0037] c) VQ-VAE decoder G_vq: The G_vq structure is symmetrical to E_vq and is responsible for reconstructing the quantized feature map z_q (size [B, 256, 32, 32]) into a pixel-level face image x_recon (size [B, 3, 256, 256]), as follows: Initial convolutional module: one 1x1 convolutional layer that maps the number of channels from D=256 back to C=512.

[0038] Upsampling Module 1 (M=1): Contains N=2 residual blocks, followed by an upsampling layer (ConvTranspose2d, stride 2), which changes the number of channels from C=512 to C=256. The output feature map size of this module is [B, 256, 64, 64].

[0039] Upsampling Module 2 (M=2): Contains N=2 residual blocks, followed by an upsampling layer (step size 2), which changes the number of channels from C=256 to C=128. The output feature map size of this module is [B, 128, 128, 128].

[0040] Upsampling Module 3 (M=3): Contains N=2 residual blocks, followed by an upsampling layer (step size 2), which changes the number of channels from C=128 to C=128. The output feature map size of this module is [B, 128, 256, 256].

[0041] Output convolutional module: a 3x3 convolutional layer that maps the number of channels from C=128 back to 3 (RGB) and uses the Tanh activation function to normalize the pixel values ​​to the range [-1, 1].

[0042] E_vq, G_vq, and D are all set to the parameter frozen state after pre-training is completed.

[0043] Step 2, Model Architecture and Forward Propagation like Figure 1 As shown, the core model of this invention is a controllable inductive moment matching generator. (Implemented as a U-Net architecture), and multiple encoders.

[0044] 2.1 Multimodal encoder: An image encoder (preferably a VAE) is used to process x_source, x_target, and x_style to extract f_source, f_target, and f_style features (sizes such as [B, 257, 1024]).

[0045] A text encoder (preferably CLIP ViT-L / 14 Text Encoder) is used to process c_text and extract f_text_edit features (size such as [B, 77, 1024]).

[0046] 2.2 Fusion Denoising U-Net ( ): Encoder: The encoder consists of multiple stacked downsampling modules. Each downsampling module is responsible for halving the spatial resolution of the feature map (H -> H / 2) and increasing the number of channels (C -> 2C). Each downsampling module specifically consists of multiple residual blocks, one multimodal guided fusion module, and one 3x3 downsampling convolutional layer with a stride of 2. The input features first flow through N residual blocks for depth feature extraction, then are fed into the multimodal guided fusion module for guided injection, and finally the resolution is reduced by the downsampling convolutional layer. The output is then fed into the next module.

[0047] Bottleneck: Located in the deepest layer of U-Net, it consists of N residual blocks and 1 multimodal guided fusion module, and does not change the spatial resolution of the feature map.

[0048] Decoder: The decoder is symmetrical to the encoder and consists of M stacked upsampling modules. Each upsampling module is responsible for doubling the spatial resolution (H / 2 -> H). The feature map from the previous layer of U-Net first passes through an upsampling layer. Then it is concatenated with the skip connection feature map from the corresponding layer of the encoder. The concatenated feature map flows through N residual blocks and one multimodal guided fusion module, and is finally output to the next upsampling module.

[0049] Preferably, a Transformer architecture similar to DiT or U-Vit is used, operating in the latent space of the main VAE (e.g., Stable DiffusionVAE). This patent uses a downsampling factor of 8 for the main VAE. The latent variable z_t being processed has a size of [B, 4, 32, 32]. Each attention module in U-Net is replaced with the multimodal guided fusion module designed in this invention.

[0050] Within a U-Net module, the detailed data flow is as follows (in conjunction with...) Figure 2 and Figure 3 ): a) Module input: f_image is a "canvas" feature map from a convolutional layer or Transformer block on the U-Net (representing the features of z_t at that scale).

[0051] c_all_guides contains a set of guided features for f_source, f_target, f_text_edit, and f_style. f_source: the identity (ID) features of the source image; f_target: the pose / expression features of the target image; f_text_edit: the semantic features of the edited text; f_style: the style features of the styled image.

[0052] b) Guided feature calculation: Decoupled gestures - Identity Attention (QKV-Attention) (e.g.) Figure 2 ): Q_target = Linear_Q(f_target); K_source = Linear_K(f_source); V_source = Linear_V(f_source); out_qkv =Attention(Q=Q_target, K=K_source, V=V_source). This step calculates the "structure" guidance that integrates the source ID and target pose.

[0053] Text editing attention: Q_image = Linear_Q_text(f_image); K_text, V_text = Linear_KV_text(f_text_edit); out_text = CrossAttention(Q=Q_image, K=K_text, V=V_text).

[0054] Style attention: Q_image_style = Linear_Q_style(f_image); K_style, V_style = Linear_KV_style(f_style); out_style = CrossAttention(Q=Q_image, K=K_style, V=V_style).

[0055] c) Gated-AdaIN Fusion (e.g.) Figure 3 ): Gating: A learnable gating scalar g (which can be dynamically predicted from f_image using a small MLP network) is used to balance content and structure: f_content_structure = (g * out_text) + ((1 -g) * out_qkv) AdaIN injection: Calculate the space adaptive scaling parameters of out_style. and offset parameters The calculation formula is as follows: in This represents the activation function of the Sigmoid Linear Unit, specifically... ; and These represent two independent convolutional layer operations (including weights). and bias ), used to map style features to modulation parameter space.

[0056] Finally, the generated parameters are used to perform an affine transformation on the normalized content feature f_content_structure to obtain the final output feature f_final_guidance. The calculation of f_final_guidance is as follows: d) Module output: The final feature f_final_guidance, which incorporates all guidance, is added back to the original "canvas" features via residual connections: f_output = f_image + f_final_guidance. f_output is then fed into the next convolutional layer of the U-Net.

[0057] Step 3: Loss Calculation and Model Training This invention employs inductive moment matching for single-stage training, combined with VQ prior loss, such as... Figure 4 As shown.

[0058] 3.1 Inductive Moment Matching Core Loss (L_IMM): In each training iteration, real data x and noise ε are sampled from a batch of data with a batch size of B (e.g., B=64). Three time points s < r < t are randomly sampled.

[0059] a) Calculate the latent variable x_t at time t and the latent variable x_r at time r: x_t = x_r = DDIM(x_t, x, r, t) b) Calculate distribution P (prediction): using the current model And input all the guiding parameters C = {f_source, f_target, f_text_edit, f_style}, and perform a one-step prediction: y_s,t = (x_t, s, t, C) c) Calculate the distribution Q (objective): using a model with stopping gradients. - and input all the guiding C values ​​to perform a one-step prediction: y_s,r = -(x_r, s, r, C) d) Calculate the MMD loss: L_IMM is calculated as the maximum mean difference between batches y_s,t and y_s,r. A Laplace kernel is used. or RBF core To estimate the MMD loss, we have: L_IMM = E[k(y_s,t, y_s,t') + k(y_s,r, y_s,r') - 2*k(y_s,t, y_s,r)] 3.2 VQ Prior Loss (L_VQ_Prior): a) Input the y_s,t generated in step 3.1 into the main VAE decoder to obtain the pixel-level image x_student (size [B, 3, 256, 256]).

[0060] b) Feed x_student into the VQ-VAE encoder E_vq frozen in step 1.3 to extract features z_e ([B,256, 32, 32]).

[0061] c) Query the frozen VQ dictionary D ([8192, 256]) to obtain the quantized feature z_q ([B, 256, 32,32]).

[0062] d) Feed z_q into the frozen VQ-VAE decoder G_vq to obtain the reconstructed image x_recon([B, 3, 256,256]).

[0063] e) Calculate the perceptual loss: L_VQ_Prior = LPIPS(x_student, x_recon).

[0064] 3.3 Other auxiliary losses: a) L_ID: Calculate the identity similarity loss between x_student and x_source using a pre-trained ArcFace network. L_ID = 1 - CosSim(ArcFace(x_student), ArcFace(x_source)).

[0065] b) L_rec: Calculate the L1 pixel loss of x_student compared to the real data x in the non-edited region.

[0066] c) L_GAN: After more than 50% of the total number of training iterations, activate a discriminator D_gan and compute L_GAN to improve realism.

[0067] 3.4 Total Loss and Optimization: The total loss is: L_total = λ_imm*L_IMM + λ_vq*L_VQ_Prior + λ_id*L_ID + λ_rec*L_rec + λ_gan*L_GAN. Preferably, λ_imm=1.0, λ_vq=0.1, λ_id=0.5, λ_rec=1.0, λ_gan=0.1.

[0068] The gradient of L_total is calculated through backpropagation, and only the IMM generative model is updated. The parameters.

[0069] Step 4, Phased Model Reasoning This invention employs a Restart Sampling strategy for inference, such as... Figure 5 As shown.

[0070] 4.1 Stage 1: Semantic Structure Generation (t=T to t_cutoff) Start with pure noise z_T ([B, 4, 32, 32]) at t=T=1.0.

[0071] In the U-Net model In this study, only QKV-Attention and text attention guidance are activated (style guidance is turned off) to determine the macroscopic structure, identity, and semantic content of a face.

[0072] Perform N denoising steps (e.g., N=2 steps), the t sequence is [1.0, 0.75, 0.5], and the latent variable z_cutoff (i.e. z_0.5) is obtained.

[0073] 4.2 Stage 2: Noise Injection (t=t_cutoff) Get z_0.5 and re-inject noise into it to degenerate it to t_restart=0.75, thus obtaining z_restart_new.

[0074] 4.3 Phase 3: Refining Style Details (t=t_restart to t=0) Denoising begins at z_restart_new (t=0.75).

[0075] In the U-Net model In this process, all guidance is activated simultaneously: QKV-Attention, text attention, and Gated-AdaIN style fusion guidance.

[0076] Perform M-step denoising (e.g., M=3 steps), the t sequence is [0.75, 0.5, 0.25, 0.0], and the final, clean latent variable z_0 is obtained.

[0077] 4.4 Final Output: z_0 ([B, 4, 32, 32]) is fed into the main VAE decoder to obtain the final, high-fidelity, stylized face-edited image ([B, 3, 256, 256]).

[0078] The training and testing of the method of this invention can be implemented on typical deep learning computing platforms. For example, a deep learning framework based on Python 3.10 and PyTorch 2.0 can be used, with an NVIDIA GeForce RTX 3090 graphics processor configured under a Linux operating system, and matching CUDA and cuDNN versions installed for accelerated computation. In the specific implementation, the experimental environment can be set up by referring to the configuration file of the inductive moment matching framework: training data is organized in the form of image folders, using training.dataset.ImageFolderDataset as the dataset interface, and setting whether to use class labels and whether to enable horizontal flip enhancement (i.e., setting use_labels=true, xflip=false in the FFHQ task) according to the task requirements, and enabling caching to improve data loading efficiency.

[0079] In terms of network structure, the IMMPrecond preconditioned network can be used as the main body of the generative model. For small and medium-sized images, a SongUNet-type U-Net structure is selected, with appropriate channel numbers and multi-scale channel expansion ratios configured (e.g., model_channels=128, channel_mult=[2,2,2]). Simultaneously, an fm noise scheduling strategy and a simple EDM preconditioned form are used in the noise modeling part. For high-resolution image tasks (such as 256×256 ImageNet), the DiT_XL_2 visual Transformer structure can be used, combined with fp16 mixed-precision training to reduce memory usage.

[0080] For optimization strategies, adaptive moment estimation optimizers such as torch.optim.RAdam or torch.optim.AdamW can be used, with the initial learning rate set to 1×10⁻⁶. 4 The momentum parameters are set to β1=0.9 and β2=0.999. Weight decay (e.g., weight_decay=0 or a non-zero value less than 0.01) is added during large-scale training when necessary. The learning rate scheduling strategy can employ multi-step or segmented decay, combined with an exponential moving average (EMA) strategy to smooth the model weights (e.g., setting ema_beta=0.9999). During training, the batch size can be set to tens to thousands depending on the dataset size, and divided according to the number of GPUs. The total number of training iterations can be set according to the task difficulty, with each tick processing a fixed number of images. Model snapshots and states can be saved periodically during training (e.g., ckpt_ticks=50, snapshot_ticks and eval_ticks are set according to the task), and metrics such as FID are used to evaluate the generation effect. During the inference and sampling phases, pushforward-based sampling operators can be used to uniformly or adaptively discretize the time interval, and combined with a classification-guided scale (e.g., cfg_scale=1.5) to improve sampling efficiency while ensuring generation quality.

[0081] Corresponding to the aforementioned embodiment of a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance, the present invention also provides an embodiment of a VQ prior resampling face editing device based on inductive moment matching and decoupling guidance.

[0082] See Figure 6This invention provides a VQ prior resampling face editing device based on inductive moment matching and decoupling guidance, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance as described in the above embodiments.

[0083] The embodiment of the VQ prior resampling face editing device provided by this invention, based on inductive moment matching and decoupling guidance, can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an VQ prior resampling face editing device based on inductive moment matching and decoupling guidance provided by the present invention. (Except for...) Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0084] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0085] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0086] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance as described in the above embodiments.

[0087] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0088] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned VQ prior resampling face editing method based on inductive moment matching and decoupling guidance.

[0089] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0090] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A VQ prior resampling face editing method based on inductive moment matching and decoupling guidance, characterized in that, include The multimodal dataset is preprocessed, and the VQ-VAE model is pre-trained. Multimodal encoders are used to extract features from multimodal data, which are then input into a U-Net network whose attention module is replaced by a decoupled pose-identity attention and gating-AdaIN style fusion computation. The VQ prior loss obtained from the VQ-VAE model is combined with the inductive moment matching core loss for single-stage training. The trained network model is used for staged model inference: in the high-step stage, only content and structure guidance are used to generate semantic skeletons; in the intermediate steps, noise is re-injected; and in the low-step stage, style guidance is activated to render details and style; finally, a high-fidelity face image is output.

2. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, In the preprocessing of the multimodal dataset, the dataset includes: source images, target images, edited text, and style images. The dataset is normalized in terms of image size and pixel value, and different encoders are selected for feature extraction.

3. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, The pre-training of the VQ-VAE model specifically includes: The Vector Quantization Autoencoder (VQ-VAE) model consists of an encoder, a decoder, and a learnable VQ dictionary. A pre-training dataset is constructed using large-scale, high-resolution real face images containing local facial components and texture details. Training and convergence: The vector quantization model is trained on the pre-training dataset. By minimizing the reconstruction loss and commitment loss, the VQ dictionary is converged into a codebook containing K discrete quantization vectors, where each quantization vector represents a local component feature of a real face. Parameter freezing: After the VQ model training converges, all parameters of the encoder, decoder and VQ dictionary D are set to a frozen state, as a fixed face prior knowledge module, which is used to calculate the VQ prior loss in step three.

4. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, The decoupled gesture-identity attention and gating-AdaIN style fusion computation includes: Decoupled Pose-Identity Attention: The pose / expression features of the target image f_target are used as the query Q, and the identity details of the source image f_source are used as the key K and value V; calculate out_qkv = Attention(Q=f_target, K=f_source, V=f_source) to obtain the structural features that fuse the identity of the source image and the pose of the target image. The gating system incorporates AdaIN's style, specifically: First, the text editing guidance feature out_text and the style guidance feature out_style are calculated. Then, a learnable gating weight g is used to balance content and structure; f_content_structure = (g * out_text) + ((1 -g) * out_qkv); Use cross-modal adaptive instance normalization to inject style into the generated image; f_final_guidance = AdaIN(content=f_content_structure, style=out_style); Finally, f_final_guidance is added back to the image features of U-Net.

5. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, The core loss for inductive moment matching specifically includes: Three random time points s, r, and t are sampled, satisfying s < r < t, and real data x and noise ε are sampled from the dataset; Calculate the latent variable x_t at time t and the latent variable x_r at time r; Calculate the predicted distribution P: y_s,t = (x_t, s, t, All_Guides), where The model with the current training parameters; Calculate the target distribution Q: y_s,r = -(x_r, s, r, All_Guides), where - Model parameters for stopping gradients; Calculate L_IMM: L_IMM = (average similarity within P + average similarity within Q) - 2 * (average similarity between P and Q), where the similarity within P is k(y_s,t, y_s,t'), which means calculating the similarity between two samples taken from the predicted distribution P; the similarity within Q is k(y_s,r, y_s,r'), which means calculating the similarity between two samples taken from the target distribution Q; and the similarity between P and Q is k(y_s,t, y_s,r), which means calculating the similarity between one sample taken from P and one sample taken from Q.

6. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, The VQ prior loss specifically includes: Obtain the student model output: match the inductive moment generator. Latent variables generated in the current training iteration _student is fed into the main decoder VAE Decoder, which decodes it into a pixel-level face image x_student; Feature extraction: Input the x_student into the VQ encoder E_vq pre-trained and frozen in step 2 to extract its continuous intermediate feature maps z_e; Feature quantization: In the VQ dictionary, for each feature vector in the z_e feature map, the nearest neighbor query is used to find and replace it with the most matching discrete quantization vector in the dictionary to obtain the quantized feature map z_q; Quantization Reconstruction: The quantized feature map z_q is input into the frozen VQ decoder G_vq to reconstruct an image x_recon that is entirely composed of real face component features; Loss calculation: The perceptual loss between the image generated by the x_student student model and the image composed of the x_recon discrete vector dictionary is calculated, and the VQ prior loss L_VQ_Prior is used for backpropagation to update the generator. The parameters are used to train the generated image features to actively approximate the feature distribution of a real human face.

7. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, The total loss in single-stage training using the inductive moment matching core loss combined with the VQ prior loss includes: a weighted fusion of the VQ prior loss, the inductive moment matching core loss, the identity preservation loss calculated by the pre-trained face recognition network, the reconstruction loss to ensure the consistency of non-edited regions, and the adversarial loss introduced in the later stages of training.

8. The VQ prior resampling face editing method based on inductive moment matching and decoupling guidance according to claim 1, characterized in that, The specific method of using only content and structure to guide the generation of semantic skeletons in the high-step stage is as follows: In the high-step denoising stage, the U-Net network only runs under decoupled pose-identity attention guidance and text guidance, generating latent variables z_cutoff containing the correct identity, pose, and content; Noise is re-injected into z_cutoff to obtain z_restart_new; denoising begins from z_restart_new until t=0. The network model runs together under all bootstraps to render style and details on the established structure and perform style transfer.

9. A face editing device based on inductive moment matching and decoupling guidance using VQ prior resampling, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a VQ prior resampling face editing method based on inductive moment matching and decoupling guidance as described in any one of claims 1-7.

Images