An Unconditional Face Image Generation Method and System Based on End-to-End Self-Distillation

Abstract

This invention relates to an unconditional face image generation method and system based on end-to-end self-distillation. The method includes: acquiring a face image; inputting it into a variational autoencoder and a face analysis network to obtain latent features and face region analysis results; constructing an end-to-end self-distillation unconditional generation network, adding current and past time-step noise to the latent features to obtain temporary strong and weak random Gaussian noise; constructing face region noise based on the face region analysis results, and combining it with the existing noise to obtain strong and weak noise; inputting the weak noise into a target SiT to extract deep representations, and inputting the strong noise into an online SiT to extract shallow representations, optimizing parameters through representation alignment loss; performing high-order face structure clustering on the deep representations to obtain a hypergraph, and calculating the Laplacian matrix as the attention structure bias for the online SiT; and using the strong noise as input for prediction noise in the online SiT, optimizing parameters through diffusion loss. This invention can efficiently and with high quality generate diverse face images, improving structural consistency and detail representation.

Description

Technical Field

[0001] This invention belongs to the field of image processing and generation technology, specifically relating to an unconditional face image generation method and system based on end-to-end self-distillation. Background Technology

[0002] In recent years, face image generation has become an important research direction in the field of computer vision. Unconditional face image generation aims to synthesize high-quality and diverse facial images without external conditions such as text or labels. This task has important applications in data augmentation, digital human creation, and facial editing.

[0003] Diffusion models have become the mainstream paradigm in image generation because they can synthesize images with higher fidelity. Lee et al., in their paper "Multi-architecture multi-expert diffusion models" (Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(12):13427-13436), combined convolution and self-attention to utilize frequency characteristics, processing low-frequency and high-frequency components separately. Huang et al., in their paper "Tfmq-dm: Temporal feature maintenance quantization for diffusion models" (Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2024: 7362-7371), introduced a temporal information block independent of the input data. However, these methods are still limited by the U-Net-based diffusion backbone structure, whose limited receptive field hinders their ability to capture global semantics.

[0004] In contrast, the recent Diffusion Transformer (DiT), based on the Vision Transformer (ViT), surpasses the traditional U-Net backbone by controlling the size of latent markers and flexibly adjusting latent space compression, achieving better modeling of global context and stronger expressiveness. The SiT proposed by Ma et al. in their paper "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers" (European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 23-40) has achieved unprecedented image generation quality, driving a paradigm shift in unconditional face generation architectures. However, traditional DiT still has limitations in detail and structural reconstruction, especially in capturing complex details and subtle variations in facial generation. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an unconditional face image generation method and system based on end-to-end self-distillation, so as to generate diverse face images more efficiently and with higher quality.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: an unconditional face image generation method based on end-to-end self-distillation, comprising the following steps:

[0007] 1) Obtain face images from the dataset, input the face images into a variational autoencoder and a pre-trained face analysis network to obtain latent feature representations and face region analysis results;

[0008] 2) An end-to-end self-distilling unconditional generative network model is constructed to generate face images. The unconditional generative network model adds current time-step noise and previous time-step noise to the input latent feature representation to obtain temporary strong random Gaussian noise and temporary weak random Gaussian noise. Simultaneously, for the input face region analysis results, face region noise is constructed by randomly selecting and combining it with the temporary strong random Gaussian noise and temporary weak random Gaussian noise to obtain strong noise and weak noise. The weak noise is input into the target SiT (Search Engine Technology) and deep target representation is extracted from it. The strong noise is input into the online SiT and shallow online representation is extracted from it. The difference between shallow representations and deep target representations is measured by representation alignment loss, and the gradient is backpropagated to the variational autoencoder to optimize the parameters of online SiT and variational autoencoder. For the deep target representation, face structure clustering is performed to obtain a face structure hypermap, and its corresponding Laplacian matrix is ​​calculated and used as the face attention structure bias term of online SiT. Online SiT takes strong noise as input, predicts noise, and uses diffusion loss to measure the difference between predicted noise and strong noise so that the model removes noise and generates images. Its gradient is backpropagated only to online SiT and not to variational autoencoder, and is used to optimize the parameters of online SiT.

[0009] 3) By performing inverse diffusion sampling on the trained online SiT model, face images can be generated without introducing any additional conditions.

[0010] Furthermore, the implementation method of step 1) is as follows:

[0011] A1) Obtain face images from the CelebA-HQ and FFHQ face datasets;

[0012] A2) The acquired facial image The input is fed into the variational autoencoder (VAE) and compared with the increment factor of the current time step. Increment factor of previous time steps Multiply to obtain the latent feature representation of the face image at the current time step. Previous time step latent feature representation In addition, facial images The data is fed into a pre-trained face analysis network to obtain face region analysis results, including eyes, nose, mouth, and eyebrows. .

[0013] Further, in step 2), the implementation method of the end-to-end self-distillation unconditional generative network model is as follows:

[0014] The end-to-end self-distillation unconditional generative network model adopts an end-to-end self-distillation representation alignment framework, including online SiT, target SiT, facial region sensitive noise injection strategy, and hypergraph guided attention mechanism.

[0015] The end-to-end self-distillation characterization alignment framework has two branches, each consisting of an online SiT... and target SiT Implementation; the online SiT and target SiT They all use the same structure, consisting of multiple Transformer blocks and a projection head; among them, the online SiT As a student network, the target SiT Then, as a teacher network, online SiT and target SiT Shared architecture, targeting SiT Through online SiT Update the historical parameters;

[0016] In the model data processing, a facial region-sensitive noise injection strategy is used to replace the random noise used in the standard diffusion model as the denoising target during training. The facial region-sensitive noise utilizes a pre-trained face analysis network to segment the face image into multiple semantically meaningful regions, including eyes, nose, mouth, and eyebrows, and injects them as a kind of facial region noise into temporary strong random Gaussian noise and temporary weak random Gaussian noise to obtain strong noise and weak noise. Semantic face segmentation is used to generate more targeted noise, enhancing the model's focus on key facial regions.

[0017] During the model iterative training process, the 20th layer features of the target SiT are extracted through the hypergraph-guided attention mechanism to obtain the deep representation of the target. And compute the deep representation of the target through a hypergraph-guided attention mechanism. The corresponding hypergraph Laplacian matrix The Laplacian matrix of the hypergraph is used as a bias term for the face attention structure and input into the attention mechanism of online SiT to improve the relationship modeling between facial regions and improve the generation quality and structural consistency of the model.

[0018] During gradient backpropagation, weak noise is input into the target SiT, and a deep target representation is extracted from its 20th layer. Strong noise is input into the online SiT, and a shallow online representation is extracted from its 8th layer. The difference between the shallow online representation and the deep target representation is measured by the representation alignment loss. The gradient is backpropagated to the variational autoencoder to optimize the parameters of the online SiT and the variational autoencoder. The online SiT takes strong noise as input, predicts noise, and uses a diffusion model loss function to measure the difference between the predicted noise and the strong noise. Its gradient backpropagation is only propagated to the SiT and not to the variational autoencoder, and is used to optimize the parameters of the online SiT.

[0019] Furthermore, the implementation method of the facial region sensitive noise injection strategy is as follows:

[0020] B1) Replace the random noise used in the standard diffusion model with noise that focuses more on the fine-grained structural regions of the face, as the denoising target during training; given an input facial image First, a face parser is applied to segment the image, dividing it into multiple semantically meaningful regions. Background regions are then uniformly removed to reduce the impact of irrelevant noise. Finally, a facial sensitive region is randomly selected from the parsing results. And inject random noise only within the selected area. ;

[0021] B2) Fusion of random noise and original noise at different time steps; specifically: merging region-sensitive noise... Weighted and added to the weak noise at the current time step weak noise at previous time steps This generates two enhanced noise representations:

[0022]

[0023]

[0024] in, This indicates strong noise that is increasing at the current time step. This represents the weak noise that was amplified in the previous time step. Indicates the current time step The noise scaling factor at each time step controls the intensity of the noise generated at that current time step. Indicates the steps taken in the past period of time Noise scaling factor at time step Indicates the current moment. Indicates earlier than the current time At that moment in time step It is noise limited to a selected facial area.

[0025] Furthermore, the implementation method of the hypergraph-guided attention mechanism is as follows:

[0026] C1) First, the target encoder is processed. Extracted facial representation Clustering is performed to group spatially and semantically similar regions together, where each cluster represents a set of visually similar facial regions; the clustering process is defined as follows:

[0027]

[0028] in, This represents the cluster assignment for each feature. It represents the number of clusters; KMeans() represents the KMeans algorithm, which assigns similar regions to the same cluster by minimizing the Euclidean distance within the cluster; each cluster is regarded as a hyperedge in the construction of a hypergraph, thereby capturing the higher-order connectivity and contextual relationships between different parts of the face;

[0029] C2) Construct a hyperedge adjacency matrix based on clustering results. Each element Indicates the region Clustering The relationship between them; assuming the batch size is Each image has There are 10 regions, totaling 100 areas. The clusters are constructed as follows:

[0030]

[0031] Based on hyperedge adjacency matrix Calculate the corresponding hypergraph Laplacian matrix. The higher-order structural dependencies between encoded regions are as follows:

[0032]

[0033] in, It is a node degree matrix, representing the number of connections in each region. It is the hypermarginality matrix, representing the size of each cluster; through the Laplace matrix... Describe the structural interconnections of facial regions in the hypergraph and introduce them as structural biases into an attention mechanism; It is directly used as an additive bias and embedded into attention computation. In, that is:

[0034]

[0035] in, A learnable or preset scaling factor. Broadcast to all attention heads.

[0036] Further, in step C2), attention calculation The implementation method is as follows:

[0037] In multi-head self-attention mechanisms, given an input feature map The query is obtained through linear projection calculation. ,key ,value matrix:

[0038]

[0039] in, , , The learnable weight matrix; the attention score matrix. Calculated using the scaled dot product attention mechanism:

[0040]

[0041] in, Let be the dimension of the key vector; the final attention output is obtained by... Apply the softmax function and adjust the value matrix. Weighted average yields:

[0042]

[0043] Each final cluster groups semantically and spatially coherent regions into a group, and these groups are used to construct the hypergraph adjacency matrix. ;based on Calculate the Laplacian matrix of the hypergraph To capture the high-order structural dependencies between facial components, the Laplacian matrix of the hypergraph is then injected as an additive structural bias into the attention mechanism. The final structure-aware self-attention representation is as follows:

[0044]

[0045] in, It is a learnable or preset scaling factor used to control the influence of structural priors.

[0046] Furthermore, in step 2), the methods for constructing the diffusion loss and the characterization alignment loss are as follows:

[0047] D1) Construct diffusion loss based on SiT, and learn the velocity field of the probabilistic flow ordinary differential equation PF-ODE through the flow-based model SiT. The forward process of SiT is defined as follows:

[0048]

[0049] in, It is the raw data. It's Gaussian noise. Indicates at time step Data at any given time and It's about time. Monotonically decreasing and increasing functions; PF ODE is given by the following formula:

[0050]

[0051] in, Representing data points Over time The rate of change of ; in order to learn the velocity field, minimize the following diffusion loss :

[0052]

[0053] in, This represents the parameters of the online diffusion model. for The time derivative is used to control the data scaling rate. for The derivative of the value is used to control the rate of change of noise intensity, and the desired operation. Indicates the original data ,noise and time step Sampling is performed and the average loss is calculated; the flow process is aligned with the data distribution by optimizing the velocity field;

[0054] D2) In characterizing alignment loss, online SiT and target SiT Different levels of noise were injected: Receive lower noise ,and Receives high noise ; Indicates the current diffusion time step. This represents the offset of past time steps; after noise injection, intermediate feature representations are extracted from specific layers of each model; specifically: from The Layer obtained, from The Layer obtained, and This represents the corresponding internal transformer block or attention layer selected for feature supervision; its expression is:

[0055]

[0056]

[0057] in, It's the batch size. It is the number of spatial markers for each image. It is a feature dimension; to facilitate better representation alignment, a learnable MLP projection head is introduced. Representing the intermediate features of the online model Mapped to To achieve optimal knowledge transfer, the model's features are aligned across different time steps and layers, minimizing the distance between student and teacher patch-level outputs; this represents the alignment loss. Represented as:

[0058]

[0059] in, Indicates the first One patch, This represents the parameters of the online diffusion model. The parameters representing VAE, This represents the parameters of the projection head and the expected operation. Represents all possible time steps High-resolution features on and time step Weighted averaging is performed; backpropagation is used to facilitate end-to-end training and jointly optimize VAE and diffusion transformer features. To assist in VAE parameter optimization; to prevent potential spatial structure degradation during backpropagation of the diffusion model, a gradient stopping operation is introduced to ensure... Used only for updating online diffusion models parameters .

[0060] Furthermore, in step 2), to enhance the reconstruction performance of the VAE, three regularization losses are combined, including reconstruction loss. GAN loss and KL divergence loss This results in the total regularization loss of the VAE. ;

[0061] Ultimately, the end-to-end training process is achieved through the following total loss. To optimize the parameters of the diffusion converter:

[0062]

[0063] in, , and They represent the online diffusion model, respectively. VAE and learnable projector The parameters, and Hyperparameters are used to control the relative importance of the corresponding loss terms.

[0064] The present invention also provides an unconditional face image generation system based on end-to-end self-distillation, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method.

[0065] Compared with existing technologies, the present invention has the following advantages: The present invention provides an unconditional face image generation method and system based on end-to-end self-distillation. This method constructs an end-to-end self-distillation unconditional generative network model. The end-to-end self-distillation representation alignment framework eliminates the need for an external teacher model by aligning shallow and deep SiT representations, achieving efficient end-to-end training while accelerating the generation process and improving the quality of latent representations. The facial region-sensitive noise injection strategy combines facial-sensitive noise with the original noise, encouraging the model to capture fine-grained details and maintain structural consistency between regions. The hypergraph-guided attention mechanism clusters target model features into a hypergraph and integrates it as a structural bias into the attention mechanism of the online model, thereby enhancing the model's ability to capture complex topological relationships between facial components. Simultaneously, this method accelerates the training of the diffusion Transformer and improves generation quality through robust representation by jointly optimizing the representation alignment loss and diffusion loss. This method and system can significantly improve the accuracy and reliability of unconditional face generation tasks in the case of large-scale unlabeled face data. It can not only effectively solve the problem of poor unconditional face image generation results, but also be fine-tuned and trained to be applicable to conditional face image generation tasks. Attached Figure Description

[0066] Figure 1 This is a flowchart of the unconditional face image generation method based on end-to-end self-distillation provided in the embodiments of the present invention;

[0067] Figure 2 This is an architecture diagram of the unconditional generation network model of end-self-distillation in an embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0069] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0070] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0071] like Figure 1 As shown, this embodiment provides an unconditional face image generation method based on end-to-end self-distillation, including the following steps:

[0072] 1) Obtain face images from the dataset, input the face images into a variational autoencoder and a pre-trained face analysis network to obtain latent feature representations and face region analysis results;

[0073] 2) An end-to-end self-distilling unconditional generative network model is constructed to generate face images. The unconditional generative network model adds current time-step noise and previous time-step noise to the input latent feature representation to obtain temporary strong random Gaussian noise and temporary weak random Gaussian noise. Simultaneously, for the input face region analysis results, one of these is randomly selected to construct face region noise, which is then combined with the temporary strong random Gaussian noise and temporary weak random Gaussian noise to obtain strong noise and weak noise. The weak noise is input into the target SiT, and the target deep representation is extracted from its 20th layer. The strong noise is input into the online SiT, and the online shallow representation is extracted from its 8th layer. The difference between the online shallow representation and the target deep representation is measured by the representation alignment loss, and the gradient is backpropagated to the variational autoencoder to optimize the parameters of the online SiT and the variational autoencoder. For the target deep representation, face structure clustering is performed to obtain the face structure hypermap, and its corresponding Laplacian matrix is ​​calculated and used as the face attention structure bias term of the online SiT. The online SiT takes strong noise as input, predicts noise, and uses diffusion loss to measure the difference between the predicted noise and the strong noise so that the model can effectively remove noise to generate images. Its gradient is backpropagated only to the online SiT and not to the variational autoencoder, and is used to optimize the parameters of the online SiT.

[0074] 3) By performing inverse diffusion sampling on the trained online SiT model, face images can be generated without introducing any additional conditions.

[0075] In this embodiment, the specific implementation method of step 1) is as follows.

[0076] A1) Obtain face images from the CelebA-HQ and FFHQ face datasets;

[0077] A2) The acquired facial image The input is fed into the variational autoencoder (VAE) and compared with the increment factor of the current time step. Increment factor of previous time steps Multiply to obtain the latent feature representation of the face image at the current time step. Previous time step latent feature representation In addition, facial images The data is fed into a pre-trained face analysis network to obtain facial region analysis results, including eyes, nose, mouth, and eyebrows. .

[0078] Figure 2 This is an architecture diagram of the end-to-end self-distilling unconditional generator network model in this embodiment. In step 2), the implementation method of the end-to-end self-distilling unconditional generator network model is as follows.

[0079] The end-to-end self-distillation unconditional generative network model adopts an end-to-end self-distillation representation alignment framework, including online SiT, target SiT, facial region sensitive noise injection strategy, and hypergraph guided attention mechanism.

[0080] The end-to-end self-distillation characterization alignment framework has two branches, each consisting of an online SiT... and target SiT Implementation; the online SiT and target SiT They all use the same structure, consisting of multiple Transformer blocks and a projection head; among them, the online SiT As a student network, the target SiT Then, as a teacher network, online SiT and target SiT Shared architecture, targeting SiT Through online SiT The historical parameters are updated.

[0081] In the model data processing, a facial region-sensitive noise injection strategy is used to replace the random noise used in the standard diffusion model as the denoising target during training. The facial region-sensitive noise utilizes a pre-trained face analysis network to segment the face image into multiple semantically meaningful regions, such as eyes, nose, mouth, and eyebrows, and injects them as a kind of facial region noise into temporary strong random Gaussian noise and temporary weak random Gaussian noise to obtain strong noise and weak noise. Semantic face segmentation is used to generate more targeted noise, enhancing the model's focus on key facial regions.

[0082] During the model iterative training process, the 20th layer features of the target SiT are extracted through the hypergraph-guided attention mechanism to obtain the deep representation of the target. And compute the deep representation of the target through a hypergraph-guided attention mechanism. The corresponding hypergraph Laplacian matrix By using the Laplacian matrix of the hypergraph as a bias term for the face attention structure in the online SiT attention mechanism, the relationship modeling between facial regions is improved, thereby enhancing the generation quality and structural consistency of the model.

[0083] During gradient backpropagation, weak noise is input into the target SiT, and a deep representation of the target is extracted from its 20th layer. Strong noise is input into the online SiT, and a shallow representation of the online SiT is extracted from its 8th layer. The difference between the shallow online representation and the deep target representation is measured by the representation alignment loss. The gradient is backpropagated to the variational autoencoder to optimize the parameters of the online SiT and the variational autoencoder. The online SiT takes strong noise as input, predicts noise, and uses a diffusion model loss function to measure the difference between the predicted noise and the strong noise, so that the model can effectively remove noise to generate images. Its gradient backpropagation is only propagated to the SiT and not to the variational autoencoder, and is used to optimize the parameters of the online SiT.

[0084] In this embodiment, the specific implementation method of the facial region sensitive noise injection strategy is as follows.

[0085] B1) Replace the random noise used in the standard diffusion model with noise that focuses more on the fine-grained structural regions of the face, as the denoising target during training; given an input facial image First, a face parser is applied to segment the image, dividing it into multiple semantically meaningful regions; then, background regions are uniformly removed to reduce the influence of irrelevant noise and prevent unnecessary learning interference; such as Figure 2 As shown in the leftmost region, a facial sensitive area was randomly selected from the analysis results. And inject random noise only within the selected area. This local noise sampling strategy ensures that noise is applied only to components that are important to the facial structure, thereby improving the model's ability to reconstruct fine-grained details in these areas.

[0086] B2) To effectively fuse facial-sensitive noise, random noise and the original noise are fused at different time steps; specifically: regional-sensitive noise is fused... Weighted and added to the weak noise at the current time step weak noise at previous time steps This generates two enhanced noise representations:

[0087]

[0088]

[0089] in, Indicates the current time step The noise scaling factor at each time step controls the intensity of the noise generated at that current time step. Indicates the steps taken in the past period of time Noise scaling factor at time step Indicates the current moment. Indicates earlier than the current time At that moment in time step It is noise limited to a selected facial area.

[0090] In this embodiment, the specific implementation method of the hypergraph-guided attention mechanism is as follows.

[0091] C1) In the hypergraph-guided attention mechanism, the first step is to focus on the target encoder. Extracted facial representation Clustering is performed, which allows the model to group spatially and semantically similar regions together, where each cluster represents a set of visually similar facial regions; the clustering process is defined as follows:

[0092]

[0093] in, This represents the cluster assignment for each feature. is the number of clusters; KMeans() represents the KMeans algorithm, which assigns similar regions to the same cluster by minimizing the Euclidean distance within the cluster; each cluster is considered as a hyperedge in the construction of a hypergraph, thereby capturing the higher-order connectivity and contextual relationships between different parts of the face.

[0094] C2) Based on the clustering results, construct a hyperedge adjacency matrix. Each element Indicates the region Clustering The relationship between them; assuming the batch size is Each image has There are 10 regions, totaling 100 areas. The clusters are constructed as follows:

[0095]

[0096] Based on hyperedge adjacency matrix Calculate the corresponding hypergraph Laplacian matrix. The higher-order structural dependencies between encoded regions are as follows:

[0097]

[0098] in, It is a node degree matrix, representing the number of connections in each region. It is the hypermarginality matrix, representing the size of each cluster; through the Laplace matrix... Describe the structural interconnections of facial regions in the hypergraph and introduce them as structural biases into an attention mechanism; It is directly used as an additive bias and embedded into attention computation. In, that is:

[0099]

[0100] in, A learnable or preset scaling factor. Broadcasting is performed among all attention heads. This structural bias ensures... It not only focuses on local content features but also integrates global and topology-aware facial region dependencies. In practice, this hypergraph-guided attention mechanism is embedded into online SiT. In multiple attention layers.

[0101] In this embodiment, in step C2), attention calculation The implementation method is as follows.

[0102] In standard multi-head self-attention mechanisms, given an input feature map... The query is obtained through linear projection calculation. ,key ,value matrix:

[0103]

[0104] in, , , The learnable weight matrix. The attention score matrix. Calculated using the scaled dot product attention mechanism:

[0105]

[0106] in, Let be the dimension of the key vector. The final attention output is obtained by... Apply the softmax function and adjust the value matrix. Weighted average yields:

[0107]

[0108] Each final cluster groups semantically and spatially coherent regions into a group, and these groups are used to construct the hypergraph adjacency matrix. ;based on Calculate the Laplacian matrix of the hypergraph To capture the high-order structural dependencies between facial components, the Laplacian matrix of the hypergraph is then injected into the attention mechanism as an additive structural bias. The final structure-aware self-attention representation is as follows:

[0109]

[0110] in, It is a learnable or preset scaling factor used to control the influence of structural priors. This form not only allows the attention mechanism to focus on local content features, but also integrates globally and topologically consistent face regions, thereby significantly improving the structural consistency and perceptual quality of the generated results.

[0111] In this embodiment, in step 2), the methods for constructing the diffusion loss and the characterization alignment loss are as follows.

[0112] D1) Construct diffusion loss based on SiT, and learn the velocity field of the probabilistic flow ordinary differential equation (PF-ODE) using the flow-based model SiT. This equation allows the model to sample from the data distribution as the data flows over time; the forward process of SiT is defined as:

[0113]

[0114] in, It is the raw data. It's Gaussian noise. Indicates at time step Data at any given time and It's about time. Monotonically decreasing and increasing functions; PF ODE is given by the following formula:

[0115]

[0116] in, Representing data points Over time The rate of change (velocity); this equation describes the evolution of data over time, ultimately leading to the generation of the data distribution; to learn this velocity field, the model minimizes the following diffusion loss. :

[0117]

[0118] in, This represents the parameters of the online diffusion model. for The time derivative is used to control the data scaling rate. for The derivative of the value is used to control the rate of change of noise intensity, and the desired operation. Indicates the original data ,noise and time step Sampling is performed and the average loss is calculated; the flow process is aligned with the data distribution by optimizing the velocity field.

[0119] D2) In characterizing alignment loss, online SiT and target SiT Different levels of noise were injected: Receive lower noise ,and Receives high noise This design maintains the relative stability of the target representation, provides reliable supervision, and prevents crashes; here, Indicates the current diffusion time step. Represents the offset of the past time step. This ensures the target model operates under micro-denoising conditions; after noise injection, intermediate feature representations are extracted from specific layers of each model; specifically, from The Layer obtained, from The Layer obtained, and This represents the corresponding internal transformer block or attention layer selected for feature supervision; its expression is:

[0120]

[0121]

[0122] in, It's the batch size. It is the number of spatial markers (e.g., image patches or regions) for each image. It is the feature dimension; to facilitate better representation alignment, a learnable MLP projection head is introduced. Representing the intermediate features of the online model Mapped to To achieve optimal knowledge transfer, the model's features are aligned across different time steps and layers, minimizing the distance between student and teacher patch-level outputs; this method achieves feature self-alignment without relying on external teacher model representations. The loss is represented as:

[0123]

[0124] in, Indicates the first One patch, This represents the parameters of the online diffusion model. The parameters representing VAE, This represents the parameters of the projection head and the expected operation. Represents all possible time steps High-resolution features on and time step Weighted averaging is performed; backpropagation is used to further facilitate end-to-end training and jointly optimize VAE and diffusion transformer features. To assist in VAE parameter optimization; and to prevent potential spatial structure degradation during backpropagation of the diffusion model, a gradient stopping operation is introduced to ensure... Used only for updating online diffusion models parameters .

[0125] Finally, to further enhance the reconstruction performance of VAE, this method combines three regularization losses, including: reconstruction loss. GAN loss and KL divergence loss This results in the total regularization loss of the VAE. .

[0126] Ultimately, the complete end-to-end training process is achieved through the following total loss. To optimize the parameters of the diffusion converter:

[0127]

[0128] in, , and They represent the online diffusion model, respectively. VAE and learnable projector Parameters, scalar and Hyperparameters are used to control the relative importance of the corresponding loss terms.

[0129] In this embodiment, the results of the proposed method of this invention and other image generation methods on the unconditional face image generation task are compared. Tables 1 and 2 show the comparison of the results of the proposed method of this invention with other image generation methods on two mainstream datasets for face generation tasks: the CelebAMask-HQ dataset and the FFHQ dataset. The comparison shows that the performance of the proposed method of this invention is superior to other existing methods.

[0130] Table 1

[0131]

[0132] Table 2

[0133]

[0134] This embodiment also provides an unconditional face image generation system based on end-to-end self-distillation, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for generating unconditional face images based on end-to-end self-distillation, characterized in that, Includes the following steps: 1) Obtain face images from the dataset, input the face images into a variational autoencoder and a pre-trained face analysis network to obtain latent feature representations and face region analysis results; 2) Construct an end-to-end self-distilling unconditional generative network model to generate face images; the unconditional generative network model adds current time step noise and previous time step noise to the input latent feature representation to obtain temporary strong random Gaussian noise and temporary weak random Gaussian noise respectively; at the same time, for the input face region analysis results, face region noise is constructed by randomly selecting and combining it with temporary strong random Gaussian noise and temporary weak random Gaussian noise to obtain strong noise and weak noise; the weak noise is input into the target SiT and the target deep representation is extracted from it, the strong noise is input into the online SiT and the online shallow representation is extracted from it, the difference between the online shallow representation and the target deep representation is measured by the representation alignment loss, and the gradient backpropagation is used to optimize the parameters of the online SiT and the variational autoencoder; For the deep representation of the target, face structure clustering is performed to obtain the face structure hypermap, and its corresponding Laplacian matrix is ​​calculated and used as the face attention structure bias term of the online SiT. The online SiT takes strong noise as input, predicts noise, and uses diffusion loss to measure the difference between the predicted noise and strong noise, so that the model removes noise and generates images. Its gradient backpropagation is only propagated to the online SiT and not to the variational autoencoder, which is used to optimize the parameters of the online SiT. 3) Generate face images by performing inverse diffusion sampling on the trained online SiT model.

2. The unconditional face image generation method based on end-to-end self-distillation according to claim 1, characterized in that, The implementation method for step 1) is as follows: A1) Obtain face images from the CelebA-HQ and FFHQ face datasets; A2) The acquired facial image The input is fed into the variational autoencoder (VAE) and compared with the increment factor of the current time step. Increment factor of previous time steps Multiply to obtain the latent feature representation of the face image at the current time step. Latent feature representation at previous time steps In addition, facial images The data is fed into a pre-trained face analysis network to obtain face region analysis results, including eyes, nose, mouth, and eyebrows. .

3. The unconditional face image generation method based on end-to-end self-distillation according to claim 1, characterized in that, In step 2), the implementation method of the end-to-end self-distillation unconditional generative network model is as follows: The end-to-end self-distillation unconditional generative network model adopts an end-to-end self-distillation representation alignment framework, including online SiT, target SiT, facial region sensitive noise injection strategy, and hypergraph guided attention mechanism. The end-to-end self-distillation characterization alignment framework has two branches, each consisting of an online SiT... and target SiT Implementation; the online SiT and target SiT They all use the same structure, consisting of multiple Transformer blocks and a projection head; among them, the online SiT As a student network, the target SiT Then, as a teacher network, online SiT and target SiT Shared architecture, targeting SiT Through online SiT Update the historical parameters; In the model data processing, a facial region-sensitive noise injection strategy is used to replace the random noise used in the standard diffusion model as the denoising target during training. The facial region-sensitive noise utilizes a pre-trained face analysis network to segment the face image into multiple semantically meaningful regions, including eyes, nose, mouth, and eyebrows, and injects them as a kind of facial region noise into temporary strong random Gaussian noise and temporary weak random Gaussian noise to obtain strong noise and weak noise. Semantic face segmentation is used to generate more targeted noise, enhancing the model's focus on key facial regions. During the model iterative training process, the 20th layer features of the target SiT are extracted through the hypergraph-guided attention mechanism to obtain the deep representation of the target. And compute the deep representation of the target through a hypergraph-guided attention mechanism. The corresponding hypergraph Laplacian matrix The Laplacian matrix of the hypergraph is used as a bias term for the face attention structure and input into the attention mechanism of online SiT to improve the relationship modeling between facial regions and improve the generation quality and structural consistency of the model. During gradient backpropagation, weak noise is input into the target SiT, and a deep target representation is extracted from its 20th layer. Strong noise is input into the online SiT, and a shallow online representation is extracted from its 8th layer. The difference between the shallow online representation and the deep target representation is measured by the representation alignment loss. The gradient is backpropagated to the variational autoencoder to optimize the parameters of the online SiT and the variational autoencoder. The online SiT takes strong noise as input, predicts noise, and uses a diffusion model loss function to measure the difference between the predicted noise and the strong noise. Its gradient backpropagation is only propagated to the SiT and not to the variational autoencoder, and is used to optimize the parameters of the online SiT.

4. The unconditional face image generation method based on end-to-end self-distillation according to claim 3, characterized in that, The implementation method of the facial region sensitive noise injection strategy is as follows: B1) Replace the random noise used in the standard diffusion model with noise that focuses more on the fine-grained structural regions of the face, as the denoising target during training; given an input facial image First, a face parser is applied to segment the image, dividing it into multiple semantically meaningful regions. Background regions are then uniformly removed to reduce the impact of irrelevant noise. Finally, a facial sensitive region is randomly selected from the parsing results. And inject random noise only within the selected area. ; B2) Fusion of random noise and original noise at different time steps; specifically: merging region-sensitive noise... Weighted and added to the weak noise at the current time step weak noise at previous time steps This generates two enhanced noise representations: in, This indicates strong noise that is increasing at the current time step. This represents the weak noise that was amplified in the previous time step. Indicates the current time step The noise scaling factor at each time step controls the intensity of the noise generated at that current time step. Indicates the steps taken in the past period of time Noise scaling factor at any given time. Indicates the current moment. Indicates earlier than the current time At that moment in time step It is noise limited to a selected facial area.

5. The unconditional face image generation method based on end-to-end self-distillation according to claim 3, characterized in that, The implementation method of the hypergraph-guided attention mechanism is as follows: C1) First, the target encoder is processed. Extracted facial representation Clustering is performed to group spatially and semantically similar regions together, where each cluster represents a set of visually similar facial regions; the clustering process is defined as follows: in, This represents the cluster assignment for each feature. It represents the number of clusters; KMeans() represents the KMeans algorithm, which assigns similar regions to the same cluster by minimizing the Euclidean distance within the cluster; each cluster is regarded as a hyperedge in the construction of a hypergraph, thereby capturing the higher-order connectivity and contextual relationships between different parts of the face; C2) Construct a hyperedge adjacency matrix based on clustering results. Each element Indicates the area Clustering The relationship between them; assuming the batch size is Each image has There are 10 regions, totaling 100 areas. The clusters are constructed as follows: Based on hyperedge adjacency matrix Calculate the corresponding hypergraph Laplacian matrix. The higher-order structural dependencies between encoded regions are as follows: in, It is a node degree matrix, representing the number of connections in each region. It is the hypermarginality matrix, representing the size of each cluster; through the Laplace matrix... Describe the structural interconnections of facial regions in the hypergraph and introduce them as structural biases into an attention mechanism; It is directly used as an additive bias and embedded into attention computation. In, that is: in, A learnable or preset scaling factor. Broadcast to all attention heads.

6. The unconditional face image generation method based on end-to-end self-distillation according to claim 5, characterized in that, In step C2), attention calculation The implementation method is as follows: In multi-head self-attention mechanisms, given an input feature map The query is obtained through linear projection calculation. ,key ,value matrix: in, , , The learnable weight matrix; the attention score matrix. Calculated using the scaled dot product attention mechanism: in, Let be the dimension of the key vector; the final attention output is obtained by... Apply the softmax function and adjust the value matrix. Weighted average yields: Each final cluster groups semantically and spatially coherent regions into a group, and these groups are used to construct the hypergraph adjacency matrix. ;based on Calculate the Laplacian matrix of the hypergraph To capture the high-order structural dependencies between facial components, the Laplacian matrix of the hypergraph is then injected as an additive structural bias into the attention mechanism. The final structure-aware self-attention representation is as follows: in, It is a learnable or preset scaling factor used to control the influence of structural priors.

7. The unconditional face image generation method based on end-to-end self-distillation according to claim 1, characterized in that, In step 2), the methods for constructing the diffusion loss and the characterization alignment loss are as follows: D1) Construct diffusion loss based on SiT, and learn the velocity field of the probabilistic flow ordinary differential equation PF-ODE through the flow-based model SiT. The forward process of SiT is defined as follows: in, It is the raw data. It's Gaussian noise. Indicates at time step Data at any given time and It's about time. Monotonically decreasing and increasing functions; PF ODE is given by the following formula: in, Representing data points Over time The rate of change of ; in order to learn the velocity field, minimize the following diffusion loss : in, This represents the parameters of the online diffusion model. for The time derivative is used to control the data scaling rate. for The derivative of the value is used to control the rate of change of noise intensity, and the desired operation. Indicates the original data ,noise and time step Sampling is performed and the average loss is calculated; the flow process is aligned with the data distribution by optimizing the velocity field; D2) In characterizing alignment loss, online SiT and target SiT Different levels of noise were injected: Receive lower noise ,and Receives high noise ; Indicates the current diffusion time step. This represents the offset of past time steps; after noise injection, intermediate feature representations are extracted from specific layers of each model; specifically: from The Layer obtained, from The Layer obtained, and This represents the corresponding internal transformer block or attention layer selected for feature supervision; its expression is: in, It's the batch size. It is the number of spatial markers for each image. It is a feature dimension; to facilitate better representation alignment, a learnable MLP projection head is introduced. Representing the intermediate features of the online model Mapped to To achieve optimal knowledge transfer, the model's features are aligned across different time steps and layers, minimizing the distance between student and teacher patch-level outputs; this represents the alignment loss. Represented as: in, Indicates the first One patch, This represents the parameters of the online diffusion model. The parameters representing VAE, This represents the parameters of the projection head and the expected operation. Represents all possible time steps High-resolution features on and time step Weighted averaging is performed; backpropagation is used to facilitate end-to-end training and jointly optimize VAE and diffusion transformer features. To assist in VAE parameter optimization; to prevent potential spatial structure degradation during backpropagation of the diffusion model, a gradient stopping operation is introduced to ensure... Used only for updating online diffusion models parameters .

8. The unconditional face image generation method based on end-to-end self-distillation according to claim 1, characterized in that, In step 2), to enhance the reconstruction performance of the VAE, three regularization losses are combined, including reconstruction loss. GAN loss and KL divergence loss This results in the total regularization loss of the VAE. ; Ultimately, the end-to-end training process is achieved through the following total loss. To optimize the parameters of the diffusion converter: in, , and They represent the online diffusion model, respectively. VAE and learnable projector The parameters, and Hyperparameters are used to control the relative importance of the corresponding loss terms.

9. An unconditional face image generation system based on end-to-end self-distillation, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, wherein when the processor executes the computer program instructions, it can implement the method as described in any one of claims 1-8.

Images