An attribute controllable face blind restoration method and system
By constructing multimodal input sequences and utilizing the cross-modal attention mechanism of the diffuse Transformer network, combined with attribute consistency and semantic contrast loss, the problem of the disconnect between restoration and attribute control in existing technologies is solved, and high-quality face images that are clear and conform to user-specified attributes are generated under severe degradation conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-02-25
- Publication Date
- 2026-06-02
Smart Images
Figure CN122134595A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of image processing and artificial intelligence technology, and specifically to a method and system for face blindness repair with controllable attributes. Background Technology
[0002] Blind face restoration refers to the recovery of high-quality faces from low-quality images affected by unknown degradation factors such as blurring, noise, downsampling, and compression. Because the degradation process is irreversible and information is missing, this problem is ill-posed; multiple different face solutions may correspond to the same observation, leading to semantic uncertainty in the restoration process.
[0003] While diffusion-based methods have made progress in perceptual quality, most still rely on visual cues, with textual cues serving only as weak aids, making reliable attribute control difficult under severe degradation. Existing text editing methods, although capable of modifying attributes, lack robust repair capabilities, and directly applying them to low-quality images amplifies artifacts. Common two-stage "repair first, then edit" approaches suffer from high inference costs and are prone to disrupting identity structures.
[0004] Current technologies suffer from the following key shortcomings, lacking a unified framework to integrate repair and attribute control: existing methods fail to integrate textual semantic instructions with structural priors of low-quality images in a single diffusion model, thus failing to achieve collaborative optimization that is both "clear and accurate in repair".
[0005] In addition, there is a lack of continuous attribute supervision mechanisms during the denoising process. Text guidance usually only applies to the input layer, lacking consistency constraints on generated attributes in intermediate stages, leading to semantic drift or attribute collapse.
[0006] Furthermore, there is a lack of high-quality paired data to support attribute-controlled training. Existing datasets are mostly coarse-grained labeled, lacking high-resolution image pairs with the same identity and only single attribute variations, making it difficult to support the training and evaluation of refined attribute-controlled models.
[0007] A search revealed that Chinese patent application number 202510268494.X relates to a method for text-based editing of facial image attributes based on a diffusion model. This method can achieve high-quality, flexible facial editing while maintaining identity consistency. However, this method fails to address the issues of using semantic instructions as prior structures, as well as semantic drift and attribute collapse. Summary of the Invention
[0008] In view of the deficiencies in the prior art, the purpose of this application is to provide a method and system for face blindness repair with controllable attributes.
[0009] The first aspect of this application provides a method for face blindness repair with controllable attributes, including: Receive a face image to be restored with an unknown cause of degradation and attribute text prompts input by the user, wherein the attribute text prompts are used to indicate the state of target attributes controlling the face to be restored; The face image and the attribute text prompts are encoded into visual tags and text tags, respectively, and then constructed into a unified multimodal input sequence; The multimodal sequence is input into a diffusion Transformer network. During the reverse denoising process, the attribute text prompts are used to semantically guide the face image restoration process, and the face image that is consistent with the attribute text prompts is output.
[0010] Optionally, the attribute text prompt is used to specify one or more editable attributes that the target face should have, including at least one of the following: whether the person is wearing glasses, a hat, beard condition, expression type, hair color, makeup style, and age range.
[0011] Optionally, the step of inputting the multimodal sequence into a diffusion Transformer network, and using a cross-modal attention mechanism during the reverse denoising process to semantically guide the face image restoration process through the attribute text prompts, and outputting a final face image consistent with the attribute text prompts, includes: The latent variables to be generated are initialized with noise randomly sampled from a standard Gaussian distribution. ; With initialized latent variables Starting from the discrete reverse time step t=T,T Iterative denoising is performed sequentially on 1,…,1; At each time step t, the current noisy latent variable is... The diffusion Transformer network is fed with a fixed, invariant multimodal conditional sequence. Each layer of the network uses cross-modal joint self-attention to allow latent variables to actively pay attention to visual and textual tags associated with them, generating latent variable-guided responses; Based on the latent variable-guided response, the noise residual estimate at the corresponding time step t is output, and the latent variable is updated according to the preset backdiffusion sampling rule to obtain... ; When the iteration reaches t=0, the pure latent variable x0 is obtained, which is then input into the pre-trained VAE decoder to reconstruct the final face image.
[0012] Optionally, the training loss function of the Diffusion Transformer network includes attribute consistency loss; The attribute consistency loss is obtained through the following process: At one or more intermediate time steps in the diffusion denoising process, VAE decoding is performed on the current noisy latent variable to obtain the intermediate reconstructed image; The intermediate reconstructed image is input into a pre-trained facial attribute encoder to extract facial attribute embeddings; The difference between the facial attribute embedding and the target attribute embedding is calculated to obtain the attribute consistency loss; The target attribute embedding is obtained by extracting the corresponding reference image through the attribute encoder.
[0013] Optionally, the training loss function of the Diffusion Transformer network is expanded to include semantic contrastive loss; The semantic contrast loss is implemented through a semantic dual-branch training strategy, specifically including: For the same face image to be repaired, two semantically mutually exclusive attribute text prompts are input in parallel to obtain two conditional branches of repair output; For the intermediate or final reconstructed images of each of the two branches, facial attribute embeddings are extracted, and the semantic contrast loss of the two facial attribute embeddings is calculated.
[0014] Optionally, the semantic contrast loss is a boundary-based margin constraint loss, defined as: ; in, and The facial attribute embeddings of the two branch output images are extracted by a pre-trained attribute encoder. This is the preset minimum interval hyperparameter; The semantic contrast loss is calculated only on the semantic dimension of the opposing attributes corresponding to the two mutually exclusive attribute text prompts.
[0015] Optionally, the two semantically mutually exclusive attribute text prompts correspond to the opposing states of the same individual in a single semantic attribute, including: wearing glasses / not wearing glasses, smiling / neutral expression, heavy makeup / light makeup, curly hair / straight hair, with beard / without beard; The two branches share the same diffuse Transformer network backbone, differing only in the text prompts in the conditional inputs.
[0016] A second aspect of this application provides a face blindness repair system with controllable attributes, comprising: A multimodal coding module receives a face image to be restored with an unknown cause of degradation and attribute text prompts input by the user, wherein the attribute text prompts are used to indicate the target attribute state of the face to be restored; The fusion module encodes the face image and the attribute text prompts into visual tags and text tags, respectively, and constructs them into a unified multimodal input sequence; The diffusion-based semantic guidance repair module inputs the multimodal sequence into the diffusion Transformer network. During the reverse denoising process, it uses a cross-modal attention mechanism to achieve semantic guidance of the attribute text prompts for the face image repair process, and outputs a face image consistent with the attribute text prompts.
[0017] A third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described herein.
[0018] A fourth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods described above.
[0019] The attribute-controllable face blindness repair method provided in this application constructs a unified multimodal input sequence by fusing attribute text prompts with the low-quality face image to be repaired, and uses a cross-modal attention mechanism in a diffuse Transformer network to achieve semantic guidance of the repair process by the text. It can generate high-quality face images that maintain the rationality of the identity structure and conform to the user-specified attributes under severe degradation conditions, which significantly improves the controllability and semantic consistency of face blindness repair.
[0020] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description
[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating an attribute-controllable face blindness repair method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the implementation architecture of a property-controllable face blindness repair method according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating the effect of a property-controllable face blindness repair method according to an exemplary embodiment; Figure 4 This is a schematic diagram of the structure of a face blindness repair system with controllable attributes, according to an exemplary embodiment. Detailed Implementation
[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.
[0023] In existing technologies, there is a lack of a unified framework to integrate repair and attribute control, making it impossible to achieve collaborative optimization that is both clear and accurate in repair. Based on the above problems, this application provides an attribute-controllable face blindness repair method to solve these issues.
[0024] Reference Figure 1 and Figure 2 As shown, a method for repairing face blindness with controllable attributes includes the following steps: S100, receive a face image to be restored with an unknown cause of degradation and an input attribute text prompt, wherein the attribute text prompt is used to indicate the state of the target attribute of the face to be restored; S200, the face image and attribute text prompts are encoded into visual tags and text tags respectively, and constructed into a unified multimodal input sequence; S300 inputs multimodal sequences into a diffusion Transformer network. During the reverse denoising process, it uses a cross-modal attention mechanism to provide semantic guidance for the face image restoration process through attribute text prompts, and outputs a face image consistent with the attribute text prompts.
[0025] The above implementation of this application, by constructing a joint input and processing flow of images and text, achieves deep fusion of low-quality image structure priors and text semantic instructions in a single diffusion Transformer network, solves the problem of separation between repair and attribute control in existing methods, and realizes integrated collaborative optimization that is both "clearly repaired and accurately modified".
[0026] To achieve multimodal coding, some specific embodiments of this application, such as Figure 2 As shown, step S100 includes the following steps: S101, Receive a low-quality face image to be repaired for an unknown cause of degradation.
[0027] Specifically, the image may contain one or more combined degradations, including blurring, noise, downsampling, and compression artifacts, and the specific degradation process cannot be clearly known.
[0028] S102, the image is input into a pre-trained variational autoencoder (VAE) encoder, its latent representation is extracted and divided into image blocks to obtain a visual label sequence, which preserves the structural and appearance information of the degenerated face.
[0029] Example image to be repaired (ILQ): A selfie taken with a mobile phone, which has a blurred face, slight mosaic (compression artifact) and noise introduced by insufficient light, which is a combined degradation case of blur and compression artifact noise.
[0030] This blurry selfie is fed into a pre-trained VAE encoder (such as the Encoder in LDM). VAE compresses it into a 4×64×64 latent variable tensor (i.e., 4 channels, 64×64 pixels). The tensor is then divided into 64 image patches of 8×8, and each patch is flattened into a 256-dimensional vector; This yields 64 visual markers. Each tag implicitly contains local structural information.
[0031] S103 receives attribute text prompts input by the user, converts the text into a word sequence through a text segmenter, and inputs it into a pre-trained text encoder (such as CLIP) to obtain the corresponding text tag sequence. The text tags contain semantic descriptions of the target attributes, such as "wearing glasses", "smiling expression", "long black hair", etc.
[0032] For example: The attribute text hint is "An Asian woman wearing thin gold-rimmed glasses, with a smile on her face."
[0033] The above text is segmented into word units: ["wear", "gold", "thin frame", "glasses", "of", "Asian", "woman", ",", "faceband", "smile"] (10 in total); The input CLIP text encoder maps each word to a 512-dimensional semantic vector. This will yield 10 text tags. Each carries a clear semantic meaning.
[0034] The above embodiments achieve cross-modal semantic alignment by encoding low-quality images and attribute text separately and generating visual and text tags, enabling text instructions to effectively guide the image restoration process and providing a unified representation basis for attribute-controllable high-fidelity reconstruction.
[0035] In order to obtain a unified multimodal input sequence, in some specific embodiments of this application, step S200 includes the following steps: concatenating the visual marker sequence and the text marker sequence in sequence to construct a unified multimodal marker sequence, which serves as the basic input for subsequent cross-modal fusion and diffusion denoising.
[0036] For example: By concatenating the above 64 visual tags and 10 text tags in sequence, a unified sequence of length 74 is obtained: .
[0037] The above embodiments, by unifying the encoding of visual and textual modalities and constructing a joint label sequence, achieve the alignment and fusion of image structural priors and textual semantic instructions, providing basic support for simultaneously completing high-fidelity restoration and controllable attribute generation in a single model.
[0038] To achieve face restoration, in some specific embodiments of this application, in step S300, a multimodal sequence is input into a diffusion Transformer network. During the reverse denoising process, a cross-modal attention mechanism is used to implement semantic guidance of the face image restoration process through attribute text prompts, and a face image consistent with the attribute text prompts is output, including the following steps: S301, with a standard Gaussian distribution Random sampling to initialize the latent variables to be generated The latent variable is located in the latent space of the pre-trained variational autoencoder (VAE); S302, in the discrete reverse time step sequence t=T,T Iterative denoising is performed sequentially on 1,…,1; S303, at each time step t The current noisy latent variable With fixed and unchanging multimodal conditional sequences (Where V is the visual marker obtained by VAE encoding of a low-quality face image, and T is the text marker extracted by a text encoder from attribute text cues) together constitute a unified input sequence, and are superimposed with learnable positional encoding and temporal step embedding. Input is fed into the diffused Transformer network; S304, in each layer of the Diffusion Transformer, through a cross-modal joint self-attention mechanism, maps the entire unified input sequence into a query, key, and value matrix, i.e., Q = XW. Q K=XW K V=XW V Where X is the network input (latent variables and multimodal conditional sequences), W Q / K / V For learnable projection weights; Calculate the attention weight matrix ; S305, Extract the row submatrix AI corresponding to the latent variables from the attention weight matrix. x Thus, the cross-modal guided response of the latent variable is obtained: H x =AI x V; S306, H x After being processed by layer normalization and feedforward network, it is fed into a dedicated noise prediction head to output the noise residual estimate of the current time step; S307, based on the preset back-diffusion sampling rule (such as the DDIM or DDPM formula), uses this noise estimation to update the latent variables, obtaining... ; S308: When iterating to t=0, a nearly noise-free pure latent variable x0 is obtained, which is then input into a pre-trained VAE decoder to reconstruct a high-fidelity, attribute-controllable final face image.
[0039] In the above embodiments, at each denoising time step, a "cue-guided repair space" jointly defined by observation data and instructions is constructed by dynamically fusing pixel-level structural constraints provided by low-quality images and semantic-level attribute guidance provided by text prompts through a cross-modal attention mechanism. During the diffusion denoising process, the latent variable representation is updated step by step, and while maintaining the consistency of identity and geometric structure, the generated results are guided to converge toward the target attributes described in the text, thereby alleviating the ambiguity and semantic uncertainty in blind repair.
[0040] The above embodiments, through the construction of a cross-modal joint attention and cue-guided repair space, realize the continuous guidance of text semantics on the entire repair process, and balance identity authenticity and attribute control within a single framework, effectively alleviating the problem of multiple solutions in blind repair.
[0041] To enhance the model's ability to perceive facial attributes and avoid attribute drift under severe degradation conditions, this application introduces attribute-aware learning (AAL) during the training phase. In one specific implementation, attribute-aware learning includes the following steps: S401, at several intermediate time steps t during the diffusion denoising process, obtain the currently predicted latent variables. The intermediate restored image is then reconstructed using a variational autoencoder (VAE) decoder. ; S402, retrieve the intermediate restored image obtained in S401. Corresponding real high-quality target image Input the attribute-aware encoder (AAE) together; S403, the attribute-aware encoder includes a face detection module and an attribute encoder: S4031, First, the face region in the input image is detected and spatially aligned to ensure that attribute extraction is not affected by pose; S4032, and then extract multidimensional attribute embedding vectors through a pre-trained attribute encoder to represent editable facial attributes such as glasses, expressions, hair color, and age; Obtain the attribute embeddings of the intermediate images respectively Attribute embedding of the target image ; S404, Calculate the attribute consistency loss between the two:
[0042] Where BCE represents the binary cross-entropy loss or the equivalent semantic classification loss function, used to supervise the intermediate generated results to approximate the real target in the attribute space; S405 combines the attribute consistency loss with the diffusion denoising reconstruction loss at the current time step. Weighted fusion yields the overall optimization objective for attribute-aware learning:
[0043] Where λ>0 is a hyperparameter used to balance the optimization weights between pixel-level reconstruction accuracy and semantic-level attribute consistency; S406, Backpropagation is performed to the diffusion Transformer network to update the model parameters, so that the denoising process can continue to focus on the target attributes while pursuing high-fidelity details, and suppress attribute misgeneration or drift caused by severe degradation.
[0044] The above embodiments, by continuously introducing attribute loss in the diffusion trajectory, transform face blindness repair from a simple pixel-driven optimization to a process of semantic-driven and pixel-driven combined action. This allows the model to maintain attention to the target attributes while pursuing clarity and detail, significantly reducing the risk of "correcting attributes in order to make it clearer".
[0045] To further enhance attribute controllability and overcome the problem that relying solely on attribute-aware learning is insufficient to fully respond to different textual instructions, some specific implementations of this application introduce a semantic dual-branch training (SDT) mechanism. This mechanism enhances the model's sensitivity to differences in textual prompts by constructing strong contrastive supervision signals. Specifically, for example... Figure 2 As shown, it includes the following steps: S501, Construct training sample pairs.
[0046] Select two high-quality face images of the same individual from the dataset. and The two differ only in a certain semantic attribute (such as "not wearing glasses" vs. "wearing glasses", "neutral expression" vs. "smiling"). By applying the same degradation operation to two high-resolution images using a unified degradation model, the same low-quality input image is generated. Configure corresponding text prompts for the source and target images respectively: Source text hints : Describes the original attribute state (e.g., "a face without glasses") Target text prompt : Describe the modified attribute state (e.g., "face wearing glasses") For example, the dataset could be AttrFace-90K, which contains large-scale pairs of face images with consistent identities and controllable attributes.
[0047] S502, perform dual-branch repair.
[0048] Combined input and The images are fed into two diffusion Transformer repair branches with shared weights to obtain the corresponding predicted images. and ; The two output images are input into a pre-trained attribute-aware encoder (AAE) to extract their multidimensional attribute embeddings. and ; S503 introduces interval-based semantic contrast loss to enhance attribute discrimination capabilities.
[0049] On the dimension labeled "attribute differences exist", calculate the Euclidean distance between the two attribute embeddings and apply boundary constraints: .
[0050] Where m>0 is the preset minimum margin hyperparameter. This loss only applies to the target attribute-related dimensions (such as "whether glasses are worn"), encouraging the two branch outputs to produce sufficiently large feature separation in this dimension, preventing the generated results from converging due to small changes in the prompts (i.e., "attribute collapse").
[0051] Specifically, multidimensional attribute embedding and It includes multiple semantic attribute dimensions, but when calculating the semantic contrast loss, it only calculates the single semantic attribute dimension indicated by the mutually exclusive attribute text prompts (such as "wearing glasses" and "not wearing glasses") in step S501. Here, "labeled" means that the corresponding single dimension is automatically determined from the multiple semantic dimensions of the multi-dimensional attribute embedding based on the semantic attribute category corresponding to the input mutually exclusive attribute text prompts, without the need for manual labeling.
[0052] S504, jointly optimize the overall training objective.
[0053] The attribute-aware learning loss of each of the two branches Compared with the semantic contrast loss mentioned above The fusion constitutes the final training objective:
[0054] Where λ is the balance coefficient. That is, by adjusting the weight coefficient, a balance is achieved between high-fidelity reconstruction and strong attribute controllability, thereby realizing face blindness repair that is both clear and natural and strictly follows the text instructions.
[0055] For example, the model was trained on a training platform consisting of eight NVIDIA GeForce RTX A6000 GPUs. The model includes a VAE encoder, a CLIP text encoder, a multimodal sequence concatenation module (with learnable positional encoding), and a diffusion Transformer network.
[0056] The Flux 1.0-Dev DiffusionTransformer backbone network was lightly tuned using the LoRA (Low-Rank Adaptation) method, where the LoRA rank was set to 64 and the scaling factor alpha was set to 4.
[0057] To improve the model's generalization ability, a Dropout mechanism was introduced into the input layers of both the text encoder and the image encoder, with a dropout probability set to 0.01. The input image resolution was uniformly adjusted to 512×512 pixels during both the training and inference phases.
[0058] The entire training process consisted of approximately 60,000 optimization steps, using a single-sample batch (batch size = 1), with Prodigy as the optimizer and an initial learning rate of 1.0.
[0059] The above embodiments, through the synergistic effect of semantic bi-branch training and interval contrast loss, force the model to generate outputs with clear attribute differences for different text prompts under the same low-quality input, thereby improving the sensitivity and reliability of attribute control and achieving fine-grained repair that is "accurate in modification and does not change the user".
[0060] Furthermore, in some specific implementations, the trained model is applied, and the results are shown in the figure below. Figure 3As shown, the first column on the left is the image to be repaired, the second column is the repair result without attribute requirements, and each subsequent column shows the diverse results generated by applying different facial attribute conditions (such as "smiling" and "wearing glasses") to the same input image, indicating that this application has a controllable and user-instruction-oriented facial repair capability.
[0061] Based on the same technical concept, other embodiments of this application provide a face blindness repair system 100 with controllable attributes, such as... Figure 4 As shown, it includes: The multimodal coding module 110 receives a face image to be restored with an unknown cause of degradation and attribute text prompts input by the user, wherein the attribute text prompts are used to indicate the target attribute state of the face to be restored; The fusion module 120 encodes the face image and attribute text prompts into visual tags and text tags respectively, and constructs them into a unified multimodal input sequence; The diffusion-based semantic guidance repair module 130 inputs multimodal sequences into a diffusion Transformer network. During the reverse denoising process, it uses a cross-modal attention mechanism to provide semantic guidance for the face image repair process through attribute text prompts, and outputs a face image consistent with the attribute text prompts.
[0062] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the face blindness repair method with controllable attributes in the above embodiments, and will not be repeated here.
[0063] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.
[0064] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.
[0065] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.
[0066] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.
[0067] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.
[0068] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.
[0069] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.
[0070] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.
[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0075] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.
Claims
1. A method for face blindness repair with controllable attributes, characterized in that, include: Receive a face image to be restored with an unknown cause of degradation and an input attribute text prompt, wherein the attribute text prompt is used to indicate the state of the target attribute of the face to be restored; The face image and the attribute text prompts are encoded into visual tags and text tags, respectively, and then constructed into a unified multimodal input sequence; The multimodal sequence is input into a diffusion Transformer network. During the reverse denoising process, the attribute text prompts are used to semantically guide the face image restoration process, and the face image that is consistent with the attribute text prompts is output.
2. The method for face blindness repair with controllable attributes according to claim 1, characterized in that, The attribute text prompt is used to specify one or more editable attributes that the target face should have, including at least one of the following: whether the person is wearing glasses, a hat, beard condition, expression type, hair color, makeup style, and age range.
3. The method for repairing face blindness with controllable attributes according to claim 1, characterized in that, The step of inputting the multimodal sequence into a diffusion Transformer network, and using a cross-modal attention mechanism during the inverse denoising process to semantically guide the face image restoration process with the attribute text prompts, and outputting a final face image consistent with the attribute text prompts, includes: The latent variables to be generated are initialized with noise randomly sampled from a standard Gaussian distribution. ; With initialized latent variables Starting from the discrete reverse time step t=T,T Iterative denoising is performed sequentially on 1,…,1; At each time step t, the current noisy latent variable is... The diffusion Transformer network is fed with a fixed, invariant multimodal conditional sequence. Each layer of the network uses cross-modal joint self-attention to allow latent variables to actively pay attention to visual and textual tags associated with them, generating latent variable-guided responses; Based on the latent variable-guided response, the noise residual estimate at the corresponding time step t is output, and the latent variable is updated according to the preset backdiffusion sampling rule to obtain... ; When the iteration reaches t=0, the pure latent variable x0 is obtained, which is then input into the pre-trained VAE decoder to reconstruct the final face image.
4. The method for repairing face blindness with controllable attributes according to claim 1, characterized in that, The training loss function for the Diffusion Transformer network includes attribute consistency loss; The attribute consistency loss is obtained through the following process: At one or more intermediate time steps in the diffusion denoising process, VAE decoding is performed on the current noisy latent variable to obtain the intermediate reconstructed image; The intermediate reconstructed image is input into a pre-trained facial attribute encoder to extract facial attribute embeddings; The difference between the facial attribute embedding and the target attribute embedding is calculated to obtain the attribute consistency loss; The target attribute embedding is obtained by extracting the corresponding reference image through the attribute encoder.
5. The method for repairing face blindness with controllable attributes according to claim 1, characterized in that, The training loss function for the Diffusion Transformer network includes semantic contrastive loss; The semantic contrast loss is implemented through a semantic dual-branch training strategy, specifically including: For the same face image to be repaired, two semantically mutually exclusive attribute text prompts are input in parallel to obtain two conditional branches of repair output; For the intermediate or final reconstructed images of each of the two branches, facial attribute embeddings are extracted, and the semantic contrast loss of the two facial attribute embeddings is calculated.
6. The method for face blindness repair with controllable attributes according to claim 5, characterized in that, The semantic contrast loss is a boundary-based margin constraint loss, defined as: ； in, and The facial attribute embeddings of the two branch output images are extracted by a pre-trained attribute encoder. This is the preset minimum interval hyperparameter; The semantic contrast loss is calculated only on the semantic dimension of the opposing attributes corresponding to the two mutually exclusive attribute text prompts.
7. The method for repairing face blindness with controllable attributes according to claim 5, characterized in that, The two mutually exclusive attribute text prompts correspond to the opposing states of the same individual in a single semantic attribute, including: wearing glasses / not wearing glasses, smiling / neutral expression, heavy makeup / light makeup, curly hair / straight hair, with beard / without beard; The two branches share the same diffuse Transformer network backbone, differing only in the text prompts in the conditional inputs.
8. A face blindness repair system with controllable attributes, characterized in that, include: A multimodal coding module receives a face image to be restored with an unknown cause of degradation and attribute text prompts input by the user, wherein the attribute text prompts are used to indicate the state of target attributes controlling the face to be restored; The fusion module encodes the face image and the attribute text prompts into visual tags and text tags, respectively, and constructs them into a unified multimodal input sequence; The diffusion-based semantic guidance repair module inputs the multimodal sequence into the diffusion Transformer network. During the reverse denoising process, it uses a cross-modal attention mechanism to achieve semantic guidance of the attribute text prompts for the face image repair process, and outputs a face image consistent with the attribute text prompts.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.
10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.