A text-driven based face image editing method and system

Abstract

The application provides a text-driven face image editing method and system, and belongs to the technical field of computer vision and digital image processing; the method comprises the following steps: a target face image is edited and coded by an instruction understanding and memory establishing module; a semantic modeling of a face attribute change process is performed in a latent space of a generative model by a latent space semantic editing module, and a latent variable is updated along a continuous trajectory consistent with an editing coding semantic; an editing result is evaluated by introducing a cross-modal semantic alignment constraint in an editing evaluation and control module, and a continuation and termination of the editing process are controlled based on the evaluation result; a generative model is guided to generate an image by a fine attribute predictor module; the generated image is evaluated for authenticity by a discriminator, so that generator parameters are optimized and a final image is generated; and the application can achieve fine face editing in line with user semantic expectations while maintaining face identity features and visual quality.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and digital image processing technology, and in particular relates to a text-driven face image editing method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Face image editing is a highly challenging and popular research area in computer vision and digital image processing. It aims to edit local features or global styles of face images based on user-inputted attributes or instructions, outputting new, realistic, and natural-looking face images that meet the user's editing goals. This technology has important applications in digital entertainment, medical diagnosis, virtual reality, and virtual social interaction.

[0004] Traditional face editing methods utilize facial feature extraction, image enhancement, and edge detection to deform and edit facial images. However, they often lack accuracy and adaptability when dealing with images exhibiting complex facial expression changes. In recent years, with the continuous development of deep learning technology, face editing methods based on deep neural networks have gradually become mainstream, demonstrating strong potential in facial expression modification, attribute change control, and personalized customization. However, these methods primarily rely on predefined attribute labels, limiting their understanding and editing capabilities regarding user intent. They cannot support continuous, free editing or accurately depict refined, personalized editing needs. With the development of multimodal learning, researchers have begun to incorporate natural language into face editing to enhance the flexibility of interaction and semantic expression, enabling a text-driven, continuous, controllable, and natural editing process.

[0005] Although existing research in this field has made some progress, the following technical shortcomings still exist: (1) Existing technologies are difficult to achieve precise semantic constraints, and over-editing is likely to occur, resulting in generated images often accompanied by artifacts and unnatural transitions.

[0006] (2) Existing technologies struggle to accurately understand a user's multiple editing intentions, leading to an inability to accurately implement multiple consecutive editing commands. This results in issues such as poor continuity between the edited content and the context, distorted identity, and inconsistent details. These problems severely limit the application of the method in real-world scenarios. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention provides a text-driven face image editing method and system that can achieve refined face editing that meets the user's semantic expectations while maintaining facial identity features and visual quality.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a text-driven face image editing method.

[0009] A text-driven face image editing method includes: Natural language editing instructions for the target face image are obtained, semantic parsing is performed, and editing history and context information from multiple rounds of interaction are stored through a dialogue semantic memory mechanism to generate editing codes; Semantic modeling of facial attribute changes is performed in the latent space of the generative model, and latent variables are updated along a continuous trajectory consistent with the edit-encoded semantics to regulate attribute strength. The editing results are evaluated based on cross-modal semantic alignment constraints, and the latent space update step size is adaptively adjusted based on the current editing state and the degree of goal achievement. An automatic termination mechanism is set based on the evaluation results to control the continuation and termination of the editing process. By using attribute-supervised loss to guide the generative model in generating images, and by using adversarial loss to constrain realism, the realism of the generated images is evaluated, thereby optimizing the generator parameters and generating the final image.

[0010] Furthermore, the generation of the edit code includes: in the first round of editing, mapping the face image to the latent space of the generative model based on the CLIP image encoder to obtain the initial latent vector; at the same time, performing semantic parsing of the natural semantic instructions based on the CLIP language encoder to generate edit codes containing target attributes, change direction and degree information.

[0011] Furthermore, based on the aforementioned dialogue semantic memory mechanism, the editing history and context information in the multi-round interaction process are managed in a unified manner. Specifically, this includes: the instruction understanding process is carried out under the joint conditions of the current input and historical memory. That is, when the current round instruction lacks a clear attribute reference or editing scope, the current round instruction is completed and corrected based on the memory state, so as to parse the relative and ellipsis-like instructions into a semantic description consistent with the global editing direction and degree.

[0012] Furthermore, when semantically modeling the facial attribute change process in the latent space of the generative model, the initial latent vector is used as the starting point for subsequent rounds of facial editing, and the latent vector is updated during the editing process to reflect the facial attribute changes; the entire editing process is carried out in the generative model based on StyleGAN.

[0013] Furthermore, a learnable latent space field function is introduced during the editing process to adaptively generate displacement based on the latent vector of the current face image, thereby achieving semantic editing related to the editing state. The field function takes the current latent vector as input and outputs a field vector consistent with the latent space dimension. The field vector is aligned by the forward mapping of the generator and the final displacement is obtained by zero-input correction.

[0014] Furthermore, the latent space update step size is adaptively adjusted based on the current editing state and the degree of goal achievement, including: first, calculating the remaining gap between the current editing result and the target semantics according to the CLIP evaluation score, and combining it with the relative editing intensity obtained from instruction parsing to obtain an initial step size candidate value; then, by introducing the step size information of the previous round in the historical editing state, the current step size is subjected to smooth constraint and the current step size is limited to a preset range to obtain the final editing step size.

[0015] Furthermore, the automatic termination mechanism includes: terminating the editing process in the latent space when either a first termination condition or a second termination condition is met; otherwise, the editing process continues; wherein the first termination condition and the second termination condition are respectively represented as: ; ; in, This represents the semantic enhancement convergence threshold, used to determine whether semantic changes have stabilized after several rounds of editing. This represents the target semantic matching threshold, used to determine whether the current generated result has reached the expected semantic strength; Indicates the first The semantic similarity between the edit state after the round ends and the target description. Indicates the first The degree to which the initial editing state of the wheel is semantically close to the target description.

[0016] A second aspect of the present invention provides a text-driven face image editing system.

[0017] A text-driven face image editing system includes: The instruction understanding and memory building module is configured to: acquire natural language editing instructions from the target face image, perform semantic parsing, and store the editing history and context information during multiple rounds of interaction through a dialogue semantic memory mechanism to generate editing codes; The latent space semantic editing module is configured to: perform semantic modeling of the facial attribute change process in the latent space of the generated model, and update the latent variables along a continuous trajectory consistent with the edited encoded semantics to regulate the attribute intensity; The editing evaluation and control module is configured to: evaluate the editing results based on cross-modal semantic alignment constraints, and adaptively adjust the latent space update step size based on the current editing state and the degree of goal achievement; and set an automatic termination mechanism based on the evaluation results to control the continuation and termination of the editing process. The fine-grained attribute predictor module is configured to guide the generative model to generate images using a constrained attribute supervised loss. The discriminator is configured to evaluate the realism of the generated image by constraining the realism to counteract the loss, in order to optimize the generator parameters and generate the final image.

[0018] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a text-driven face image editing method as described in the first aspect of the present invention.

[0019] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a text-driven face image editing method as described in the first aspect of the present invention.

[0020] The above one or more technical solutions have the following beneficial effects: (1) After each round of editing, this invention uses cross-modal semantic alignment constraints to quantitatively evaluate the consistency between the current generated result and the target semantics. Based on this, it dynamically senses the real-time gap between the editing process and the target, and adaptively adjusts the step size of subsequent latent space updates based on this gap, thereby avoiding the attribute overshoot and latent space oscillation problems common in fixed step size modes. At the same time, the automatic termination mechanism set based on the evaluation results can stop the editing process in time when the semantic changes tend to converge or have reached the target strength, avoiding over-editing. This invention makes the entire editing process a smooth convergence process dynamically guided by the semantic target, effectively suppressing the generation of artifacts and unnatural transitions, and ensuring the high-quality visual realism of the generated image.

[0021] (2) Instead of processing single-turn instructions in isolation, this invention uses a dialogue semantic memory mechanism to structure and store all editing history and context information in multi-turn interactions. When parsing the current input instruction in each turn, it is jointly modeled with the historical memory, so as to accurately understand the continuous editing needs of human spoken language. On this basis, the initial latent vector is used as the fixed starting point of multi-turn editing and is recursively updated along a semantically consistent continuous trajectory in the latent space, so that the editing results of each turn are naturally connected with the historical context in terms of semantics and remain stable in terms of identity features. This solves the problems of context breakage, identity distortion and inconsistency of details that are common in multi-turn editing in existing technologies, and significantly improves the application capability of the model in real multi-turn interaction scenarios.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a flowchart of a text-driven face image editing method according to Embodiment 1 of the present invention.

[0025] Figure 2 This is an example diagram of the dataset in Embodiment 1 of the present invention.

[0026] Figure 3 This is a diagram showing the effect of multi-round editing of facial attributes in Embodiment 1 of the present invention.

[0027] Figure 4 This is a comparison chart showing the editing effects of the method of the present invention and existing methods regarding age and smile attributes.

[0028] Figure 5 This is a comparison chart showing the editing effects of the method of the present invention and existing methods on the attributes of bangs, glasses, and beard.

[0029] Figure 6 This is a comparison chart of the ablation experiment results in Embodiment 1 of the present invention. Detailed Implementation

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0032] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0033] The overall approach of this invention is as follows: This invention provides a text-driven face image editing method that integrates a dialogue memory mechanism, cross-modal semantic alignment, and adaptive control strategies. Given a face image, this method can continuously and controllably edit facial attributes during multiple rounds of interaction based on the user's editing instructions described in natural language, generating a face editing result that is highly consistent with the target semantics, preserves identity, and is visually realistic. The model incorporates a dialogue memory-driven editing management mechanism that integrates contextual semantic awareness, uniformly maintaining and updating historical editing trajectories, current attribute states, and contextual semantic information. This accurately understands the user's relative instructions and implicit intentions during continuous interaction, effectively solving the editing deviation problem caused by insufficient semantic alignment between visual space and linguistic instructions in multi-round editing. A cross-modal semantic alignment loss is introduced to optimize the editing process in a unified visual-linguistic semantic space, guiding the editing process to continuously converge towards the target semantic state, thereby achieving stable control over the semantic consistency and visual quality of the editing result. An adaptive step size adjustment strategy is introduced at the editing control level. The editing amplitude is dynamically adjusted according to the distance between the current editing result and the target state in the semantic space, thereby effectively avoiding overshoot and oscillation problems during attribute change.

[0034] Example 1 This embodiment discloses a text-driven face image editing method.

[0035] like Figure 1 As shown, a text-driven face image editing method includes: Natural language editing instructions for the target face image are obtained, semantic parsing is performed, and editing history and context information from multiple rounds of interaction are stored through a dialogue semantic memory mechanism to generate editing codes; Semantic modeling of facial attribute changes is performed in the latent space of the generative model, and latent variables are updated along a continuous trajectory consistent with the edit-encoded semantics to regulate attribute strength. The editing results are evaluated based on cross-modal semantic alignment constraints, and the latent space update step size is adaptively adjusted based on the current editing state and the degree of goal achievement. An automatic termination mechanism is set based on the evaluation results to control the continuation and termination of the editing process. By using attribute-supervised loss to guide the generative model in generating images, and by using adversarial loss to constrain realism, the realism of the generated images is evaluated, thereby optimizing the generator parameters and generating the final image.

[0036] Based on the above method, this invention can achieve refined face editing that meets user semantic expectations while maintaining facial identity features and visual quality. To facilitate understanding of the technical solution of this invention, the specific implementation methods of this invention will be further explained and described below.

[0037] The facial image editing model implemented in this method is based on a framework consisting of four parts: an instruction understanding and memory building module, a latent space semantic editing module, an editing evaluation and control module, and a fine-grained attribute predictor and discriminator. The instruction understanding and memory building module is responsible for semantically parsing the natural language editing instructions input by the user and, combined with a dialogue memory mechanism, storing the editing history and contextual information from multiple rounds of interaction. This generates editing codes containing target attributes, direction of change, and degree of change information, ensuring consistency and coherence in instruction understanding during continuous dialogue scenarios. The latent space semantic editing module performs semantic modeling of the facial attribute change process in the latent space of the generated model. By progressively updating latent variables along a semantically consistent continuous trajectory of the editing codes, it achieves fine-grained control over the degree of attribute changes. The editing evaluation and control module evaluates the editing results by introducing cross-modal semantic alignment constraints and adaptively adjusts the latent space update step size based on the current editing state and the degree of goal achievement, thereby ensuring semantic consistency while avoiding overshoot. Furthermore, an automatic termination mechanism is designed based on the evaluation results to control the continuation or termination of the editing process. The fine-grained attribute predictor helps the model further optimize the generated images by constraining attribute supervision loss, thereby ensuring that the editing results can maintain the accuracy and consistency of attributes while meeting user needs. The discriminator is optimized together with the generator, and evaluates the realism of the generated images by constraining realism to counteract loss, thereby driving the generator to continuously improve and generate results that are more and more consistent with the distribution of real images.

[0038] The specific implementation process of the method of this invention can be summarized as follows: The user inputs the face image they want to edit. and natural language commands First, in the first round of editing, the face image... Pre-trained CLIP image encoder Mapping to the model's latent space yields the corresponding latent vectors. Meanwhile, natural language instructions Pre-trained CLIP language encoder Perform semantic parsing and generate edit codes. This includes target attributes. , change direction and the degree of change Key information such as...

[0039] Then, the model, based on the changing patterns of facial attributes in the latent space, guides continuous updates of latent variables and, in conjunction with the currently stored editing state and historical context information, updates the latent vectors multiple times to achieve a gradual adjustment of attribute levels. Specifically, in the... During the round editing process, the model is based on the current latent vector. Editing and coding and the stored dialogue memory state Generates a target encoding that represents the desired outcome of the current round. The latent vector is updated based on this objective.

[0040] After each round of editing, the model integrates the existing editing history through a dialogue memory mechanism to form an updated memory state. This information is then stored in a structured format as a context-aware, multi-turn edit memory file to display the current editing context and historical decision information to the user. Additionally, a multimodal alignment constraint is introduced to update the latent vectors. Input generator Obtain the edited face image The edited result (i.e., the edited face image) and the target text description. The semantic similarity is obtained by evaluating the semantic consistency between them. Furthermore, an overall optimization objective is introduced to guide the editing process in the latent space towards updates consistent with linguistic semantics. Simultaneously, the model adjusts the current semantic similarity... The update step size in the latent space is adaptively adjusted according to the degree of goal achievement. This determines whether to continue adjusting the attributes and how much adjustment to make.

[0041] Based on fine attribute predictor In each round of editing, the edited attribute labels are predicted based on the latent vectors of the currently generated image. It also calculates the deviation between the current edit and the target attribute. (Discriminator) Through adversarial training and generators Together, we optimize the generator, pushing it to produce increasingly realistic images. Finally, the model provides feedback to the user, determining their next action.

[0042] Through the above editing process, the method achieves continuous, stable, and interpretable fine-grained editing of facial attributes in multi-turn interaction scenarios. The following sections detail the various component modules and the loss function within the model.

[0043] 1) Instruction comprehension and dialogue memory building module.

[0044] In text-driven face editing tasks, users often refine their editing goals through multiple rounds of natural language instructions. Therefore, the model not only needs to accurately parse the semantic information of a single-round instruction but also needs to maintain a consistent understanding of the editing context throughout continuous interactions. Currently, most face editing tools only operate on single editing instructions given by the user, lacking the storage and feedback of editing history and contextual information from multiple rounds of interactions. This makes it difficult to understand relative instructions or inherit previous editing states, which can easily lead to problems such as inconsistent editing or failure to meet user needs in continuous editing scenarios.

[0045] To this end, this invention introduces the concept of contextual semantic awareness and designs an instruction understanding and dialogue memory establishment module, which combines user instruction semantics with multi-round editing history to maintain the consistency and coherence of editing direction in subsequent latent space editing processes.

[0046] With the first Taking the editing process as an example, the user inputs natural language commands. After language encoder Get edit code .

[0047] (1) Mapped into a structured semantic representation: (2) in, Indicates the target editing attributes. For editing purposes, This indicates the intensity of editing or the relative range of change.

[0048] In multi-turn face editing, relying solely on the current instruction can easily lead to semantic inconsistencies or inconsistent editing across multiple interactions. To address this, this invention introduces a dialogue memory mechanism to uniformly manage the editing history and contextual information during multi-turn interactions. Specifically, in the first... Before the editing round begins, the model stores a dialogue memory state. This includes key information such as edited attributes, corresponding attribute changes, and the most recent system feedback result. The process of understanding the instruction occurs during the current input. With historical memory Under the combined conditions, when the current round of instructions lacks a clear attribute reference or editing range, the module will complete and correct it based on the memory state, thereby resolving relative or omitted instructions (such as "a little stronger" or "almost there") into semantic descriptions consistent with the global editing direction and degree. In specific implementation, the model first, based on the current attribute state and the editing history recorded in the dialogue memory, combined with the editing direction... and relative editing range Update the edit level of the target attribute and map it to a predefined discrete level range. To obtain a standardized target semantic text description This text description clearly expresses the target semantics expected to be achieved in the current round in a concise and stable form, and is used for subsequent cross-modal semantic consistency evaluation and adaptive control of the editing process.

[0049] In the current number After each round of editing is completed, the system integrates the newly generated editing results with the instruction parsing information into the memory state, forming an updated memory representation. To enhance the interpretability and interactive transparency of the model, the updated dialogue memory state is stored in a structured form as a context-aware multi-turn edit memory file. This memory file is used to show the user the current editing context and historical decision information, and also provides a basis for understanding relative instructions in subsequent turns.

[0050] Through the instruction understanding and dialogue memory building module, the model can maintain a full understanding of the user's editing intent in multi-turn dialogue scenarios and make full use of historical context information during continuous editing, thereby providing continuous, consistent and reliable editing control signals for the latent space semantic editing module.

[0051] 2) Latent space semantic editing module.

[0052] After obtaining the semantic encoding output by the instruction understanding and memory building module, this invention proposes a latent space semantic editing module to further achieve controllable editing consistent with semantics in the latent space of the generative model. This module aims to achieve continuous and controllable adjustment of facial attributes in the latent space of the face editing model. By learning the semantic rules of facial attribute changes in the latent space and inputting editing information from the instruction understanding and dialogue memory module during the latent space update process, this module obtains the update direction and magnitude of the latent vectors, thereby achieving fine-tuning of the degree of facial attributes.

[0053] First, given an input face image Through the image encoder Mapping this onto the latent space of the generative model yields the corresponding initial latent vector. , means as follows (3) This latent vector As the starting point for subsequent rounds of face editing, the face is progressively updated during the editing process to reflect changes in facial attributes. This editing process is carried out in the StyleGAN generative model, which uses a mapping network to transform the input latent vector into an intermediate latent space with clearer semantic representation and greater editability. ,by The latent vectors in the space are used as representations of the edit state. Let the input face image be at the _i ... The latent vector corresponding to the wheel is , Indicates and A real vector space of the same dimension.

[0054] Traditional latent space editing methods based on a fixed global direction edit attributes along the same direction, which can easily lead to over-editing, identity drift, or other attribute corruption. In reality, the optimal editing direction and step size may differ for different identities and initial attribute strengths. Therefore, this invention uses a learnable latent space field function. This enables it to adaptively generate displacements based on the latent vector of the current face, thereby achieving semantic editing related to the editing state. Specifically, taking the first... Taking round editing as an example, the field function uses the current latent vector As input, the output is a field vector with the same dimension as the latent space. It can be seen as a local "velocity" or "deformation direction" in the latent space that describes the trend of semantic change.

[0055] (4) By using a learnable latent space field function, the editing increment no longer depends on a predefined fixed linear direction, but is dynamically determined by the latent vector of the current sample, thus better adapting to different identities and initial attribute states.

[0056] In the specific implementation, to ensure that the output of the field function is consistent with the representation space of StyleGAN and to avoid the constant bias term in the generator interfering with the editing direction, this invention uses the field vector... The displacement is obtained by aligning the forward mapping of the generator and applying zero-input correction. ,make It is closer to "pure increments", thus making the editing direction more stable and interpretable.

[0057] (5) in This represents the forward mapping operation inside the generator. It is a zero vector.

[0058] Furthermore, considering that different generation layers of StyleGAN have significantly different effects on image semantics and details, the model utilizes... The space assigns independent latent vectors to each style layer to achieve hierarchical, fine-grained decoupling and controllable editing of facial semantics. Let the generator have a total of... An injectable layer, then This represents the sequence of latent vectors obtained layer by layer. If the current round has not yet formed an independent... The potential meaning is that the current situation is as follows. Latent vectors in space Based on this, each generation layer uses the same latent vector as its starting point, thus obtaining the initial... To achieve hierarchical control, only the front... The input to the generation layer is incremented, while the inputs to other layers remain unchanged. That is, latent vectors related to global semantic attributes (such as age, expression, and hairstyle) are adjusted first, while high-level latent vectors are kept unchanged to preserve high-frequency information such as skin texture, lighting details, and identity features. This is combined with the editing direction information provided by the preceding instruction parsing. For each layer The latent vector is updated according to the following formula: (6) And for all : (7) in, This is the adaptive step size coefficient.

[0059] No. Latent vectors obtained by round editing This will serve as input for the next round of editing. Through multiple rounds of interactive editing, latent space semantic editing is performed recursively, effectively reducing detail degradation and identity drift caused by over-editing. This results in more stable, natural editing results with better controllability and interpretability, achieving smooth and controllable progressive face editing.

[0060] 3) Edit the evaluation and control module.

[0061] In the latent space semantic editing module, the editing process is represented as a continuous process of progressively updating latent vectors in the latent space, with each edit adjusting the results based on the previous round. However, relying solely on a fixed editing step size or a preset number of editing rounds makes it difficult to achieve stable and consistent editing results under different samples and varying semantic intensity requirements.

[0062] To this end, the present invention introduces an editing evaluation and control module, which dynamically adjusts and terminates the latent space editing process through cross-modal semantic evaluation and adaptive control mechanisms, thereby achieving fine and controllable progressive semantic editing.

[0063] With the first Taking round editing as an example, the latent vectors obtained by the latent space semantic editing module are... After passing through the generator, the corresponding generated face image is obtained through mapping. .

[0064] (8) in, This is the generator for StyleGAN.

[0065] To evaluate the consistency between the current editing result and the target semantics, a cross-modal semantic evaluation model, CLIP, is introduced, mapping the generated image and the target semantic text description to a shared semantic embedding space. Specifically, this involves generating a face image. and the target semantic text determined by the instruction understanding and memory building module. Image encoders via CLIP With text encoder The corresponding feature representation is as follows: (9) (10) Cosine similarity is used to calculate the degree of matching between the current generated result and the target semantics, which reflects the semantic closeness of the current editing state to the target description: (11) in, Cosine similarity, with values ​​ranging from... .

[0066] Based on the semantic evaluation score, the editing control module adaptively adjusts the latent space update magnitude. Specifically, the next round of latent space update can be represented as: (12) in, Predicted from the latent space field function, This is the adaptive step size coefficient.

[0067] The step size coefficient is determined by the current semantic evaluation score, the user's expected relative editing intensity, and the historical editing status. Specifically, the model first evaluates the CLIP score (i.e., cosine similarity). Calculate the remaining gap between the current edit result and the target semantics, and compare it with the relative edit intensity obtained from instruction parsing. Combining these factors, initial candidate step sizes are obtained. Then, the model incorporates the step size information from the previous round in the historical editing state to apply a smooth constraint to the current step size, limiting it to a predefined range, thus obtaining the final editing step size. This process can be represented as: (13) in, For smoothing coefficients, This represents the upper and lower bounds of the step size.

[0068] By adopting an adaptive update step size strategy, when there is still a large gap between the current generated result and the target semantics, the model can automatically increase the editing step size to accelerate semantic changes; while when the semantic matching degree gradually improves, the editing step size will decrease accordingly, thereby effectively avoiding over-editing and update oscillations in the latent space.

[0069] Furthermore, the editing evaluation and control module can also introduce an automatic termination mechanism based on the CLIP semantic evaluation results to determine whether the editing process has reached semantic saturation or the user's expected level. When the conditions are met... or At that time, the editing process in the latent space will terminate. Here, A semantic convergence threshold is used to determine whether semantic changes have stabilized after several rounds of editing. The target semantic matching threshold is used to determine whether the current generated result has reached the expected semantic strength.

[0070] After each round of editing, the model generates corresponding natural language feedback, mainly including confirming whether the current attribute level meets expectations, providing optional subsequent editing suggestions, and asking the user for the next instruction. This effectively enhances the interactivity and interpretability of the editing process, forming a sustainable multi-turn dialogue process.

[0071] In summary, the editing evaluation and control module, by introducing CLIP-based semantic consistency evaluation, an adaptive update step size strategy, and an automatic termination mechanism, enables the model to provide real-time feedback and dynamic adjustments to the editing process in the latent space based on the current editing effect. This transforms the previously unconstrained multi-round editing process into a controllable process guided by semantic objectives. The model can more stably control the editing process, ensuring that the final generated image has high-quality visual effects.

[0072] 4) Overall loss function.

[0073] The training process of the model in this invention is constrained by four types of loss functions: attribute supervision loss, identity preservation loss, realism adversarial loss, and cross-modal semantic alignment loss. Each loss term jointly optimizes the latent space editing process from multiple perspectives, such as semantic consistency, visual realism, and identity stability, thereby guiding the model to maintain the naturalness and identity consistency of the generated results while achieving fine semantic editing.

[0074] (1) Attribute supervision loss.

[0075] Attribute-supervised loss is used to constrain the direction and magnitude of changes in the target attribute in the edited generated image. By imposing stronger constraints on the target attribute branch, it guides the model to achieve stable and controllable attribute editing along the expected semantic direction while keeping other attributes stable. To enable the mapping network to represent semantic fields, a pre-trained fine-grained attribute predictor is used. As a supervisory constraint, it guides the network to learn the direction of changes in location-related attributes in the latent space. Let the model contain... There are editable attributes, and the fine-grained attributes corresponding to the original image are denoted as follows: ,in These are discrete tags used to characterize the intensity level of an attribute. (The rest of the text appears to be a list of tags or labels, possibly related to a specific attribute or statistic.) Taking the first round of editing as an example, in the first round of editing... When performing semantic field learning on each attribute, the edited image will be... The target attribute tag is set to The model calculates the cross-entropy loss for each attribute branch separately, and assigns a greater weight to the currently edited target attribute, thereby guiding the model to focus on changes in that attribute. The attribute supervision loss function is specifically expressed as follows: (14) in, Indicates the strength level of the attribute. It is a binary indicator variable corresponding to the target attribute. It is a predictor The Softmax output, i.e. . yes The Middle The first attribute, the first The probability components corresponding to each intensity level.

[0076] (2) Loss of identity preservation.

[0077] To mitigate potential identity loss during semantic editing, this invention introduces an identity preservation loss. Given the first... Face image before editing in the round-editing process Edited image The model imposes a consistency constraint on the identity features of the images before and after editing. By minimizing this loss, the model can effectively maintain the stability of facial identity features while editing the target attributes. Specifically, it uses an existing pre-trained face recognition model to extract deep features and constrains the features extracted before and after image editing to keep them as consistent as possible. The identity preservation loss function is expressed as follows: (15) in, This is a network for extracting identity features for Arcface.

[0078] (3) Realism counteracts loss.

[0079] To improve the overall visual realism of the editing results, this invention uses adversarial realism loss to constrain the generated image to approximate a real human face image at a global level. Generator Using the previous latent vector Generate face image before editing generator Using the updated latent vector Generate edited face image Through the discriminator Distinguish between face images before editing Compared with edited face images Discriminator Distinguishing between edited face images Face image before editing The specific representation of the adversarial realism loss function is as follows: (16) in, This indicates the desired operation on the image.

[0080] (4) Cross-modal semantic alignment loss.

[0081] Cross-modal semantic alignment loss is one of the key constraints introduced in this invention, used to directly incorporate natural language semantics into the optimization objective of latent space editing. Specifically, based on the CLIP cross-modal model, for the... Image generated by wheel editing With target semantic text description Consistency in the semantic space is measured to improve the semantic controllability of text-driven editing. The cross-modal semantic alignment loss function is specifically expressed as follows: (17) In summary, the total loss function of the model in this invention is expressed as follows: (18) in, , , , These are the weighting factors for different losses.

[0082] Furthermore, to demonstrate the effectiveness and significance of the method provided by this invention, the following experiments were conducted in this embodiment: 1) Data set.

[0083] This invention is experimentally verified using the CelebA-Dialog dataset. This dataset provides continuous label partitioning for the attributes of each image, with some examples as follows: Figure 2 As shown, for attributes such as bangs, smile, glasses, beard, and age, the dataset labels whether these attributes exist and provides different severity labels (0-5) for each attribute, allowing for more detailed adjustments in facial editing. Additionally, each image in the CelebA-Dialog dataset is accompanied by a corresponding text description of the editing request. The experiment uses 25,000 face images and their corresponding attribute labels and text descriptions as the training set, and 5,000 face images and their corresponding attribute labels and text descriptions as the test set. There is no overlap between the training and test datasets to prevent any data leakage, and all images have a resolution of 128×128.

[0084] 2) Training strategies.

[0085] The machine configuration and development / running environment used in the experiments for this invention were: Ubuntu 20.04; NVIDIA A100 GPU; CUDA 11.0; Python 3.7. During training, the Adam optimizer was used, the learning rate was set to 0.001, the model was trained for 3000 iterations, and the batch size was set to 2048. After multiple experimental tests, the weight parameters were set to... , , , .

[0086] 3) Evaluation criteria.

[0087] This invention employs Identity Preservation, Irrelevant Attribute Preservation, and FID (Image Generation Quality Assessment) as quantitative evaluation metrics for text-driven face editing results. Identity Preservation measures the degree of perturbation to identity information during the editing process by extracting identity feature vectors from the face images before and after editing and calculating the Euclidean distance between them, thus determining whether a change in face identity has occurred. Irrelevant Attribute Preservation assesses whether the editing has caused undesirable changes to facial attributes other than the target attributes by predicting multiple attributes in the editing result and using cross-entropy scores to measure the consistency between the predicted attributes and the ground truth labels. FID measures the difference in distribution between the edited image and the ground truth face image in the deep feature space, statistically reflecting the degree to which the generated result deviates from the real data distribution and measuring the overall visual quality of the generated result.

[0088] 4) Experimental results diagram.

[0089] To visually demonstrate the editing process, Figure 3 Two examples are given to progressively enhance the attributes of a given face according to user instructions. The first and second lines respectively progressively enhance the smile and age according to user instructions. In these examples, the user will progressively adjust the smile and age in the face image using natural language commands. In each round of interaction, the corresponding attributes of the face image are adjusted according to the instructions.

[0090] 5) Quantitative comparison results.

[0091] To demonstrate the effectiveness and superiority of the proposed method in face editing, a quantitative comparison was conducted on the CelebA dataset with Enjoy your editing, Talk-to-edit, MM2Latent, HyperStyle, AdaTrans, and InstructPix2Pix methods. Table 1 shows the results of the quantitative comparison. It is evident that the proposed method achieves the best results in all three metrics, fully demonstrating its excellent identity preservation, irrelevant attribute preservation ability, and overall visual quality in face editing tasks. These superior results are due to the proposed model's ability to accurately understand relative semantic instructions in multi-turn interactions and stably update the latent space, thereby ensuring the continuity, controllability, and visual naturalness of the editing process.

[0092] Table 1. Quantitative evaluation results of different methods

[0093] 6) Visual comparison results.

[0094] To visually demonstrate the advantages of the method of this invention in multi-round continuous editing, a qualitative comparative experiment was conducted between the method of this invention and the aforementioned six methods. Since the MM2Latent, HyperStyle, and AdaTrans methods can only edit age and smile attributes, therefore... Figure 4 The results demonstrate the visual contrast of different methods progressively enhancing the age and smile attributes of a given face. Additionally, Figure 5 The invention's method is shown in comparison with Enjoy your editing, Talk-to-edit, and InstructPix2Pix for enhancing the attributes of bangs, glasses, and beard.

[0095] Depend on Figure 4 and Figure 5 It is evident that the MM2Latent and HyperStyle methods suffer from significant degradation in quality and distortion of details. Facial details (such as wrinkles and pores) are excessively erased, leading to decreased recognizability of facial features and affecting identity consistency. The Enjoy your editing and InstructPix2Pix methods produce good results in the first edit, but distortion and facial deformation occur in the second. Talk-to-edit and AdaTrans achieve good results in both edits, generally achieving natural editing effects in response to user commands, but they still have problems controlling irrelevant attributes, resulting in additional attribute changes in the results. Compared to these methods, the method of this invention achieves more stable and natural editing effects in multi-round face editing tasks. The synthesized image is smoother and more coherent during attribute changes, with a more realistic overall visual effect and no obvious loss of detail. When progressively enhancing target attributes, the image transitions naturally without over-editing. In terms of identity preservation, the method of this invention can better preserve the original facial structure and individual characteristics of a person while adjusting the target attributes. The model understands user instructions more accurately and can reasonably control the direction and degree of attribute changes according to semantic content, making the editing results closer to user expectations.

[0096] 7) Ablation experiment.

[0097] To verify the effectiveness of the dialogue memory mechanism, cross-modal semantic alignment, and adaptive control strategy in the method of this invention, a series of ablation experiments were conducted. The quantitative evaluation results and visual results of the experiments are shown in Table 2 and 2, respectively. Figure 6As shown, M1 removes the dialogue memory mechanism, directly inputting the edited encoding obtained from each round of instructions into the model without context memory management; M2 removes the cross-modal semantic alignment loss term, without aligning the generated image with the user's instructions; M3 removes the cross-modal semantic evaluation and adaptive control mechanism, allowing the model to update the latent space with a uniform step size; M4 is the complete model of this invention. The comparison results in Table 2 show that the complete model of this invention outperforms all other ablation models in various aspects.

[0098] Table 2 Quantitative evaluation results of ablation experiments

[0099] Figure 6 The first column shows the input image, the second column shows the result of the first round of editing "Add some smiles", and the third column shows the result of the second round of editing "More". Figure 6 As can be seen, M1 directly inputs the editing code corresponding to each round of instructions into the model. Due to the lack of a dialogue memory mechanism, it cannot effectively understand relative semantic instructions such as "a little stronger," making it difficult to achieve progressive editing in multi-round interactions. M2, lacking the constraint of cross-modal semantic alignment loss, suffers from irrelevant attribute interference during the editing process, resulting in a deviation between the generated result and the user's intent. M3, due to the lack of adaptive step size control, exhibits obvious over-editing during secondary editing, resulting in unnatural facial expressions and texture distortions. M4, the complete model of this invention containing all modules, can accurately understand relative instructions in multi-round interactions, maintain identity consistency and visual naturalness during the editing process, and generates results that best fit the text semantics.

[0100] This invention addresses the challenges of inheriting editing context and parsing relative semantics in multi-turn interactions, as well as over-editing and semantic oscillations during latent space updates. It proposes a text-guided, context-aware face editing model. While maintaining identity consistency and visual quality stability, the model effectively responds to the user's progressively refined editing needs across multiple turns of dialogue, enabling continuous and controllable adjustments to facial attributes. The model incorporates a dialogue memory-driven editing management mechanism based on contextual semantic awareness, which uniformly stores and updates historical editing trajectories, current semantic states, and editing context information. This allows the model to accurately understand relatively ambiguous instructions, ensuring consistent semantic understanding. The model utilizes cross-modal semantic alignment loss as the core optimization objective, guiding latent space updates towards semantically stable convergence. Combined with an adaptive step size adjustment strategy, it effectively avoids latent space oscillations and over-editing, achieving smooth and precise attribute control. Experimental results demonstrate that the proposed method outperforms existing mainstream methods in multi-turn text-driven face editing tasks in terms of semantic consistency, identity preservation, and visual quality, synthesizing higher-quality editing results that better meet user expectations.

[0101] Additionally, it should be noted that: Figure 1 The phrases "Make the person wear glasses" and "OK, it is done. Do you have any other requirements?" are examples of text instructions for editing facial images; similarly, Figure 2 The last column in the middle Figure 3 Both the Chinese and English versions are text-based instruction examples.

[0102] Example 2 This embodiment discloses a text-driven face image editing system.

[0103] A text-driven face image editing system includes: The instruction understanding and memory building module is configured to: acquire natural language editing instructions from the target face image, perform semantic parsing, and store the editing history and context information during multiple rounds of interaction through a dialogue semantic memory mechanism to generate editing codes; The latent space semantic editing module is configured to: perform semantic modeling of the facial attribute change process in the latent space of the generated model, and update the latent variables along a continuous trajectory consistent with the edited encoded semantics to regulate the attribute intensity; The editing evaluation and control module is configured to: evaluate the editing results based on cross-modal semantic alignment constraints, and adaptively adjust the latent space update step size based on the current editing state and the degree of goal achievement; and set an automatic termination mechanism based on the evaluation results to control the continuation and termination of the editing process. The fine-grained attribute predictor module is configured to guide the generative model to generate images using a constrained attribute supervised loss. The discriminator is configured to evaluate the realism of the generated image by constraining the realism to counteract the loss, in order to optimize the generator parameters and generate the final image.

[0104] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0105] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a text-driven face image editing method as described in Embodiment 1 of this disclosure.

[0106] Example 4 The purpose of this embodiment is to provide an electronic device.

[0107] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a text-driven face image editing method as described in Embodiment 1 of this disclosure.

[0108] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0109] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0110] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A text-driven face image editing method, characterized in that, include: Natural language editing instructions for the target face image are obtained, semantic parsing is performed, and editing history and context information from multiple rounds of interaction are stored through a dialogue semantic memory mechanism to generate editing codes; Semantic modeling of facial attribute changes is performed in the latent space of the generative model, and latent variables are updated along a continuous trajectory consistent with the edit-encoded semantics to regulate attribute strength. The editing results are evaluated based on cross-modal semantic alignment constraints, and the latent space update step size is adaptively adjusted based on the current editing state and the degree of goal achievement. An automatic termination mechanism is set up based on the evaluation results to control the continuation and termination of the editing process; By using attribute-supervised loss to guide the generative model in generating images, and by using adversarial loss to constrain realism, the realism of the generated images is evaluated, thereby optimizing the generator parameters and generating the final image.

2. The text-driven face image editing method as described in claim 1, characterized in that, The generation of the edit code includes: in the first round of editing, mapping the face image to the latent space of the generative model based on the CLIP image encoder to obtain the initial latent vector; at the same time, performing semantic parsing on the natural semantic instructions based on the CLIP language encoder to generate edit codes containing target attributes, change direction and degree information.

3. The text-driven face image editing method as described in claim 1, characterized in that, Based on the aforementioned dialogue semantic memory mechanism, the editing history and context information in the multi-round interaction process are managed in a unified manner. Specifically, the understanding process of the instruction is carried out under the joint conditions of the current input and the historical memory. That is, when the current round instruction lacks a clear attribute reference or editing scope, the current round instruction is completed and corrected based on the memory state, so as to parse the relative and ellipsis-like instructions into a semantic description consistent with the global editing direction and degree.

4. The text-driven face image editing method as described in claim 1, characterized in that, When semantically modeling the process of facial attribute changes in the latent space of the generative model, the initial latent vector is used as the starting point for subsequent rounds of facial editing, and the latent vector is updated during the editing process to reflect the changes in facial attributes; the entire editing process is carried out in the generative model based on StyleGAN.

5. The text-driven face image editing method as described in claim 4, characterized in that, A learnable latent space field function is introduced during the editing process to adaptively generate displacement based on the latent vector of the current face image, thereby achieving semantic editing related to the editing state. The field function takes the current latent vector as input and outputs a field vector with the same latent space dimension. The field vector is aligned by the forward mapping of the generator and the final displacement is obtained by zero-input correction.

6. The text-driven face image editing method as described in claim 1, characterized in that, The latent space update step size is adaptively adjusted based on the current editing state and the degree of goal achievement. This includes: first, calculating the remaining gap between the current editing result and the target semantics based on the CLIP evaluation score, and combining it with the relative editing intensity obtained from instruction parsing to obtain an initial step size candidate value; then, by introducing the step size information from the previous round in the historical editing state, the current step size is smoothed and constrained within a preset range to obtain the final editing step size.

7. The text-driven face image editing method as described in claim 1, characterized in that, The automatic termination mechanism includes: terminating the editing process in the latent space when either a first termination condition or a second termination condition is met; otherwise, the editing process continues; wherein the first termination condition and the second termination condition are respectively expressed as: ； ； in, This represents the semantic enhancement convergence threshold, used to determine whether semantic changes have stabilized after several rounds of editing. This represents the target semantic matching threshold, used to determine whether the current generated result has reached the expected semantic strength; Indicates the first The semantic similarity between the edit state after the round ends and the target description. Indicates the first The degree to which the initial editing state of the wheel is semantically close to the target description.

8. A text-driven face image editing system, characterized in that, include: The instruction understanding and memory building module is configured to: acquire natural language editing instructions from the target face image, perform semantic parsing, and store the editing history and context information during multiple rounds of interaction through a dialogue semantic memory mechanism to generate editing codes; The latent space semantic editing module is configured to: perform semantic modeling of the facial attribute change process in the latent space of the generated model, and update the latent variables along a continuous trajectory consistent with the edited encoded semantics to regulate the attribute intensity; The editing evaluation and control module is configured to: evaluate the editing results based on cross-modal semantic alignment constraints, and adaptively adjust the latent space update step size based on the current editing state and the degree of goal achievement; and set an automatic termination mechanism based on the evaluation results to control the continuation and termination of the editing process. The fine-grained attribute predictor module is configured to guide the generative model to generate images using a constrained attribute supervised loss. The discriminator is configured to evaluate the realism of the generated image by constraining the realism to counteract the loss, in order to optimize the generator parameters and generate the final image.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps of a text-driven face image editing method as described in any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the text-driven face image editing method as described in any one of claims 1-7.

Images