Multimodal Text-Guided Image Editing and Optimization Methods and Systems for AI Digital Humans

Abstract

This invention relates to the field of artificial intelligence technology and discloses a multimodal text-guided image editing optimization method and system for AI digital humans. The method includes the following steps: S100, initializing the source image through an identity and inverse attribute encoder, separating a fixed identity core vector and an initial set of editable attribute vectors; S200, extracting context from the evolutionary history using a path encoder, and combining this context with natural language instructions to generate a context-aware state transition operator; S300, applying the operator to update the main attributes, and then performing a secondary adjustment through an attribute constraint field to ensure logical consistency, generating a new set of attribute vectors; S400, fusing the fixed identity core vector and the new set of attribute vectors by an image decoder to synthesize and output the final image. This invention achieves coherent, logically consistent, and identity-consistent progressive image editing by introducing an evolutionary history tree, an attribute constraint field, and an identity attribute separation mechanism.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal text-guided image editing optimization method and system for AI digital humans. Background Technology

[0002] With the rapid development of artificial intelligence technology, digital humans (or virtual humans, AI digital humans) are increasingly widely used in media, entertainment, customer service, and other fields as a new type of human-computer interaction medium and digital content carrier. To enhance the expressiveness and realism of digital humans, their appearance, especially facial images, needs to be edited precisely, efficiently, and in accordance with user intent. Directly editing digital human images using natural language commands has become an important technological direction in this field due to its intuitiveness and convenience.

[0003] Currently, mainstream technologies for achieving this goal are typically built upon deep generative models, particularly those represented by StyleGAN (Generative Adversarial Networks). The general process of such technologies is as follows: First, through a process called "GAN inversion," a given digital human source image is mapped (or encoded) into a specific latent vector in the latent space of the generative model. This latent vector can be understood as a condensed digital description of the source image. Then, when receiving a user's natural language editing instruction (such as "make him smile"), the system uses a large-scale vision-language pre-trained model (such as CLIP) to interpret the semantics of the text instruction. This vision-language model guides the system to find a specific direction vector in the latent space, which is considered to correspond to the semantics of the text instruction ("smile"). Finally, by performing vector operations on this direction vector and the original latent vector, a new latent vector is obtained and input into the generator, thereby synthesizing a new, edited image.

[0004] While existing technologies have achieved semantic editing of digital human images to some extent by mapping text instructions to editing directions in the latent space of the generative model, some shortcomings remain. First, this paradigm treats each edit as an independent, stateless transformation process. Its optimization objective focuses solely on the matching degree between the current image and the current text instruction, completely ignoring the contextual position of that edit within the entire editing sequence. This design stems from the model's lack of a memory mechanism, making it unable to understand the cumulative nature of editing operations.

[0005] Secondly, there is a high degree of entanglement between the latent space dimensions of generative models. An edit direction identified as corresponding to a specific semantic meaning (such as "increased age") often inevitably carries changes in other attributes in the dataset that are statistically related to that semantic meaning (such as "changes in hair color" or "changes in posture"). The fundamental reason is that the model lacks prior knowledge of human facial structure and attribute logic; it can only reproduce the statistical correlations in the training data and cannot effectively decouple the main instruction from accompanying effects. Therefore, when performing a targeted edit, it often triggers a series of unexpected and illogical attribute changes, thereby compromising the realism of the image and the accuracy of the edit.

[0006] Finally, most editing methods modify the same latent vector directly, which typically encodes both the person's identity and variable attributes. This hybrid encoding of identity and attributes means that each vector stacking operation aimed at modifying attributes causes the latent vector to deviate from its original position. Although mathematical regularization can limit the magnitude of this deviation, the identity information is gradually eroded with each edit, eventually leading to inconsistencies between the person in the composite image and the source image—in other words, identity consistency is difficult to guarantee. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a multimodal text-guided image editing optimization method and system for AI digital humans. It solves the problems of inconsistent editing processes, unintended changes caused by attribute modifications, and difficulty in maintaining the consistency of human identity when performing multi-step text-guided image editing due to the stateless editing process, highly entangled attributes, and mixed encoding of identity and attributes.

[0008] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a multimodal text-guided image editing optimization method for AI digital humans. This method constructs the image editing process as a temporal evolution process in a structured semantic space, thereby achieving strong constraints on the identity characteristics of digital humans, effective utilization of editing history, and logical self-consistency adjustment of the edited content.

[0009] This invention provides a multimodal text-guided image editing optimization method for AI digital humans, comprising the following steps:

[0010] S100. Obtain the source image of the AI ​​digital human, and initialize the source image into a semantic base state using an identity encoder and an inverse attribute encoder. The semantic base state includes an identity core vector, an initial set of editable attribute vectors, and an initial structure template. The identity core vector is used to represent the inherent identity information of the AI ​​digital human.

[0011] S200: Extract context information from evolutionary history data that records historical editing operations based on the path encoder and generate a path context vector; receive and encode the natural language editing instructions input by the user in real time; generate a state transition operator based on the path context vector, the encoded natural language editing instructions, and the current set of editable attribute vectors.

[0012] S300. Based on the state transition operator, the main attribute of the current editable attribute vector set is updated to obtain a temporary attribute set. The temporary attribute set is then adjusted a second time through a preset attribute constraint field to generate a new editable attribute vector set, thereby completing the evolution of the semantic state.

[0013] S400. Based on the image decoder, the identity core vector and the new editable attribute vector set are synthesized and the edited multimodal text guidance image is output.

[0014] Preferably, in step S100, the step of initializing the source image to a semantic base state using an identity encoder and an inverse attribute encoder includes:

[0015] The resolution of the source image is normalized using an identity encoder, and the identity features of the AI ​​digital human are extracted from the normalized source image. The core identity vector is then generated based on the extracted identity features.

[0016] The initial set of editable attribute vectors is decoupled from the source image using an inverse attribute encoder, wherein the set of editable attribute vectors includes expression, lighting, pose, and image details, and the image details include skin, hair, eyes, and lips;

[0017] The structure extraction network is used to generate an initial structure template based on the source image, and the semantic base state is generated based on the identity core vector, the initial set of editable attribute vectors, and the initial structure template.

[0018] Preferably, in step S200, the step of extracting context information from the evolutionary history data recording historical editing operations and generating a path context vector based on the path encoder further includes: encoding the sequence of editing operations from the root node to the current node in the evolutionary history tree using the path encoder to obtain the path context vector v. path The evolutionary history tree is used to store evolutionary history data that records historical editing operations.

[0019] Preferably, in step S200, the step of generating a state transition operator based on the path context vector, the encoded natural language editing instructions, and the current set of editable attribute vectors further includes:

[0020] The current editable attribute vector set is generated based on the encoded natural language editing instructions and the initial editable attribute vector set, wherein the current editable attribute vector set stores the updated values ​​of expression, lighting, pose and image details;

[0021] The current editable attribute vector set, the encoded natural language editing instructions, the current editable attribute vector set, and the path context vector are input into the state transition operator generator network. Based on the state transition operator generator network and the attribute vector set calculation formula, the editable attribute vector set is modified, and the state transition operator is output. The state transition operator defines the amount of modification ΔA made by the user to the editable attribute vector set. n The formula for calculating the attribute vector set is:

[0022] ΔA n =F op (A n ,e text ,v path );

[0023] In the formula, ΔA n The amount of user-defined modifications to the editable attribute vector set for the state transition operator; F op For the state transition operator generator network; A n For the current set of editable attribute vectors; e text These are encoded natural language editing instructions; v path This is the path context vector.

[0024] Preferably, all of the evolutionary history data is recorded in an evolutionary history tree;

[0025] After each evolution of the semantic state in step S300 is completed, the newly generated semantic state is synchronously stored as a new node in the evolution history tree, and a parent-child connection is established between it and the semantic state node before evolution. The evolution history tree is composed of semantic state nodes and the connection relationships between semantic state nodes, and the evolution history data is specifically stored in the semantic state nodes.

[0026] Preferably, in step S300, the step of updating the main attributes of the current editable attribute vector set based on the state transition operator includes:

[0027] Based on the vector superposition calculation formula, the modification amount ΔA n With the current editable attribute vector set A n Perform vector superposition to obtain the temporary attribute set A′. n+1The formula for calculating the vector superposition is as follows:

[0028] A′ n+1 =A n +ΔA n ;

[0029] In the formula, A′ n+1 For the temporary attribute set; A n For the current set of editable attribute vectors; ΔA n The amount of modification is mentioned.

[0030] Preferably, in step S300, the step of performing a secondary adjustment on the temporary attribute set through a preset attribute constraint field to generate a new editable attribute vector set includes:

[0031] The temporary attribute set is input into the preset attribute constraint field, which calculates and outputs the quadratic attribute adjustment amount δA based on the attribute association rules of the editable attribute vector set. n+1 ;

[0032] Based on the vector set superposition formula, the quadratic attribute adjustment amount and the current editable attribute vector set are superimposed onto the primary temporary attribute set to obtain the new editable attribute vector set A. n+1 The formula for superimposing the vector sets is:

[0033] A n+1 =(A n +ΔA n )+δA n+1 ;

[0034] In the formula, A n For the current set of editable attribute vectors; ΔA n A is the amount of user-defined modification to the editable attribute vector set for the state transition operator; n+1 For the new editable attribute vector set; δA n+1 This is the adjustment amount for the secondary attribute.

[0035] Preferably, the preset attribute constraint field is a graph neural network model, where the nodes built into the graph neural network model represent different editable attributes, and the edges between the nodes represent the logical correlation strength between the attributes.

[0036] Preferably, in step S400, the step of synthesizing the identity core vector and the new editable attribute vector set based on the image decoder and outputting the edited multimodal text guidance image includes:

[0037] The identity core vector, the new editable attribute vector set, and the initial structure template corresponding to the new editable attribute vector set are input into the image decoder for synthesis and rendering to generate and output the edited multimodal text guidance image.

[0038] Accordingly, the present invention also provides a multimodal text-guided image editing and optimization system for AI digital humans, the system comprising a ground state construction module, a state transition module, a main attribute update module, and an image synthesis module;

[0039] in,

[0040] The ground state construction module is used to acquire the source image of the AI ​​digital human and initialize the source image into a semantic ground state using an identity encoder and an inverse attribute encoder. The semantic ground state includes an identity core vector, an initial set of editable attribute vectors, and an initial structure template. The identity core vector is used to represent the inherent identity information of the AI ​​digital human.

[0041] The state transition module is used to extract context information from evolutionary history data that records historical editing operations based on the path encoder and generate a path context vector, receive and encode the natural language editing instructions input by the user in real time, and generate a state transition operator based on the path context vector, the encoded natural language editing instructions, and the current set of editable attribute vectors.

[0042] The main attribute update module is used to update the main attributes of the current editable attribute vector set based on the state transition operator to obtain a temporary attribute set, and then adjust the temporary attribute set a second time through a preset attribute constraint field to generate a new editable attribute vector set, thereby completing the evolution of semantic state.

[0043] The image synthesis module is used to synthesize the identity core vector and the new editable attribute vector set based on the image decoder and output the edited multimodal text guidance image.

[0044] This invention provides a multimodal text-guided image editing optimization method and system for AI digital humans.

[0045] It has the following beneficial effects:

[0046] 1. This invention extracts a core identity vector that remains unchanged during the editing process during the initialization stage and uses this vector as input in the final image synthesis step. This ensures that no matter how much the editable attribute vector set is modified, the final synthesized digital human image can maintain the core identity features consistent with the source image, effectively avoiding the identity drift problem caused by feature accumulation changes during continuous editing.

[0047] 2. This invention records the evolutionary history and extracts path context vectors from it to generate context-aware state transition operators. This feature allows the system to refer to previous editing operation sequences when processing new natural language editing instructions. Therefore, editing behavior is no longer isolated but has temporal coherence, thereby improving the accuracy of understanding continuous editing intentions and generating more logically coherent editing results.

[0048] 3. After updating the main attribute, this invention further adjusts it through a preset attribute constraint field. This allows the invention to automatically fine-tune other related attributes based on preset attribute association rules after a single attribute is modified. This ensures that the newly generated editable attribute vector set is logically self-consistent, avoiding illogical combinations between different attributes such as hairstyle, makeup, and clothing, thus improving the overall realism and harmony of the final synthesized image. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the process of the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of the present invention;

[0051] Figure 3 This is a flowchart illustrating the structured semantic ground state initialization process of the present invention;

[0052] Figure 4 This is a flowchart illustrating the process of generating the context-aware state transition operator of the present invention.

[0053] Figure 5 This is a flowchart illustrating the semantic state evolution process of the present invention;

[0054] Figure 6 This is a flowchart illustrating the final image synthesis process of the present invention. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Please see the appendix Figure 1 , Figure 1 This is a flowchart of a multimodal text-guided image editing optimization method for AI digital humans according to an embodiment of the present invention. The method may include the following steps:

[0057] Step S100: Obtain the source image of the AI ​​digital human and initialize it as a structured semantic base state. This semantic base state is a data structure composed of multiple data components, containing all the necessary information for subsequent editing. In one embodiment, this step is specifically accomplished by three parallel network modules: using an identity encoder to extract an identity core vector from the source image; using an inverse attribute encoder to decouple an initial set of editable attribute vectors from the source image; and using a structure extraction network to generate an initial structure template from the source image. The identity core vector represents the inherent identity information of the digital human that should not be changed during editing. The initial set of editable attribute vectors corresponds to multiple editable semantic dimensions in the source image, such as hairstyle, expression, and makeup.

[0058] Step S200: Obtain natural language editing instructions, and based on the context information extracted from the evolutionary history that records historical editing operations, combine the natural language editing instructions to generate a context-aware state transition operator. This evolutionary history is stored in an evolutionary history tree, where each node represents a historical semantic state.

[0059] First, a path encoder is used to encode the sequence of edit operations from the root node to the current node in the evolutionary history tree, resulting in a path context vector v. path Subsequently, the current editable attribute vector set A is... n Natural language editing instructions e after text encoder processing text and path context vector v path A common input is fed into a state transition operator generator network F op The network outputs a modification A. n This modification is the state transition operator. Its calculation method is as follows:

[0060] ΔA n =F op (A n ,e text ,v path );

[0061] In the formula, ΔA n For the amount of modification; F op For a state transition operator generator network; A n For the current set of editable attribute vectors; e text Encoded natural language editing instructions; v path This is the path context vector.

[0062] Step S300: Apply the state transition operator to the current editable attribute vector set to update the main attribute, and then perform a secondary adjustment on the updated main attribute result through a preset attribute constraint field to generate a new internal editable attribute vector set, thereby completing the semantic state evolution. In one embodiment, this step is decomposed into two sub-steps. First, perform the main attribute update, i.e., the modification amount ΔA generated in step S200. n With the current editable attribute vector set A n Perform vector overlay to obtain a temporary attribute set A′. n+1 The calculation method is as follows:

[0063] A′ n+1 =A n +ΔA n ;

[0064] In the formula, A′ n+1 A is a temporary set of attributes; n For the current set of editable attribute vectors; ΔA n This refers to the amount of modification.

[0065] Next, for this temporary attribute set A′ n+1 Make a second adjustment. Adjust A′ n+1 The input is fed into an attribute constraint field, which calculates and outputs a quadratic attribute adjustment δA based on internally defined association rules between attributes (in one embodiment, the constraint field is a graph neural network model). n+1 .

[0066] Finally, this secondary attribute adjustment is superimposed onto the temporary attribute set to obtain the final, internally harmonious new editable attribute vector set A. n+1 The calculation method is as follows:

[0067] A n+1 =(A n +ΔA n )+δA n+1 ;

[0068] In the formula, A n The set of editable attribute vectors before the update; ΔA n The amount of modification corresponding to updating the primary attribute; A n+1 For a new set of editable attribute vectors; δA n+1 This is the amount of adjustment for secondary attributes.

[0069] Step S400: Based on the invariant identity core vector and the new editable attribute vector set, the final edited image is synthesized. This step is accomplished by an image decoder that receives the identity core vector that remains constant throughout the editing process and the new editable attribute vector set A generated in step S300. n+1 The system takes a structural template corresponding to the new state as input, and reconstructs and outputs a high-fidelity digital human image that reflects the editing intent and maintains identity consistency based on the above information.

[0070] Please see the appendix Figure 2 , Figure 2 This is a schematic diagram of a multimodal text-guided image editing optimization system for digital humans according to an embodiment of the present invention. The system may include:

[0071] The ground state construction module 10 is used to execute step S100 of the aforementioned method, acquire the source image of the AI ​​digital human, and perform the initialization operation of the structured semantic ground state. In one embodiment, this module integrates an identity encoder, an inverse attribute encoder, and a structure extraction network. Upon receiving the source image, the module processes the image in parallel and outputs three data components: an identity core vector, an initial editable attribute vector set A0, and an initial structure template. The module passes the identity core vector to the image synthesis module 50 and passes the initial editable attribute vector set A0 as the starting point for editing to the state transition module 20.

[0072] The state transition module 20 executes step S200 of the aforementioned method, obtains natural language editing instructions, and reads the evolutionary history tree from an external or internal memory. This module also obtains the current attribute vector set A from the subsequent secondary adjustment module 40 or the main attribute update module 30. n .

[0073] In one embodiment, this module integrates a path encoder, a text encoder, and a state transition operator generator network. After processing the above inputs, the module calculates and outputs a state transition operator, i.e., the modification amount ΔA. n And pass it to the main attribute update module 30.

[0074] The main attribute update module 30 is used to execute the main attribute update part of step S300 in the aforementioned method, and receives the modification amount ΔA from the state transition module 20. n and the current attribute vector set A n And a temporary attribute set A′ is calculated through vector superposition operation. n+1 The temporary attribute set A′ n+1 It is immediately passed to the secondary adjustment module 40 for further processing.

[0075] Secondary adjustment module 40 is used to perform the secondary adjustment part of step S300 in the aforementioned method. This module receives a temporary attribute set A′ from the main attribute update module 30. n+1 .

[0076] In one embodiment, this module internally implements a graph neural network as an attribute constraint field. This module utilizes this network to process A′. n+1 Calculate a quadratic attribute adjustment amount δA n+1 and compare it with A′ n+1 The data is superimposed to generate the final, new, editable attribute vector set A. n+1 The new vector set A n+1 On one hand, it is passed to the image synthesis module 50 to generate images, and on the other hand, it is used to update the evolutionary history tree as the "current state" for the next edit.

[0077] Image synthesis module 50, used to perform step S400 in the aforementioned method, receives three inputs: an identity core vector received from and maintained by the ground state construction module 10; the latest editable attribute vector set A received from the secondary adjustment module 40; and an identity core vector vector received from the ground state construction module 10. n+1 ; and a corresponding structural template.

[0078] In one embodiment, this module is an image decoder network. This module integrates all the input information, synthesizes and outputs a final edited image, thus completing a full editing process.

[0079] Please see the appendix Figure 3 , Figure 3 This is a schematic diagram of the structured semantic base state initialization process according to an embodiment of the present invention. This process corresponds to step S100 in the aforementioned method flow and is executed by the base state construction module 10. The input of this step is a source image of an AI digital human, and the output is a structured semantic base state. This semantic base state consists of three independent data components: an identity core vector, an initial set of editable attribute vectors, and an initial structural template, providing complete and decoupled initial conditions for subsequent editing operations.

[0080] In one specific embodiment, the extraction of the identity core vector is accomplished through an identity encoder. This identity encoder can be a pre-trained face recognition network, such as a network based on a deep residual network or a variant thereof. This network receives the source image as input and outputs a fixed-dimensional feature vector. This vector specifically represents the high-level, pose-, illumination-, and expression-invariant identity information of the digital human face. This identity core vector is calculated once during the initialization step and stored, remaining constant throughout all subsequent editing and synthesis processes, thus serving as a technical guarantee for maintaining the consistency of the digital human's identity.

[0081] The decoupling of the initial set of editable attribute vectors is accomplished through an inverse attribute encoder. This encoder maps the input source image into the latent space of a pre-defined generative model. In one embodiment, the encoder may employ a pixel-to-style-to-pixel or encoder-for-an-encoder architecture, aiming to transform the source image into the W+ latent space of a pre-trained stylized generative adversarial network. The output of this operation is a set of vectors, namely the initial set of editable attribute vectors A0.

[0082] The different vectors in this set correspond to the inputs of different levels of the StyleGAN generator, thus controlling different semantic attributes of the image, such as coarse-grained pose and face shape, medium-grained facial features, and fine-grained skin texture and lighting. This vector set A0 serves as the starting state for editing and will be modified in subsequent steps.

[0083] The initial structural template is generated through a structural extraction network. In one embodiment, this network can be a specialized model for human pose estimation or facial keypoint detection, such as the OpenPose network model or other similar convolutional neural networks. After processing the source image, the network outputs data containing spatial structural information of the digital human.

[0084] This data can be a set of two-dimensional or three-dimensional coordinate points that describe the location of key points on the digital human face (such as the corners of the eyes, the tip of the nose, and the corners of the mouth) or major joints of the body. This initial structural template provides a clear geometric constraint for the subsequent image synthesis step S400 to ensure that the structure of the person in the synthesized image remains consistent with the source image.

[0085] Please see the appendix Figure 4 , Figure 4 This is a schematic diagram of a context-aware state transition operator generation process according to an embodiment of the present invention. This process corresponds to step S200 in the aforementioned method flow and is executed by the state transition module 20. The core function of this step is to generate an instruction vector, i.e., a state transition operator, for modifying the current semantic state based on the current natural language editing instruction and in combination with the historical editing context.

[0086] In one specific embodiment, the evolutionary history is explicitly recorded in a tree-like data structure, namely an evolutionary history tree. Each node in the tree represents a defined semantic state and stores the complete set of editable attribute vectors corresponding to that state. The edges of the tree represent a state evolution event, connecting the parent node before evolution to the child node after evolution. This data structure records every edit branch and evolution from the initial state, providing a data foundation for extracting contextual information.

[0087] The process of extracting contextual information is as follows: When a new edit is needed, the node containing the current semantic state is first located in the evolutionary history tree. Then, a unique edit path is determined from the root node (i.e., the initial semantic base state) to the current node. The edit operation sequence corresponding to the node sequence along this path is extracted.

[0088] Subsequently, a path encoder receives the edit operation sequence as input. In one embodiment, the path encoder is a gated recurrent unit (GRU) network. This network processes the input edit operation sequence sequentially over time steps and ultimately outputs a fixed-dimensional vector, which is the path context vector v. path This constitutes a semantic summary of the entire editing history.

[0089] The generation of state transition operators is accomplished by a state transition operator generator network. In one embodiment, this network is a multilayer perceptron (MLP). The input to this network consists of three parts of data concatenated: the current set of editable attribute vectors A. n The text feature vector e is obtained by processing the current natural language editing instructions by a text encoder (such as the text encoding part of the CLIP model). text ; and the path context vector v generated in the preceding steps path These three vectors are concatenated into a longer feature vector.

[0090] The concatenated long feature vector is input into a multilayer perceptron, where it is processed through multiple fully connected layers and nonlinear activation functions. The output layer of the multilayer perceptron outputs a vector with the same dimension as the editable attribute vector set; this vector is the state transition operator ΔA, which defines the specific modification amount. n .

[0091] Please see the appendix Figure 5 , Figure 5 This is a schematic diagram of a semantic state evolution process according to an embodiment of the present invention. This process corresponds to step S300 in the aforementioned method flow, and its execution involves the main attribute update module 30 and the secondary adjustment module 40. This step receives the state transition operator ΔA generated in step S200. n and the current attribute vector set A n Output a new, editable attribute vector set A that has evolved and is internally self-consistent. n+1 .

[0092] The evolution of this semantic state specifically includes two sequentially executed stages. The first stage is the main attribute update, executed by the main attribute update module 30. This module updates the state transition operator ΔA. nThe defined modification amount is applied unbiasedly to the current editable attribute vector set A. n .

[0093] In one specific embodiment, the application process is a direct vector superposition operation, the output of which is a temporary attribute set A′. n+1 It accurately reflects the main attribute changes required by natural language editing instructions.

[0094] The second stage is a secondary adjustment based on the attribute constraint field, executed by the secondary adjustment module 40. The input to this stage is the temporary attribute set A′ generated in the previous stage. n+1 In one embodiment, the attribute constraint field is a pre-trained graph neural network (GNN). The topology of this GNN is designed to correspond to the semantics of the editable attributes: each node in the graph represents a specific editable attribute (e.g., "hair color," "lip thickness," "smile intensity," etc.), while the edges between nodes and their weights define the pre-defined logical correlation strength between these attributes. For example, there is a strongly positively correlated edge weight between the "smile intensity" attribute node and the "upturned corners of the mouth" attribute node.

[0095] When the temporary attribute set A′ n+1 After being input into the graph neural network, the network performs calculations through its internal message-passing mechanism. If A′ n+1 If a node's attribute value changes significantly, this change propagates to its neighboring nodes through weighted edges, allowing us to calculate the necessary attribute adjustments for these neighboring nodes to maintain logical consistency. The network ultimately integrates the calculation results from all nodes and outputs a quadratic attribute adjustment δA. n+1 For example, if the main attribute update significantly enhances the "eye opening level" attribute, the network will output a δA. n+1 The vector will then fine-tune the values ​​of related attributes such as "eyebrows raised" and "eyelid shape".

[0096] Finally, the adjustment amount δA of the quadratic attribute n+1 Superimposed on the temporary attribute set A′ n+1 To obtain the final, new editable attribute vector set A. n+1 .

[0097] Please see the appendix Figure 6 , Figure 6 This is a schematic diagram of the final image synthesis process according to an embodiment of the present invention. This process corresponds to step S400 in the aforementioned method flow and is executed by the image synthesis module 50. The purpose of this step is to reconstruct and output a final, high-fidelity digital human image based on the determined semantic state (defined by identity and attribute vectors) and structural information.

[0098] In one specific embodiment, the image synthesis function is implemented by an image decoder network. This network can be the generator part of a pre-trained StyleGAN (Generative Adversarial Network). This generator network has the ability to generate realistic images from abstract latent space vectors. The input to the decoder consists of three independent data components: an identity core vector extracted in step S100 and subsequently kept constant; and a new set of editable attribute vectors A ultimately generated in step S300. n+1 ; and a structure template corresponding to the new set of editable attribute vectors.

[0099] These three input components are utilized differently within the image decoder to collectively determine the content of the output image. A new set of editable attribute vectors, A. n+1 (Its data form is the W+ space vector of StyleGAN) is directly input into each layer of the generator network to modulate the weights of the convolutional layers, thereby precisely controlling the style, texture, lighting, color and various high-level semantic attributes of the generated image.

[0100] An invariant identity core vector is used to constrain the identity of the generation process. In one embodiment, this identity vector can be fine-tuned by applying a small adapter network to the intermediate feature maps of the generator network to ensure that the generated face is consistent in identity with the source image.

[0101] Structural templates are used to provide geometric and spatial constraints on the generated images. For example, the key point location information defined in the structural template can be converted into a spatial attention map. This attention map is applied to the feature maps of specific layers of the generator network, guiding the network to generate the corresponding facial organs or body parts in the correct locations, ensuring that the pose and structure of the person in the final image meet expectations.

[0102] Through the above mechanism, the image decoder effectively integrates identity information, semantic attribute information, and structural information, and finally synthesizes and outputs an edited image.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal text-guided image editing optimization method for AI digital humans, characterized in that, Includes the following steps: S100. Obtain the source image of the AI ​​digital human, and initialize the source image into a semantic base state using an identity encoder and an inverse attribute encoder. The semantic base state includes an identity core vector, an initial set of editable attribute vectors, and an initial structure template. The identity core vector is used to represent the inherent identity information of the AI ​​digital human. S200: Extract context information from evolutionary history data that records historical editing operations based on the path encoder and generate a path context vector; receive and encode the natural language editing instructions input by the user in real time; generate a state transition operator based on the path context vector, the encoded natural language editing instructions, and the current set of editable attribute vectors. S300. Based on the state transition operator, the main attribute of the current editable attribute vector set is updated to obtain a temporary attribute set. The temporary attribute set is then adjusted a second time through a preset attribute constraint field to generate a new editable attribute vector set, thereby completing the evolution of the semantic state. S400. Based on the image decoder, the identity core vector and the new editable attribute vector set are synthesized, and the edited multimodal text guidance image is output.

2. The multimodal text-guided image editing optimization method for AI digital humans according to claim 1, characterized in that, In step S100, the step of initializing the source image to a semantic base state using an identity encoder and an inverse attribute encoder includes: The resolution of the source image is normalized using an identity encoder, and the identity features of the AI ​​digital human are extracted from the normalized source image. The core identity vector is then generated based on the extracted identity features. The initial set of editable attribute vectors is decoupled from the source image using an inverse attribute encoder, wherein the set of editable attribute vectors includes expression, lighting, pose, and image details, and the image details include skin, hair, eyes, and lips; The structure extraction network is used to generate an initial structure template based on the source image, and the semantic base state is generated based on the identity core vector, the initial set of editable attribute vectors, and the initial structure template.

3. The multimodal text-guided image editing optimization method for AI digital humans according to claim 2, characterized in that, In step S200, the step of extracting context information from the evolutionary history data that records historical editing operations and generating a path context vector based on the path encoder further includes: encoding the sequence of editing operations from the root node to the current node in the evolutionary history tree using the path encoder to obtain the path context vector v. path The evolutionary history tree is used to store evolutionary history data that records historical editing operations.

4. The multimodal text-guided image editing optimization method for AI digital humans according to claim 3, characterized in that, In step S200, the step of generating a state transition operator based on the path context vector, the encoded natural language editing instructions, and the current set of editable attribute vectors further includes: The current editable attribute vector set is generated based on the encoded natural language editing instructions and the initial editable attribute vector set, wherein the current editable attribute vector set stores the updated values ​​of expression, lighting, pose and image details; The current editable attribute vector set, the encoded natural language editing instructions, the current editable attribute vector set, and the path context vector are input into the state transition operator generator network. Based on the state transition operator generator network and the attribute vector set calculation formula, the editable attribute vector set is modified, and the state transition operator is output. The state transition operator defines the amount of modification ΔA made by the user to the editable attribute vector set. n The formula for calculating the attribute vector set is: ΔA n =F op (A n ,e text ,v path ); In the formula, ΔA n The amount of user-defined modifications to the editable attribute vector set for the state transition operator; F op For the state transition operator generator network; A n For the current set of editable attribute vectors; e text These are encoded natural language editing instructions; v path This is the path context vector.

5. The multimodal text-guided image editing optimization method for AI digital humans according to claim 3, characterized in that, All of the aforementioned evolutionary history data were recorded in the evolutionary history tree; After each evolution of the semantic state in step S300 is completed, the newly generated semantic state is synchronously stored as a new node in the evolution history tree, and a parent-child connection is established between it and the semantic state node before evolution. The evolution history tree is composed of semantic state nodes and the connection relationships between semantic state nodes, and the evolution history data is specifically stored in the semantic state nodes.

6. The multimodal text-guided image editing optimization method for AI digital humans according to claim 4, characterized in that, In step S300, the step of updating the main attributes of the current editable attribute vector set based on the state transition operator includes: Based on the vector superposition calculation formula, the modification amount ΔA n With the current editable attribute vector set A n Perform vector superposition to obtain the temporary attribute set A′. n+1 The formula for calculating the vector superposition is as follows: IN' n+1 = Yes n +ΔA n ; In the formula, A′ n+1 For the temporary attribute set; A n For the current set of editable attribute vectors; ΔA n The amount of modification is [amount].

7. The multimodal text-guided image editing optimization method for AI digital humans according to claim 1, characterized in that, In step S300, the step of performing a secondary adjustment on the temporary attribute set using a preset attribute constraint field to generate a new editable attribute vector set includes: The temporary attribute set is input into the preset attribute constraint field, which calculates and outputs the quadratic attribute adjustment amount δA based on the attribute association rules of the editable attribute vector set. n+1 ; Based on the vector set superposition formula, the quadratic attribute adjustment amount and the current editable attribute vector set are superimposed onto the primary temporary attribute set to obtain the new editable attribute vector set A. n+1 The formula for superimposing the vector sets is: A n+1 =(A n +ΔA n )+δA n+1 ; In the formula, A n For the current set of editable attribute vectors; ΔA n A is the amount of user-defined modification to the editable attribute vector set for the state transition operator; n+1 For the new editable attribute vector set; δA n+1 This is the adjustment amount for the secondary attribute.

8. The multimodal text-guided image editing optimization method for AI digital humans according to claim 1, characterized in that, The preset attribute constraint field is a graph neural network model. The nodes built into the graph neural network model represent different editable attributes, and the edges between the nodes represent the logical correlation strength between the attributes.

9. The multimodal text-guided image editing optimization method for AI digital humans according to claim 1, characterized in that, In step S400, the step of synthesizing the identity core vector and the new editable attribute vector set based on the image decoder and outputting the edited multimodal text guidance image includes: The identity core vector, the new editable attribute vector set, and the initial structure template corresponding to the new editable attribute vector set are input into the image decoder for synthesis and rendering to generate and output the edited multimodal text guidance image.

10. A system for implementing the multimodal text-guided image editing optimization method for AI digital humans as described in claim 1, characterized in that, The system includes a ground state construction module, a state transition module, a main attribute update module, and an image synthesis module; in, The ground state construction module is used to acquire the source image of the AI ​​digital human and initialize the source image into a semantic ground state using an identity encoder and an inverse attribute encoder. The semantic ground state includes an identity core vector, an initial set of editable attribute vectors, and an initial structure template. The identity core vector is used to represent the inherent identity information of the AI ​​digital human. The state transition module is used to extract context information from evolutionary history data that records historical editing operations based on the path encoder and generate a path context vector, receive and encode the natural language editing instructions input by the user in real time, and generate a state transition operator based on the path context vector, the encoded natural language editing instructions, and the current set of editable attribute vectors. The main attribute update module is used to update the main attributes of the current editable attribute vector set based on the state transition operator to obtain a temporary attribute set, and then adjust the temporary attribute set a second time through a preset attribute constraint field to generate a new editable attribute vector set, thereby completing the evolution of semantic state. The image synthesis module is used to synthesize the identity core vector and the new editable attribute vector set based on the image decoder and output the edited multimodal text guidance image.

Images