A face image generation method, device, computer equipment, medium and program product

Abstract

Embodiments of the present application disclose a face image generation method and device, computer equipment, a medium and a program product. A prompt text and a prompt image are obtained, parameter extraction is performed on the prompt image to obtain model parameter values for representing a target face, the model parameter values are converted to obtain first attribute features for representing internal attributes of the face, feature extraction is performed on the prompt text to obtain first text features, and a first face image is generated according to the first text features and the first attribute features. Thus, by introducing the first attribute features for representing the internal attributes of the face, the face attributes are refined into the internal attributes of the face and external attributes of the face. In the process of generating the first face image, more attention is paid to the internal attributes of the face that need to be retained and the internal attributes of the face or the external attributes of the face that need to be changed, so that the required internal attributes of the face are retained while meeting the modification requirements, and the accuracy of the generated face image is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer equipment, medium, and program product for generating facial images. Background Technology

[0002] With the development of computer technology, images can be modified through computer programs or artificial intelligence technology.

[0003] In related technologies, image modification is generally achieved through text-based image models. These models can automatically generate a corresponding image from a given text description (i.e., prompt text). The prompt text guides the image generation process, ensuring the generated image possesses the features included in the prompt text.

[0004] However, among related technologies, facial images generated based on prompt text have low accuracy. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a facial image generation method, apparatus, computer device, medium, and program product to improve the accuracy of facial image generation.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide a method for generating a facial image, the method comprising:

[0008] Obtain prompt text and prompt image, wherein the prompt text is text describing the modification requirements, and the prompt image is an image including the target face;

[0009] The prompt image is subjected to parameter extraction to obtain model parameter values ​​that characterize the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters are used to describe different inherent facial attributes of the target face.

[0010] The model parameter values ​​are transformed to obtain a first attribute feature used to characterize the intrinsic facial attributes;

[0011] Feature extraction is performed on the prompt text to obtain the first text feature;

[0012] A first facial image is generated based on the first text feature and the first attribute feature. The first facial image is an image obtained by modifying the target face based on the prompt text.

[0013] On the other hand, embodiments of this application provide a facial image generation apparatus, the apparatus comprising: an acquisition unit, an extraction unit, a conversion unit, and a generation unit;

[0014] The acquisition unit is used to acquire prompt text and prompt image, wherein the prompt text is text describing the modification requirements and the prompt image is an image including the target face;

[0015] The extraction unit is used to extract parameters from the prompt image to obtain model parameter values ​​that characterize the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters are used to describe different inherent facial attributes of the target face.

[0016] The conversion unit is used to convert the model parameter values ​​to obtain a first attribute feature that characterizes the intrinsic facial attributes.

[0017] The extraction unit is also used to extract features from the prompt text to obtain first text features;

[0018] The generation unit is configured to generate a first facial image based on the first text features and the first attribute features, wherein the first facial image is an image obtained by modifying the target face based on the prompt text.

[0019] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0020] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0021] The processor is configured to execute the methods described above according to instructions in the computer program.

[0022] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0023] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the methods described above.

[0024] As can be seen from the above technical solution, the prompt text and prompt image are obtained, and model parameters are used to construct a parametric model of the face. Different model parameter values ​​can represent different faces. Parameters are then extracted from the prompt image to obtain model parameter values ​​representing the target face. These model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters describe different intrinsic facial attributes of the target face, and different attribute parameter values ​​of the same attribute parameter describe different degrees of the corresponding intrinsic facial attributes. This achieves decoupling of multiple facial attributes that constitute the face, allowing for more refined modifications to the face based on any attribute parameter in the model parameters, making the process more intuitive. The model parameter values ​​are converted into first attribute features, and features are extracted from the prompt text to obtain first text features. Based on the first text features and the first attribute features, a first facial image is generated. Therefore, by introducing first attribute features to represent intrinsic facial attributes, facial attributes are refined into intrinsic and extrinsic facial attributes. During the generation of the first facial image, greater attention is paid to the intrinsic facial attributes that need to be retained and the intrinsic or extrinsic facial attributes that need to be changed. This ensures that the required intrinsic facial attributes are retained while meeting modification needs, improving the accuracy of the generated facial image. Furthermore, different attribute parameters in the model parameters are used to modify different intrinsic facial attributes, enabling modifications to various parts of the face, decoupling the modification of facial parameters, and further improving the accuracy of the generated facial images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of a computer system for a facial image generation method provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram illustrating an application scenario of a facial image generation method provided in an embodiment of this application;

[0028] Figure 3 A flowchart illustrating a facial image generation method provided in this application embodiment;

[0029] Figure 4 A schematic diagram of an interface for determining prompt text and prompt image is provided for an embodiment of this application;

[0030] Figure 5A schematic diagram illustrating parameter extraction as provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram illustrating a conversion model parameter value provided in an embodiment of this application;

[0032] Figure 7 A schematic diagram of a first facial image provided in an embodiment of this application;

[0033] Figure 8 A schematic diagram illustrating the modification of a first facial image provided in an embodiment of this application;

[0034] Figure 9 A schematic diagram of a second facial image provided in an embodiment of this application;

[0035] Figure 10 A schematic diagram illustrating a method for obtaining a first facial image based on two attention networks, as provided in an embodiment of this application;

[0036] Figure 11 A schematic diagram illustrating the training process of a first attention network provided in an embodiment of this application;

[0037] Figure 12 A schematic diagram illustrating the training process of a second attention network provided in an embodiment of this application;

[0038] Figure 13 A schematic diagram illustrating an adjustment noise reduction process provided in an embodiment of this application;

[0039] Figure 14 A schematic diagram illustrating a noise reduction process provided in an embodiment of this application;

[0040] Figure 15 A schematic diagram of an identity measurement model provided in an embodiment of this application;

[0041] Figure 16 A schematic diagram illustrating the application process of a facial recognition model provided in this application embodiment;

[0042] Figure 17 This is a schematic diagram of the structure of a facial image generation device provided in an embodiment of this application;

[0043] Figure 18 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0044] Figure 19 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application will now be described with reference to the accompanying drawings.

[0046] In related technologies, text-generated image models generally include a text encoder and a generative model (such as a generative adversarial network or a variational autoencoder). First, the text encoder receives the prompt text and converts it into a digital representation, i.e., a text feature vector. Then, the generative model maps these text feature vectors to the image space. During image generation, the text-generated image model gradually reduces noise and restores the structural information of the image until an image matching the text description is generated.

[0047] In other words, the text-based image model aims to learn a concept representing a face. This learned concept is then used to generate images with textual cues under novel conditions (such as different expressions) and styles, thereby controlling the external facial attributes of the image, such as age and whether glasses are worn. By using textual descriptions of these external facial attributes, the attributes of the generated image are controlled within the latent space through a cross-attention mechanism.

[0048] However, facial images depict more facial details, including numerous facial attributes. Controlling facial attributes via prompts only controls external attributes such as age and glasses. These external attributes primarily reflect the face's outward appearance or additional features. Modifying internal facial attributes, such as identity, expression, and texture, via prompts presents challenges in accurately preserving individual identity and its nuanced internal attributes. Furthermore, the diversity of internal facial attributes makes it difficult for users to clearly articulate their modification requests, and hinders fine-grained control over facial attributes, resulting in lower accuracy of the generated facial images.

[0049] Based on this, embodiments of this application provide a facial image generation method. By introducing a first attribute feature to characterize the intrinsic attributes of the face, facial attributes are refined into intrinsic facial attributes and extrinsic facial attributes. During the generation of the first facial image, greater attention is paid to the intrinsic facial attributes that need to be retained and the intrinsic or extrinsic facial attributes that need to be changed. This allows for the preservation of the required intrinsic facial attributes while meeting modification needs, thereby improving the accuracy of the generated facial image. Furthermore, different attribute parameters in the model parameters are used to modify different intrinsic facial attributes, enabling modifications to various parts of the face and decoupling the modification of facial parameters, further improving the accuracy of the generated facial image.

[0050] To facilitate understanding of the facial image generation method provided in the embodiments of this application, the computer system for the facial image generation method will be described below.

[0051] See Figure 1 This figure is a schematic diagram of a computer system for a facial image generation method provided in an embodiment of this application. Figure 1As shown, this application scenario includes a terminal device 100 and a server 200, and the terminal device 100 and the server 200 can communicate with each other through a communication network.

[0052] The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, private network, or any combination of virtual private network. In some embodiments, custom or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0053] Terminal device 100 can be electronic devices such as mobile phones, tablets, in-vehicle terminals (vehicle systems), wearable devices, personal computers (PCs), intelligent voice interaction devices, smart home appliances, aircraft, and unmanned vending terminals. A client application for the target application can be installed and run on terminal 100. This target application can be an application that supports facial image generation, or other applications that support facial modeling, facial rendering, and facial storage; this application does not limit its scope. Furthermore, this application does not limit the form of the target application, including but not limited to applications (Apps), mini-programs, etc., installed on terminal device 100, and can also be in the form of web pages.

[0054] Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and other basic cloud computing services. Server 200 can be a backend server for the aforementioned target application, used to provide backend services to the clients of the target application.

[0055] To facilitate understanding of the facial image generation method provided in this application embodiment, the following example illustrates the application scenario of the facial image generation method by taking the server as the execution subject of the facial image generation method and the server interacting with one of the terminal devices.

[0056] See Figure 2 The figure is a schematic diagram of an application scenario of a facial image generation method provided in an embodiment of this application.

[0057] Terminal device 100 can obtain prompt text and prompt images through user operations (such as upload operations, selection operations, etc.). The prompt image provides relevant information about the target face, and the prompt text includes the direction the user wants to modify, so that the target face in the prompt image can be modified subsequently based on the prompt text. Figure 2 As shown, the prompt image provides the target face awaiting modification, and the prompt text adds framed glasses to the person. The terminal device 100 sends the prompt text and prompt image to the server 200.

[0058] After receiving the prompt text and prompt image, server 200 extracts parameters from the prompt image to obtain model parameter values ​​representing the target face, and converts these model parameter values ​​into first attribute features. The model parameter values ​​are the numerical values ​​of the model parameters used to construct a parametric model of the face. Different model parameter values ​​can represent different faces. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters describe different intrinsic facial attributes of the target face, and different attribute parameter values ​​describe different degrees of the corresponding intrinsic facial attributes. Therefore, subsequent modifications to the face can be made based on any of the attribute parameters in the model parameters.

[0059] Server 200 extracts features from the prompt text to obtain a first text feature, which is used to characterize the user's modification request. Server 200 generates a first facial image based on the first text feature and the first attribute feature. (Continue to the next step...) Figure 1 An image was generated showing the target face wearing framed glasses. Therefore, by introducing a first attribute feature to characterize the intrinsic properties of the face, facial attributes are refined into intrinsic and extrinsic attributes. During the generation of the first facial image, greater attention is paid to the intrinsic attributes that need to be preserved and the intrinsic or extrinsic attributes that need to be modified. This allows for the retention of desired intrinsic attributes while meeting modification requirements, improving the accuracy of the generated facial image. Furthermore, different attribute parameters in the model are used to modify different intrinsic facial attributes, enabling modifications to various parts of the face and decoupling the modification of facial parameters, further improving the accuracy of the generated facial image.

[0060] The facial image generation method provided in this application embodiment can be executed by a server. However, in other embodiments of this application, the terminal device may also have similar functions to the server to execute the facial image generation method provided in this application embodiment, or the terminal device and the server may jointly execute the facial image generation method provided in this application embodiment. This embodiment does not limit this.

[0061] The following describes in detail a facial image generation method provided in this application through method embodiments.

[0062] See Figure 3 This figure is a schematic flowchart of a facial image generation method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of this facial image generation method as an example. Figure 3 As shown, the facial image generation method includes S301-S305.

[0063] S301: Get the prompt text and prompt image.

[0064] The prompt image is an image that includes the target face, which is the face to be modified. The face included in the prompt image directly reflects the inherent attributes of the face, which are attributes fixed to the face, such as identity attributes, expression attributes, and texture attributes. Among them, identity attributes are unique to each face and are used to distinguish individuals. Expression attributes are used to reflect the emotional state of the face. Texture attributes describe details such as skin texture, blemishes, and luster.

[0065] The prompt text describes the modification requests, which can target either intrinsic or extrinsic facial attributes. Extrinsic attributes are those that define facial features through external attributes and additional characteristics, such as age and accessories. Age reflects the state of the face; for example, the skin condition of an elderly person differs from that of a child. Accessories can influence facial style and alter the appearance of the face, such as whether or not glasses are worn.

[0066] The prompt image carries the inherent facial attributes of the target face, while the prompt text carries information for modifying the target face, that is, it describes the requirements for image generation so as to satisfy the modification requirements of the prompt text while ensuring that the identity of the target face in the prompt image is not changed too much.

[0067] Prompt text and images can be obtained from object input. Objects can input prompt text and images into a computer device based on image generation requirements; prompt text and images can be obtained through object input. See also Figure 4 This figure is a schematic diagram of an interface for determining prompt text and prompt images according to an embodiment of this application. The text content in the prompt text input box 410 is used as the prompt text, and the image 420 uploaded by the object in the image upload component is extracted as the prompt image. When there are multiple prompt images, sub-prompt text corresponding to each prompt image can be determined in the prompt text.

[0068] All data collected in this application (such as prompt text and prompt images) is collected with the consent and authorization of the data subject (such as user, organization or enterprise), and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0069] S302: Extract parameters from the prompt image to obtain model parameter values ​​used to represent the target face.

[0070] Model parameters are used to construct a parametric model of the face. A parametric model decomposes the facial model into parameters. As one possible implementation, model parameters include shape parameters and texture parameters. Shape parameters describe the geometric structure of the face, such as the position, size, and shape of facial features. Texture parameters describe the appearance features of the face, such as skin color, texture, and lighting conditions. Different combinations of shape and texture parameters result in different faces.

[0071] Taking the face as an example, the model parameters can be a 3D Morphable Model (3DMM). 3DMM is a statistical modeling method that constructs an average face template using a large amount of face scan data and learns the shape and texture variation patterns of the face. Its core idea is to represent the face as a parameterized model, with parameters including shape parameters and texture parameters. Shape parameters are typically represented as shape vectors, which contain the 3D coordinates of facial key points (such as the corners of the eyes and mouth). By adjusting the values ​​of the shape parameters, face models with different geometric shapes can be generated. Texture parameters are typically represented as texture vectors, which contain the color information of each pixel in the face image. By adjusting the values ​​of the texture parameters, face images with different appearance features can be generated. Thus, each face can be represented as a linear superposition of shape vectors and texture vectors.

[0072] Model parameter values ​​are the numerical values ​​corresponding to model parameters. Different model parameter values ​​can represent different faces. Model parameter values ​​include multiple attribute parameter values, each of which corresponds to an attribute parameter. Different attribute parameters are used to describe the different inherent properties of the face. Different attribute parameter values ​​are used to describe the different degrees of the corresponding inherent properties of the face. Therefore, the face can be modified based on any attribute parameter in the model parameters.

[0073] This application does not specifically limit the method of parameter extraction from the prompt image. For example, 3D reconstruction can be performed based on the target face included in the prompt image, and model parameter values ​​can be obtained based on the target face model obtained from the 3D reconstruction. Alternatively, parameter extraction can be achieved by comparing the model with a pre-trained 3D face reconstruction model. The following explanation will continue using 3DMM as an example.

[0074] See Figure 5 This figure is a schematic diagram of parameter extraction provided in an embodiment of this application. Figure 5 In this study, the R model (Refine Network, Rnet) is used to extract parameters from the prompt image. Rnet uses multiple convolutional layers to extract high-level features of the prompt image. These convolutional layers can gradually identify important information in the image, such as edges, textures, and complex structures. Then, by using the identified important information, parameters are predicted to obtain a set of attribute parameters, namely identity parameter (α), expression parameter (β), texture parameter (δ), pose parameter (ρ), and illumination parameter (γ). Different attribute parameters correspond to different attribute parameter values.

[0075] S303: Transform the model parameter values ​​to obtain the first attribute feature used to characterize the intrinsic properties of the face.

[0076] To effectively inject model parameter values, these values ​​can be converted into features, resulting in first attribute features. These first attribute features characterize the intrinsic facial attributes. The first attribute features include multiple dimensions, each corresponding to a different intrinsic facial attribute, thus reflecting the characteristics of the target face across various intrinsic facial attributes.

[0077] This application does not specifically limit the conversion method in its embodiments. For example, the conversion of model parameter values ​​can be achieved through mapping relationships. Another example is the use of conditional projection networks to convert model parameter values, which will be discussed below. Figure 6 The following explanation uses a conditional projection network, specifically a multilayer perceptron (MLP), as an example.

[0078] See Figure 6 This figure is a schematic diagram of a conversion model parameter value provided in an embodiment of this application. Figure 6 In this model, the multilayer perceptron includes a parameter normalization layer 601, a linear layer 602, an activation function 603, and a layer normalization layer 604. Through the multilayer perceptron, the model parameter values ​​can be transformed into first attribute features.

[0079] The model parameter values ​​are normalized through a parameter normalization layer 601 to ensure that the model parameter sizes are within a reasonable range. Normalization prevents some model parameter values ​​from becoming too large or too small during training, thus avoiding gradient vanishing or exploding problems. A linear transformation is performed through a linear layer 602 to extract features, enabling the subsequent extraction of more useful information. An activation function 603 allows the network to learn non-linear relationships in the input data, thereby enabling the solution of more complex tasks. This embodiment does not specifically limit the activation function, such as GELU (Gaussian Error Linear Unit), ReLU (Rectified Linear Unit), sigmoid, and tanh. Layer normalization 604 is a regularization technique that reduces internal covariate shift by normalizing the input data of each neuron. This helps to accelerate training and improve the model's generalization ability.

[0080] S304: Extract features from the prompt text to obtain the first text features.

[0081] Feature extraction is performed on the prompt text to obtain the first text feature, which is used to represent the modification requirement for the target face. Through feature extraction, the modification requirement is extracted from the prompt text and made into a form that is easier for computer devices to understand.

[0082] This application does not specifically limit the method of feature extraction for the prompt text. For example, feature extraction can be performed by a text encoder or by a convolutional neural network (CNN). This application does not make specific limitations in this regard, and those skilled in the art can make settings according to actual needs.

[0083] As one possible approach, the first attribute feature and the first text feature reside in the same feature space to achieve feature alignment. For example, the first text feature is typically located in the text embedding space, so the model parameter values ​​can be transformed to the text embedding space, thus placing the first text feature and the first attribute feature in the same feature space, which is convenient and quick.

[0084] The text embedding space is a high-dimensional vector space where each vector represents a word or sentence. By projecting data into the text embedding space, semantic relationships between words can be captured. In the text embedding space, each piece of text is mapped to a high-dimensional vector, where each dimension represents a feature (such as the frequency of a word, a component of a word vector, etc.), and the value of the vector represents the weight or importance of that feature in the text.

[0085] S305: Generate a first facial image based on the first text features and the first attribute features.

[0086] The first text feature is used to provide the modification requirements for the target face, and the first attribute feature is used to provide the characteristics of the target face in various intrinsic facial attributes. The target face is modified based on the modification requirements indicated by the first text feature, and the modification of the first attribute feature is kept as small as possible during the modification process. This ensures that the required intrinsic facial attributes are preserved while meeting the modification requirements, thereby improving the accuracy of the generated facial image.

[0087] See Figure 7 This figure is a schematic diagram of a first facial image provided in an embodiment of this application. Figure 7 The first facial image diagram 710 obtained in the process is based on Figure 4 The prompt text and image shown are provided. If the user is not satisfied with the generated result, they can continue to modify control 720. See below for specific modification methods. Figure 8 The three modification methods shown will not be elaborated upon here.

[0088] The embodiments of this application do not specifically limit the method of generating the first facial image. For example, it can be generated through text-to-imagine T2I, Generative Adversarial Network (GAN), Diffusion Model, etc. C1-C2 and D1-D3 will be used as examples in the following descriptions, and will not be repeated here.

[0089] As described in the above technical solution, the process involves acquiring prompt text and a prompt image, using model parameters to construct a parametric model of the face. Different model parameter values ​​can represent different faces. Parameters are then extracted from the prompt image to obtain model parameter values ​​representing the target face. These model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters describe different intrinsic facial attributes of the target face, and different attribute parameter values ​​describe different degrees of the corresponding intrinsic facial attributes. This decoupling of multiple facial attributes allows for more precise modification of the face based on any attribute parameter in the model parameters, making the process more intuitive. The model parameter values ​​are converted into first attribute features, and features are extracted from the prompt text to obtain first text features. Based on the first text features and first attribute features, a first facial image is generated. Therefore, by introducing first attribute features to represent intrinsic facial attributes, facial attributes are refined into intrinsic and extrinsic facial attributes. During the generation of the first facial image, greater attention is paid to the intrinsic facial attributes that need to be retained and the intrinsic or extrinsic facial attributes that need to be changed. This ensures that the required intrinsic facial attributes are retained while meeting modification needs, improving the accuracy of the generated facial image. Furthermore, different attribute parameters in the model parameters are used to modify different intrinsic facial attributes, enabling modifications to various parts of the face, decoupling the modification of facial parameters, and further improving the accuracy of the generated facial images.

[0090] As one possible implementation, if the user is not satisfied with the generated first facial image, it can be modified again.

[0091] See also Figure 7 If the user is not satisfied with the generated first facial image, they can continue to modify it. A trigger operation can be executed on the continue modification control 710. At this time, three modification methods can be displayed to achieve further modification: modifying the prompt text, modifying attribute parameters, and modifying the prompt image. Users can choose one, two, or all three modification methods; this application does not specifically limit this choice.

[0092] See Figure 8 This figure is a schematic diagram illustrating the modification of a first facial image according to an embodiment of this application. For example, the prompt text can be modified by re-entering the prompt text, such as in... Figure 8 The previously entered prompt text is displayed in the prompt text input box 810, and can be directly modified based on this prompt text.

[0093] For example, the prompt image can be modified by re-uploading it, such as in... Figure 8The system triggers an operation on the click-to-upload-image control 820, prompting the user to re-upload a new image. Additionally, it can display previously uploaded images while simultaneously displaying the click-to-upload-image control 820.

[0094] For example, multiple attribute parameters can be displayed to allow users to select the desired parameter for modification. The attribute parameter value can be modified by entering a numerical value or increasing the value. The following explanation uses the modification of an attribute parameter as an example; see A1-A4 for details.

[0095] A1: Get the value of the first target attribute parameter corresponding to the first target attribute parameter.

[0096] The first target attribute parameter is one of the multiple attribute parameters included in the model parameters, and the value of the first target attribute parameter is the numerical value corresponding to the first target attribute parameter.

[0097] This application does not specifically limit the method of obtaining the first target attribute parameter value. Two methods are described below as examples.

[0098] (1) While displaying the multiple attribute parameters included in the model parameters, display the attribute parameter values ​​corresponding to each of the multiple attribute parameters so that the user can clearly understand the current degree of each attribute parameter. Select one attribute parameter from the multiple attribute parameters as the first target attribute parameter, and input the first target attribute parameter value corresponding to the first target attribute parameter.

[0099] See Figure 8 This figure is a schematic diagram illustrating facial image editing according to an embodiment of this application. Figure 8 In this application, the modification of attribute parameters is achieved through a combination of drop-down lists and input boxes. Other methods are also possible, and this application does not impose specific limitations on this. For example, multiple attribute parameters can be displayed through drop-down list 830, such as attribute parameters for representing a smile, attribute parameters for representing a beard, etc. The user can modify one of the attribute parameters by selecting one, such as selecting the attribute parameter representing a beard. After selecting the attribute parameter, the corresponding attribute parameter value can be first displayed through input box 840, such as in... Figure 7 If the face in the first facial image has no beard, then the attribute parameter value corresponding to the beard is 0. You can re-enter the attribute parameter value you want to modify, such as 100 (not shown in the figure).

[0100] (2) Users can directly issue modification instructions via text, voice, or other commands. Taking text as an example, the edit text is obtained, which is used to modify the first facial image. The first target attribute parameter or the newly input prompt text is extracted from the edit text so that the modification of the first facial image can be achieved based on the first target attribute parameter or the newly input prompt text.

[0101] Therefore, modifying the first facial image using method (1) is intuitive, quick, and highly instructive. Modifying the first facial image using method (2) is simpler for the user and improves the user experience.

[0102] A2: Update the model parameter values ​​based on the first target attribute parameter values ​​to obtain the first updated model parameter values.

[0103] The model parameter values ​​of the first facial image are updated based on the first target attribute parameter value. For example, the attribute parameter value representing the beard in the model parameter values ​​is changed from 0 to 100, thereby obtaining the updated model parameter values, which are the first updated model parameter values.

[0104] A3: Transform the parameter values ​​of the first updated model to obtain the second attribute feature.

[0105] The second attribute feature is the attribute feature obtained by transforming the first updated model parameter value. The relevant transformation method can be found in the transformation method of model parameter value in S303 above, and will not be repeated here.

[0106] A4: Based on the first text features and the second attribute features, the second facial image is obtained.

[0107] The second facial image is obtained by modifying the target face based on the prompt text and the first target attribute parameter value, and it is also obtained by further modifying the first facial image based on the first target attribute parameter value.

[0108] The method of obtaining the second facial image based on the first text feature and the second attribute feature can be found in the method of obtaining the first facial image based on the first text feature and the first attribute feature in the aforementioned S304, and will not be repeated here.

[0109] See Figure 9 This figure is a schematic diagram of a second facial image provided in an embodiment of this application. Based on Figure 7 Based on the generated first facial image, through Figure 8 A beard was added to the face in the first facial image, thus obtaining Figure 9 The face shown.

[0110] Therefore, if the user is not satisfied with the first facial image, they can continue to modify the prompt text, prompt image, and various attribute parameters of the model to obtain a modified second facial image. Multiple modification methods provide users with rich modification options. Moreover, the modification of various attribute parameters is enabled, allowing precise control over various facial attributes, fine-grained modification of the face, improved editing effects, and enhanced user experience.

[0111] As one possible implementation, the attribute parameters can be modified not only after obtaining the first facial image, but also before obtaining the first facial image. That is, the embodiments of this application provide a specific implementation of S303, which is to convert the model parameter values ​​to obtain a first attribute feature used to characterize the intrinsic attributes of the face.

[0112] Specifically, the second target attribute parameter is one of multiple attribute parameters, and the value of the second target attribute parameter is the numerical value corresponding to the second target attribute parameter. The second target attribute parameter can be the same as or different from the first target attribute parameter; this application does not impose any specific limitations on this. After obtaining the value of the second target attribute parameter corresponding to the second target attribute parameter, and after obtaining the model parameter value based on the prompt image, the model parameter value can be updated based on the second target attribute parameter value to obtain a second updated model parameter value. The second updated model parameter value is then transformed to obtain the first attribute feature.

[0113] Therefore, not only can the model parameter values ​​obtained from the prompt image be modified based on the prompt text to obtain the first facial image, but the model parameters obtained from the prompt image can also be modified based on the prompt text and the second target attribute parameter values ​​to obtain the first facial image. By enabling attribute parameter modification before obtaining the first facial image, the modification methods are enriched, the modification speed is improved, and the user experience is enhanced.

[0114] As one possible implementation, not only can model parameters be introduced, but key points can also be introduced. That is, the embodiments of this application provide a specific implementation of S303, which is to convert the model parameter values ​​to obtain a specific implementation of the first attribute feature used to characterize the intrinsic attributes of the face. See B1-B3 for details.

[0115] B1: Extract key points from the prompt image to obtain key point parameter values.

[0116] Keypoints are important areas on the face. Taking a human face as an example, keypoints can include the eyes, mouth, nose, etc. Keypoint parameter values ​​are numerical values ​​corresponding to keypoints, used to characterize the positions of multiple keypoints on the target face.

[0117] The embodiments of this application do not specifically limit the method of key point extraction. For example, key point extraction can be performed using a pre-trained key point detection model. The key point detection model can be an Active Shape Model (ASM), an Active Appearance Model (AAM), etc. This application does not specifically limit it.

[0118] B2: Transform the keypoint parameter values ​​to obtain the first pose feature used to characterize the pose of the target face.

[0119] In order to effectively inject keypoint parameter values, the keypoint parameter values ​​can be converted into features to obtain the first pose feature, which is used to characterize the pose of the target face.

[0120] This application does not specifically limit the conversion method. For example, the conversion of key point parameter values ​​can be achieved through mapping relationships. Another example is using a multilayer perceptron to convert key point parameter values, such as based on... Figure 6 The multilayer perceptron shown realizes the conversion of key point parameter values, that is, the input of key point parameter values ​​to... Figure 6 In a multilayer perceptron, parameter normalization, feature extraction, introduction of nonlinear relationships, and parameter regularization are performed to obtain the first pose features. For details, please refer to [link to documentation]. Figure 6 The embodiments shown are not described in detail here.

[0121] As one possible implementation, the first attribute feature and the first pose feature are located in the same feature space to enable feature alignment. For example, if the first attribute feature is located in the text embedding space, the keypoint parameter values ​​can be transformed to the text embedding space, thus making the first pose feature and the first attribute feature the same feature space as the first text feature, which is convenient and quick.

[0122] B3: Generate a first facial image based on the first pose feature, the first text feature, and the first attribute feature.

[0123] The first pose feature is used to provide the pose of the target face, the first text feature is used to provide the modification requirements for the target face, and the first attribute feature is used to provide the characteristics of the target face in various intrinsic facial attributes. The target face is modified based on the modification requirements indicated by the first text feature, and the modification of the first attribute feature is kept as small as possible during the modification process. The pose of the target face is also taken into account, so as to ensure that the modification requirements are met while the required intrinsic facial attributes are preserved, thereby improving the accuracy of the generated facial image.

[0124] Therefore, by introducing facial key points, it is possible to introduce first pose features along with first attribute features and first text features, thereby introducing pose conditions and enabling the generated facial images to have more accurate pose changes. This is beneficial for cross-pose facial generation, improves the accuracy and flexibility of facial generation, and enhances the user experience.

[0125] As one possible implementation, the fusion of features can be achieved through a cross-attention mechanism to obtain the first facial image. Based on this, this application provides a specific implementation of S303, namely, a specific implementation of generating the first facial image based on the first text features and the first attribute features. Two methods are described below as examples, where method one is specifically C1-C2, and method two is specifically D1-D3.

[0126] Method 1: The first text feature and the first attribute feature are fused through a cross-attention mechanism.

[0127] C1: Based on the first text feature and the first attribute feature, the first fused feature is obtained by fusing them through a cross-attention mechanism.

[0128] The first fusion feature is a feature obtained based on the first text feature and the first attribute feature. The cross-attention mechanism is an attention mechanism used to mix two different embedding sequences. It is mainly used for information interaction between different inputs (i.e., different embedding sequences), thereby effectively aligning and focusing on inputs from different sources, and thus effectively capturing the correlation between the two inputs.

[0129] This application does not specifically limit the fusion method to cross-attention mechanism; two methods are described below as examples.

[0130] Method 1: The first text feature and the first attribute feature can be fused using a cross-attention mechanism to obtain the first fused feature.

[0131] After the first attribute feature and the first text feature are fused through a cross-attention mechanism, the first attribute feature can learn the first text feature and integrate it into a predetermined form guided by the first fused feature. For example, the first attribute feature can guide the generation of an image of a target face. The first text feature includes the semantic feature of "square glasses." After being fused with the first attribute feature through a cross-attention mechanism, the feature of "square glasses" is integrated into the first attribute feature. The first attribute feature can then learn the feature of "square glasses" and guide the generation of features of the target face with "square glasses," thus obtaining the first fused feature.

[0132] Method Two: First, feature extraction is performed on the prompt image to obtain image features. Then, a cross-attention mechanism is used to fuse the first text features and the image features, and a cross-attention mechanism is used to fuse the first attribute features and the image features. Finally, a cross-attention mechanism is used to fuse the two fused features again to obtain the first fused feature.

[0133] As one possible implementation, a cross-attention mechanism can be implemented through a first attention network and a second attention network. The first attention network is used to fuse the first attribute features and image features, and the second cross-attention mechanism is used to fuse the first text features and image features. This will be explained based on E1-E3 later, and will not be elaborated here.

[0134] C2: Denoise the noisy image based on the first fusion feature to obtain the first facial image.

[0135] The noisy image is obtained based on the prompt text and noise. The image is obtained based on the prompt text, and noise is added to the image. For example, the noise can be random noise obtained by sampling Gaussian noise, etc. This application does not make specific limitations on this.

[0136] Taking denoising using a text-based graph model as an example, noise and cue text can be used as inputs to the text-based graph model. In the context of the text-based graph model, noise can be regarded as a "random seed," which introduces the randomness and diversity needed to generate the image. The cue text provides the semantic information needed to generate the image. This random seed, along with the cue text, is input into the text-based graph model, which learns how to combine the two to generate diverse images that match the text description.

[0137] The text-based image processing (TPS) model generates an initial, noisy image based on the prompt text and noise. The TPS model then performs a series of iterative denoising steps. In each step, based on the indications of the first fused feature, the TPS model attempts to remove a portion of the noise in the current image, gradually bringing it closer to a clear image that matches the prompt text description. This process is similar to progressively refining and optimizing the image until a satisfactory result is achieved, ultimately yielding the first facial image.

[0138] Therefore, by using a cross-attention mechanism to fuse the first text feature and the first attribute feature, attention is paid to inputs from different modalities, thereby learning the interaction between the two modalities, achieving more effective information fusion, and thus more accurately capturing and understanding the relationship between data from different modalities, thereby improving task performance.

[0139] Method 2: The first text feature, the first attribute feature, and the first pose feature are fused through a cross-attention mechanism.

[0140] D1: Combine the first pose feature and the first attribute feature to obtain the first combined feature.

[0141] Both the first pose feature and the first attribute feature are extracted from the prompt image. The first spliced ​​feature obtained by splicing the two can describe the prompt image more richly from multiple dimensions, thereby improving the accuracy of the subsequently generated first facial image.

[0142] D2: Based on the first concatenation feature and the first text feature, the second fused feature is obtained by fusing them through a cross-attention mechanism.

[0143] The second fusion feature is a feature obtained based on the first splicing feature and the first text feature.

[0144] The first splicing feature and the first text feature can be directly fused through a cross-attention mechanism, or indirectly fused through image features. For details, please refer to the aforementioned fusion methods for the first text feature and the first attribute feature, which will not be repeated here.

[0145] D3: Denoise the noisy image based on the second fusion feature to obtain the first facial image.

[0146] For specific noise reduction methods, please refer to the aforementioned noise reduction method for C2, which will not be repeated here.

[0147] Therefore, by first concatenating the first pose feature and the first attribute feature, the resulting first concatenated feature can more richly describe the prompt image through multiple dimensions. Then, by using a cross-attention mechanism, the first text feature and the first concatenated feature are fused together. This enables attention to inputs from different modalities, thereby learning the interaction between the two modalities, achieving more effective information fusion, and thus more accurately capturing and understanding the relationship between data from different modalities, thereby improving task performance.

[0148] The process of implementing the cross-attention mechanism through the first attention network and the second attention network is described below. Specifically, this application provides a specific implementation of C1, which is to fuse the first text feature and the first attribute feature through the cross-attention mechanism to obtain the first fused feature. See E1-E3 for details.

[0149] E1: The first attribute feature and the image feature are fused according to the first attention network to obtain the first sub-fused feature.

[0150] Image features are features extracted from the prompt image and are used to describe the characteristics of the target face in the prompt image. Although both the first attribute features and image features come from the prompt image, they belong to different modalities. By fusing the first attribute features and image features through a first attention network, the image features can better learn the first attribute features, thereby achieving effective information fusion.

[0151] As one possible implementation, the first attribute feature and the first pose feature can be concatenated to obtain the first concatenated feature. The first concatenated feature and the image feature can then be fused using a first attention network to obtain the first sub-fused feature, thereby achieving more effective information fusion.

[0152] See Figure 10 The figure is a schematic diagram of obtaining a first facial image based on two attention networks according to an embodiment of this application.

[0153] exist Figure 10 In the process, based on the prompt image, key point parameter values ​​and model parameter values ​​are obtained, and then... Figure 6 The conditional projection layer transforms the keypoint parameter values ​​and model parameter values ​​to obtain the first pose feature and the first attribute feature, respectively. The first attribute feature and the first pose feature are concatenated to obtain the first concatenated feature. A feature extraction network extracts features from the cue image to obtain image features. The image features and the first concatenated feature are input into a first attention network, where a cross-attention mechanism is executed to fuse the image features and the first concatenated feature, thus obtaining the first sub-fused feature.

[0154] E2: The first text features and image features are fused using the second attention network to obtain the second sub-fused features.

[0155] The first text features and the image features belong to different modalities. By fusing the first text features and the image features through the second attention network, the image features can learn the first text features better, thereby achieving effective information fusion.

[0156] See also Figure 10 The text encoder extracts features from the prompt text to obtain the first text features. The first text features and image features are then input into the second attention network. The second attention network implements a cross-attention mechanism, which enables the first text features and image features to be fused to obtain the second sub-fused features.

[0157] E3: The first fusion feature is obtained based on the first sub-fusion feature and the second sub-fusion feature.

[0158] The embodiments of this application do not specifically limit the method of obtaining the first fusion feature. For example, the first sub-fusion feature and the second sub-fusion feature can be added together to obtain the first fusion feature, or the first sub-fusion feature and the second sub-fusion feature can be concatenated. Thus, the first fusion feature not only learns image features, but also first text features and first attribute features related to the image, thereby improving the accuracy of the obtained first fusion feature.

[0159] See also Figure 10After obtaining the first fusion feature, the first fusion feature can guide the U-Net model in the denoising process of the noisy image, thereby obtaining a more accurate first facial image. This application does not specifically limit the denoising process; it will be described based on H1-H4 later and will not be repeated here. The U-Net model is the core of the diffusion model, and in the diffusion model, the U-Net model is typically used for iterative denoising of random latent image representations.

[0160] Therefore, by using the first attention network and the second attention network, the first text feature and the first splicing feature are fused with the image feature respectively to obtain the first fused feature. This allows the first fused feature to fully learn the image feature, the first text feature, the first pose feature, and the first attribute feature, thereby better learning the characteristics of the target face and the modification requirements. As a result, the obtained first facial image is more accurate and the user experience is improved.

[0161] The training process of the first and second attention networks will be explained below. First, the training process of the first attention network will be explained based on F1-F7, and then the training process of the second attention network will be explained based on G1-G7.

[0162] F1: Obtain the first noisy image sample and the prompt image sample.

[0163] The first noisy image sample is a noisy image used for training, and the cue image sample is a cue image used for training.

[0164] F2: Extract features from the prompt image sample to obtain image feature samples.

[0165] Image feature samples are image features obtained by extracting features from prompt image samples. The feature extraction method can be found in the aforementioned feature extraction method for prompt images, and will not be repeated here.

[0166] See Figure 11 This figure is a schematic diagram of the training process of a first attention network provided in an embodiment of this application. Figure 11 In this process, features can be extracted from the prompt image samples using a pre-trained feature extraction network to obtain image feature samples.

[0167] F3: Extract parameters from the prompt image samples to obtain model parameter value samples.

[0168] The model parameter value sample is the model parameter value obtained by extracting parameters from the prompt image sample. The parameter extraction method can be found in the aforementioned method for extracting parameters from the prompt image, and will not be repeated here.

[0169] F4: Transform the model parameter value samples to obtain attribute feature samples.

[0170] Attribute feature samples are attribute features obtained by transforming model parameter value samples. The transformation method can be found in the aforementioned transformation method for model parameter values, and will not be repeated here.

[0171] As one possible implementation, keypoint extraction can be performed on the cue image samples to obtain keypoint parameter values, which can then be transformed to obtain pose feature samples. See also... Figure 11 The system extracts key points from the prompt image samples to obtain key point parameter values. These values ​​are then transformed using a pre-trained conditional projection network to obtain pose feature samples. Similarly, it extracts parameters from the prompt image samples to obtain model parameter value samples. These model parameter value samples are then transformed using a pre-trained conditional projection network to obtain attribute feature samples.

[0172] F5: Based on the initial first attention network, the image feature samples and attribute feature samples are fused to obtain the first sub-feature sample.

[0173] The first sub-feature sample is the feature sample obtained by fusing image feature samples and attribute feature samples through the initial first attention network.

[0174] The initial first attention network is the first attention network that has not yet been fully trained. As one possible implementation, the cross-attention network of the pre-trained diffusion model can be copied to obtain the initial first attention network, which can then be used to embed the first attribute features and the first pose features.

[0175] See also Figure 11 The pose feature sample and the attribute feature sample can be concatenated to obtain the concatenated feature sample. The concatenated feature sample and the image feature sample are then input into the initial first attention network. The cross attention mechanism is executed through the initial first attention network to achieve the fusion of the concatenated feature sample and the image feature sample to obtain the first sub-feature sample.

[0176] The process of training the initial first attention network can be explained by the following formula. Specifically, image features and stitched feature samples are projected onto three matrices in the initial first attention network, namely the query matrix Q, the key matrix, and the query matrix Q. Value matrix See formulas (1) to (3) for details.

[0177] Q = ZW q (1)

[0178]

[0179] Where Q is the query matrix, which is the query matrix copied from the cross-attention network and does not need to be adjusted; The key matrix is ​​obtained by adjusting the key matrix in the replicated cross-attention network; The value matrix is ​​obtained by adjusting the value matrix in the replicated cross-attention network; Z represents the image feature samples; c p To concatenate feature samples, the pose feature sample v is used. k and attribute feature sample v c obtained by splicing; W q , is the learnable projection matrix, and is the model parameters to be adjusted.

[0180] Then, calculate the query matrix Q and the key matrix. For the similarity between them, please refer to formula (4).

[0181]

[0182] Where A is the query matrix Q and the key matrix. The similarity between them; The query matrix is ​​the dot product of the key matrix and the query matrix, representing the similarity between the two sequences at different positions; d is the dimension of the key matrix, which acts as a scaling factor to avoid excessively large values. These similarities are converted into a probability distribution using the softmax function, representing the attention weight of the query for each key.

[0183] Finally, these attention weights are applied to the values ​​to obtain the output vector, which is equivalent to extracting the information of interest from the value sequence and inputting it into the next network. See formula (5) for details.

[0184]

[0185] in, The first sub-feature sample; A is the query matrix Q and the key matrix. The similarity between them; It is a value matrix.

[0186] F6: Denoise the first noisy image sample based on the first sub-feature sample to obtain the first predicted face image.

[0187] The diffusion model is guided by the first sub-feature sample to denoise the first noisy image, resulting in the first predicted face image. The first predicted face image is a face image obtained based on the cue image sample.

[0188] See also Figure 11 The first sub-feature sample and the first noisy image sample are input into the U-Net model. The first sub-feature sample guides the U-Net model to denoise the first noisy image sample, thus obtaining the first predicted face image.

[0189] F7: Based on the difference between the first predicted facial image and the prompt image sample, adjust the model parameters of the initial first attention network to obtain the first attention network.

[0190] The initial training objective of the first attention network is to enable it to effectively fuse spliced ​​feature samples and image feature samples, thereby obtaining features that can accurately describe the cue image sample. Therefore, the model parameters of the initial first attention network can be adjusted based on the differences between the first predicted facial image and the cue image sample to obtain the final first attention network. This results in a smaller and smaller difference between the first predicted facial image and the cue image sample, meaning the resulting first attention network can effectively fuse image feature samples and spliced ​​feature samples.

[0191] As one possible implementation, the conditional projection network can be an untrained initial conditional projection network. Based on the difference between the first predicted facial image and the prompt image samples, the model parameters of the initial first attention network are adjusted to obtain the first attention network. The model parameters of the initial conditional projection network are also adjusted to obtain the conditional projection network. Thus, by jointly training the conditional projection network and the first attention network, the accuracy of the conditional projection network can be improved.

[0192] Therefore, the training processes of the first attention network and the second attention network are decoupled. The second attention network is kept stationary, and the first attention network is trained first. This enables the first attention network to better integrate image features and splicing features, or to better integrate image features and first attribute features, thereby introducing first attribute features or first pose features to improve the accuracy of the subsequently obtained first facial image.

[0193] After training the first attention network, the second attention network is trained. See details below.

[0194] G1: Obtain labeled cue sample pairs and second image noise samples.

[0195] The second noisy image sample is a noisy image used for training. The second noisy image sample may be the same as or different from the first noisy image. This application does not make any specific limitation in this regard.

[0196] The cue sample pairs include cue text samples and cue image samples. The cue text samples are the cue text used for training, and the cue image samples are the cue images used for training, the same as the cue image samples used in F1-F7. The labels describe the images obtained by modifying the faces included in the cue image samples based on the cue text samples.

[0197] G2: Extract features from the prompt text sample to obtain text feature samples.

[0198] Text feature samples are text features obtained by extracting features from prompt text samples. The feature extraction method can be found in the aforementioned feature extraction method for prompt text, and will not be repeated here.

[0199] See Figure 12 This figure is a schematic diagram illustrating the training process of a second attention network provided in an embodiment of this application. Figure 12 In this process, a trained text encoder is used to extract features from the prompt text samples to obtain text feature samples.

[0200] G3: The image feature samples and attribute feature samples are fused according to the first attention network to obtain the second sub-feature samples.

[0201] The first attention network can fuse image feature samples and attribute feature samples through an attention mechanism to obtain the second sub-feature sample.

[0202] One possible implementation involves extracting key points from the prompt image sample to obtain key point parameter value samples, extracting parameters from the prompt image sample to obtain model parameter value samples, transforming the key point parameter value samples to obtain pose feature samples, transforming the model parameter value samples to obtain attribute feature samples, and fusing the attribute feature samples and pose feature samples to obtain a concatenated feature sample. Then, a second sub-feature sample is obtained by fusing the image feature samples and attribute feature samples using a first attention network.

[0203] See also Figure 12 The first attention network is trained based on the aforementioned F1-F7. The spliced ​​feature samples and image feature samples obtained based on the prompt image samples are input into the first attention network. The image feature samples and spliced ​​feature samples are fused by the first attention network to obtain the second sub-feature samples.

[0204] G4: Based on the initial second attention network, the image feature samples and text feature samples are fused to obtain the third sub-feature samples.

[0205] The third sub-feature sample is the feature sample obtained by fusing image feature samples and text feature samples through the initial second attention network.

[0206] The initial second attention network is a second attention network that has not yet been fully trained. One possible implementation is to use the cross-attention network of a pre-trained diffusion model as the initial second attention network to train the second attention network. Another possible implementation is to use low-rank adaptation (LoRA) to fine-tune the initial second attention network to obtain the second attention network; this application does not specifically limit this approach.

[0207] See also Figure 12 Text feature samples and image feature samples can be input into an initial second attention network, and then fused through the initial second attention network to obtain a third sub-feature sample. Figure 12 In this study, the inherent cross-attention layer, i.e., the initial second attention network, included in the pre-trained diffusion model for text feature injection, was fine-tuned. The initial second attention network was updated using a low-rank weight, while the first attention network was frozen.

[0208] The process of training the initial second attention network can be explained by the following formula. Specifically, for each query matrix W in the multi-head attention module... q Key matrix W k Sum matrix W v The fine-tuning equations are as follows (see formulas (6)-(8)).

[0209] W q ′ =W q +ΔW q (6)

[0210] W k ′ =W k +ΔW k (7)

[0211] W v ′ =W v +ΔW v (8)

[0212] Wherein, ΔW q ΔW k ΔW v This is a low-rank update obtained during fine-tuning, which occurs within the initial second attention network while keeping the first attention network frozen; W q W k and W v These are the original query matrix, key matrix, and value matrix within the initial second attention network; W q ′ Wk ′ and W v ′ The query matrix, key matrix, and value matrix are the result of the initial second attention network update.

[0213] After fine-tuning, the joint attention mechanism involves a fine-tuned cross-attention network (the second attention network) and a newly added cross-attention network (the first attention network). First, the query matrix Q and key matrix K of the initial second attention network are calculated. ′ Sum matrix V ′ For details, please refer to formulas (9) to (11).

[0214] Q ′ =ZW q ′ (9)

[0215] K ′ =ZW k ′ (10)

[0216] V ′ =ZW v ′ (11)

[0217] Among them, W q ′ W k ′ and W v ′ Here, Z represents the query matrix, key matrix, and value matrix after the initial second cross-attention network update; Q represents the image feature samples; ′ K ′ V ′ The query matrix, key matrix, and value matrix of the trained second attention network are the model parameters.

[0218] G5: Based on the second and third sub-feature samples, obtain the fused feature sample.

[0219] See also Figure 12 Based on the second and third sub-feature samples, the fused feature samples are obtained, as detailed in formulas (12) to (14).

[0220] Attn1 = Attention(Q) ′ ,K ′ V ′ (12)

[0221]

[0222] Z new=Attn1+Attn2 (14)

[0223] Where Attn1 is the third sub-feature sample; Attn2 is the second sub-feature sample, Z new For fusion of feature samples.

[0224] G6: Denoise the second noisy image sample based on the fused feature samples to obtain the second predicted face image.

[0225] The second noisy image is denoised using a diffusion model guided by fusion features to obtain a second predicted face image, which is a face image obtained based on cue image samples and cue text samples.

[0226] See also Figure 12 The fused feature samples and the second noisy image are input into the U-Net model. The fused feature samples guide the U-Net model to denoise the second noisy image samples, thus obtaining the second predicted face image.

[0227] G7: Based on the difference between the second predicted facial image and the label, adjust the model parameters of the initial second attention network to obtain the second attention network.

[0228] The initial training objective of the second attention network is to enable it to effectively fuse text and image feature samples, thereby obtaining features that accurately describe the modification requirements. This, in turn, allows it to be combined with the first attention network to produce a more accurate facial image. Therefore, the model parameters of the initial second attention network can be adjusted based on the difference between the predicted facial image and the label, resulting in a second attention network where the difference between the predicted facial image and the label gradually decreases. This means the second attention network and the first attention network can better fuse features from different modalities.

[0229] Therefore, the first attention network is obtained by copying the cross-attention network from the pre-trained diffusion model and training it. After obtaining the first attention network, its model parameters are kept unchanged, and the cross-attention network included in the pre-trained diffusion model is fine-tuned. The second attention network is obtained through joint training. In other words, by training the two attention networks in stages, different conditional mechanisms can be effectively used across stages, enhancing the model's ability to focus on attributes of different granularities, thereby improving the accuracy of the obtained facial images.

[0230] As one possible implementation, the denoising process may result in the generated facial image being dissimilar to the target face, i.e., an identity inconsistency problem occurs during facial synthesis. Based on this, this application provides a specific implementation of E3, namely, a specific implementation of obtaining the first fusion feature based on the first sub-fusion feature and the second sub-fusion feature.

[0231] Specifically, the noisy image is denoised multiple times based on the first fusion feature to obtain the first facial image. The following explanation uses the (n+1)th denoising process as an example, where n is a positive integer; see H1-H4 for details.

[0232] H1: Obtain the first undetermined facial image.

[0233] The undetermined facial image is the facial image obtained during the denoising process. The first undetermined facial image is obtained by performing the nth denoising operation on the noisy image based on the first fusion feature; that is, the facial image obtained after the nth denoising operation. For example, after performing the first denoising operation on the noisy image, the undetermined facial image obtained after the first denoising operation is obtained; after performing the second denoising operation on the undetermined facial image obtained after the first denoising operation, the undetermined facial image obtained after the second denoising operation is obtained. And so on, until the nth denoising operation is performed, resulting in the first undetermined facial image.

[0234] H2: Determine the similarity between the model parameter values ​​and the first undetermined facial image.

[0235] In many applications, such as film and television production and social media entertainment, the purpose of facial generation is not to completely change an individual's identity, but rather to modify or replace the target's facial image to a certain extent while preserving the original identity characteristics. This could involve simulating a specific style, emotion, or situation to enhance visual effects, create a specific atmosphere, or meet entertainment needs. Therefore, it is necessary to ensure that identity information remains unchanged during the facial generation process; whether identity information changes can be reflected by calculating similarity.

[0236] Specifically, even after multiple denoising operations, the first undetermined facial image should resemble the target face; that is, the similarity between the first undetermined facial image and the target face should be greater than a similarity threshold. To verify the similarity between the first undetermined face and the target face, the similarity between the model parameter values ​​used to characterize the target face and the first undetermined face can be calculated.

[0237] This application does not specifically limit the method of calculating similarity, such as based on Euclidean distance, cosine similarity, etc. In this application embodiment, the similarity between the model parameter value and the first undetermined facial image is calculated by training the identity measurement model. This will be explained based on I1-I3 later, and will not be repeated here.

[0238] H3: Determine the degree of denoising in the (n+1)th denoising step based on similarity.

[0239] To prevent identity inconsistency issues during the denoising process, the denoising level of the (n+1)th denoising step can be determined based on the similarity between the model parameter values ​​and the first candidate face image. This ensures that the similarity between the face in the candidate face image obtained by performing the (n+1)th denoising step based on this denoising level and the target face is greater than a similarity threshold, thereby avoiding identity inconsistency issues during face synthesis.

[0240] The degree of denoising refers to the reduction in the amount of noise in a noisy image. The greater the degree of denoising, the less noise is in the resulting image of the face to be determined. This application does not specifically limit the method of determining the degree of denoising, for example, the degree of denoising can be determined by formula (15).

[0241]

[0242] Where grad is the gradient, used to indicate the degree of noise reduction; score t z represents the similarity between the first undetermined facial image obtained from the nth denoising step and the model parameter values. t The first undetermined facial image is obtained from the nth denoising step.

[0243] H4: Based on the first fusion feature and the denoising degree of the (n+1)th denoising, the first undetermined facial image is denoised to obtain the second undetermined facial image.

[0244] The first fusion feature is used to guide the (n+1)th denoising process, which uses the denoising degree of the (n+1)th denoising to denoise the first undetermined facial image, thereby obtaining the second undetermined facial image, thus completing the (n+1)th denoising process.

[0245] For example, the noise used in the (n+1)th denoising iteration after the denoising level update can be expressed as Equation (16).

[0246]

[0247] in, The noise used for the (n+1)th denoising step; ∈ θ (z t α represents the noise used to denoise the first undetermined facial image obtained through the nth denoising step using the U-Net model; t grad represents the noise ratio in the nth denoising iteration of the U-Net model, used to determine the intensity of noise removal in the nth denoising iteration of the U-Net model; grad is the gradient.

[0248] Therefore, the noise obtained from the (n+1)th denoising step can be expressed as formula (17).

[0249]

[0250] Among them, z t-1 The second undetermined facial image obtained from the (n+1)th denoising step; α t-1 The noise proportion in the (n+1)th denoising iteration of the U-Net model is used to determine the intensity of noise removal in the (n+1)th denoising iteration of the U-Net model; z t The first undetermined facial image obtained from the nth denoising step; α t The noise ratio in the nth denoising step of the U-Net model is used to determine the intensity of noise removal in the nth denoising step of the U-Net model. The noise used for the (n+1)th denoising step.

[0251] See Figure 13 This figure is a schematic diagram of an adjustment noise reduction process provided in an embodiment of this application. Figure 13 In the process, the number of denoising iterations is set to t, and the noisy image z is denoised. t The input is fed into the U-Net model, which continuously denoises the image to be determined, resulting in multiple undetermined facial images. The similarity between each undetermined facial image and the model parameter values ​​is calculated to obtain the similarity between each undetermined facial image and the model parameter values, thereby guiding the next denoising step.

[0252] See Figure 14 This figure is a schematic diagram of a denoising process provided in an embodiment of this application. Noise and conditions are input into a diffusion model to generate a first facial image. The conditions include first text features and first attribute features, etc. During the generation of the first facial image based on the diffusion model, a similarity-guided denoising process can be used to determine the facial image to be determined based on the highest similarity image z. gt Generate a first facial image, thereby improving the accuracy of the generated first facial image.

[0253] Therefore, by introducing the similarity between the denoised face image and the model parameter values ​​obtained in each denoising step, the degree of denoising to be used in the next denoising step is calculated. This ensures that the denoised face image obtained after each denoising step has a high similarity to the target face, thereby avoiding the problem of identity inconsistency during the face synthesis process and improving the accuracy of the generated face image.

[0254] As one possible implementation, the process of calculating similarity based on the aforementioned identity measurement model is described below, see I1-I3 for details. This identity measurement model includes a first encoding model and a second encoding model. Feature extraction is performed through the first encoding model and the second encoding model. After the features are transformed into the same feature space, the similarity between the parametric features and the noise features obtained from the feature extraction is calculated.

[0255] I1: Extract features from the model parameter values ​​based on the first encoding model to obtain parameter features.

[0256] The identity measurement model includes not only a first encoder and a second encoder, but also a similarity calculation between parametric features and noise features.

[0257] See Figure 15 This figure is a schematic diagram of an identity measurement model provided in an embodiment of this application. Figure 15 As shown in Figure (A), the model parameter values ​​p can be... id The input is fed into the first encoding model, and feature extraction is performed using the model parameter values ​​to obtain parameter features.

[0258] I2: Based on the second coding model, feature extraction is performed on the first undetermined facial image to obtain noise features.

[0259] By using the first and second coding models, parametric features and noise features can be transformed into the same feature space, thereby enabling features from different modalities to calculate similarity in the same feature space.

[0260] See also Figure 15 In Figure (A), the first undetermined facial image z t The input is fed into the second coding model, where features are extracted to obtain noise features.

[0261] I3: Determine the similarity between parametric features and noise features.

[0262] See also Figure 15 In Figure (A), the similarity between parametric features and noise features is calculated.

[0263] The training process of the first and second encoding models is described below, see J1-J6 for details.

[0264] J1: Get parameter image sample pairs.

[0265] A parameter-image sample pair includes model parameter value samples and image samples. The model parameter value samples are the model parameter values ​​used for training, and the image samples are the images used for training. Model parameter value samples and image samples in the same parameter-image sample pair are used to describe the same face.

[0266] See also Figure 15 In Figure (B), N pairs of parameter image samples can be obtained. Taking the nth pair of parameter image samples as an example, the parameter values ​​of the model included in the nth pair of parameter image samples can be expressed as follows: The image samples included in the nth pair of parameter image samples can be represented as follows:

[0267] J2: Obtain the noisy image sample based on the image sample and the number of times noise is added.

[0268] By adding noise to the image sample multiple times, a noisy image sample is obtained, thus simulating the denoising process from the noisy image sample to the image sample.

[0269] This application does not specifically limit the noise addition method; please refer to the following examples. Figure 15 In Figure (B), a pre-trained noise scheduler can be used to add noise to the image samples by an additional t number of times, thus obtaining a noisy image sample. Continuing with the example of the nth pair of parameter image samples, the noisy image sample obtained after adding noise to the image samples in the nth pair of parameter image samples can be represented as follows:

[0270] J3: Extract features from the model parameter value samples based on the initial first coding model to obtain parameter feature samples.

[0271] The initial first encoding model is the first encoding model that has not yet been fully trained.

[0272] J4: Based on the initial second coding model, feature extraction is performed on the noisy image samples to obtain noise feature samples.

[0273] The initial second encoding model is a second encoding model that has not yet been fully trained.

[0274] See also Figure 15 In Figure (B), continuing with the nth pair of parameter image samples as an example, the nth pair of parameter image samples includes the parameter values ​​of the model. The obtained parameter feature samples can be represented as I n The nth pair of parameter image samples includes the image samples to obtain the noisy image sample. The obtained noise feature samples can be represented as L n .

[0275] J5: Determine the predicted similarity between parametric feature samples and noise feature samples.

[0276] The predicted similarity is the similarity between parametric feature samples and noisy feature samples. See also... Figure 15 In Figure (B), the predicted similarity is obtained by calculating the dot product between the parameter feature samples and the noise feature samples. Thus, the similarity matrix can be obtained through N pairs of parameter image samples.

[0277] J6: Based on the training direction of maximizing predicted similarity, adjust the model parameters of the initial first encoding model and the initial second encoding model to obtain the first encoding model and the second encoding model.

[0278] The model parameter values ​​and image samples come from the same face. Based on the training direction that maximizes the predicted similarity, the model parameters of the initial first encoding model are adjusted to obtain the first encoding model. The model parameters of the initial second encoding model are adjusted to obtain the second encoding model. Then, the parameter features and noise features can be transformed into the same feature space through the first and second encoding models, so that the similarity of features of different modalities can be calculated in the same feature space.

[0279] Therefore, the identity assessment model not only includes a first encoding model and a second encoding model, but also calculates the similarity between parametric features and noise features. By training the first and second encoding models, parametric features and noise features belonging to different modalities can be transformed into the same feature space, thus calculating the similarity between parametric features and noise features more accurately. This improves the accuracy of the denoising process, avoids inconsistencies in identity during face synthesis, and enhances the accuracy of the generated facial images.

[0280] As one possible approach, since the noise removed during the denoising process varies in size, the focus of denoising differs depending on the noise level. For example, during the first one-third to two-thirds of the denoising process, some noise has already been removed, and the process then focuses more on determining the facial layout, such as the position of facial features. From the second-thirds to the completion of denoising, the noise is generally smaller, and the process then focuses more on refining the facial identity information, such as making the generated face more similar to the target face. Therefore, to adapt to the different characteristics of different stages of denoising, different identity measurement models can be trained for each stage. Training methods can be found in J1-J6 mentioned above, to obtain identity measurement models more suitable for different denoising stages. The following section uses two identity measurement models as examples for illustration.

[0281] The two identity measurement models are the first identity measurement model and the second identity measurement model. The noise addition and denoising processes are opposites; the denoising process removes noise from a noisy image, while the noise addition process adds noise to the image. Although the aforementioned noise scheduler adds noise to image samples, it is intended to simulate the denoising process.

[0282] The total number of denoising attempts minus the number of times noise was added during the training of the first identity measurement model (equivalent to the number of denoising attempts) is greater than the first threshold, and this value is less than or equal to the second threshold. The total number of denoising attempts minus the number of times noise was added during the training of the second identity measurement model is greater than the second threshold, and this value is less than or equal to the third threshold. The first threshold is less than the second threshold, and the second threshold is less than the third threshold. The usage process of the two identity measurement models is explained below.

[0283] If the number of denoising attempts n is greater than the first threshold and n is less than or equal to the second threshold, it indicates that the denoising process is performed earlier. In this case, a larger number of noise addition attempts are required. The first identity measurement model can be used to determine the similarity between the model parameter values ​​and the first undetermined facial image.

[0284] If n is greater than the second threshold and n is less than or equal to the third threshold, it indicates that the denoising process is relatively late. At this time, a smaller number of noise additions are required, and a second identity measurement model can be used to determine the similarity between the model parameter values ​​and the first undetermined facial image. The first identity measurement model and the second identity measurement model are different models.

[0285] It should be noted that although both the first identity measurement model and the second identity measurement model can be trained using the methods described above (J1-J6), they need to be trained with different training data in order to make them applicable to different stages.

[0286] Specifically, the training data for the first identity measurement model includes a noise-adding count greater than a first threshold and less than or equal to a second threshold. Training is then performed on noisy image samples (as described in J2 above) obtained based on this noise-adding count. Similarly, the training data for the second identity measurement model includes a noise-adding count greater than a second threshold and less than or equal to a third threshold. Training is then performed on noisy image samples (as described in J2 above) obtained based on this noise-adding count. Thus, by training with noisy image samples from different stages, first and second identity measurement models suitable for different stages are obtained.

[0287] One possible approach is to obtain the total number of denoising attempts, set one-third of the total number of denoising attempts as the first threshold, two-thirds of the total number of denoising attempts as the second threshold, and the total number of denoising attempts as the third threshold. This allows the denoising level to be guided only in the latter two-thirds of the denoising process, avoiding excessive noise content in the first third of the denoising process. Over-guiding denoising would reduce the accuracy of the generated face, and it also shortens the computation time of the denoising process, thus improving denoising efficiency.

[0288] As one possible approach, it is possible not only to guide the degree of denoising in the last two-thirds of the denoising process, but also to set different identity measurement models for different stages of denoising, taking into account the different characteristics of each stage.

[0289] Specifically, the last two-thirds of the denoising process is divided into multiple stages. For example, the first stage covers one-third to two-thirds of the denoising process, while the second stage covers two-thirds to the completion of denoising. The first stage uses one identity measurement model, and the second stage uses another. This allows for guidance on the denoising level only in the last two-thirds of the process, with different identity measurement models used in different stages. This not only shortens the computation time and improves denoising efficiency but also enhances the accuracy of facial recognition in the image. The following explanation uses a total of T denoising iterations as an example.

[0290] when When the first identity measurement model is used to calculate the similarity, see formula (18).

[0291]

[0292] Where t is the number of denoising attempts, and score is... t z represents the similarity between the first undetermined facial image obtained from the nth denoising step and the model parameter values. t The first undetermined facial image obtained from the nth denoising step; To calculate the similarity between a first undetermined facial image and the model parameter values ​​using a first identity measurement model.

[0293] when When the first identity measurement model is used to calculate the similarity, see formula (19).

[0294]

[0295] Among them, score t z represents the similarity between the first undetermined facial image obtained from the nth denoising step and the model parameter values. t The first undetermined facial image obtained from the nth denoising step; To calculate the similarity between the first undetermined facial image and the model parameter values ​​using a second identity measurement model.

[0296] Therefore, based on the characteristics of the denoising process, the denoising process is divided into multiple stages, and different identity measurement models are trained for different stages. Then, different identity measurement models are used in the denoising process, so that similarity is calculated based on identity measurement models that are more suitable for different denoising stages, thereby improving the accuracy of the denoising process and the accuracy of the generated facial images.

[0297] To facilitate a further understanding of the technical solutions provided in the embodiments of this application, the following description takes the execution subject of the facial image generation method provided in the embodiments of this application as a server and the training of a facial recognition model as an example to provide an overall exemplary introduction to the facial image generation method.

[0298] The application process of the facial recognition model will be explained below.

[0299] See Figure 16 This figure is a schematic diagram illustrating the application process of a facial recognition model provided in an embodiment of this application. Figure 16 In the process, users first take a pre-taken registration photo. Face detection is performed on the registration photo, and the Region of Interest (ROI) is extracted. The result is sent to the backend and stored in the registration database. When a user wants to undergo facial recognition, a photo of the user is taken. Face detection is performed on the recognition photo, and the ROI is extracted. The result is sent to the backend, and the results are compared with the results of every face stored in the registration database to ultimately determine the user's identity. The recognition result is then returned to the frontend.

[0300] As facial recognition technology is widely used in practical applications such as payment and identity verification, it faces the problem that facial data may be relatively scarce for certain groups or under certain conditions, resulting in lower accuracy of the trained facial recognition models.

[0301] Based on this, the facial image generation method provided in this application embodiment can generate multiple facial images for the same face through prompt text, thereby expanding the facial images used for training. In turn, the facial recognition model can be trained with images of different styles but representing the same face, thereby improving the accuracy of the facial recognition model.

[0302] The following section explains the process of facial image augmentation based on a facial image generation model.

[0303] First, the training process of the facial image generation model will be explained. This model includes a conditional projection network, a first attention network, a second attention network, and an identity assessment model. The first and second attention networks are trained in stages, while the conditional projection network is trained together with the first attention network. The training process is detailed below. Figure 11 The training process of the second attention network in the illustrated embodiment is described below. Figure 12 The training process of the identity measurement model shown in the embodiment is described in [reference needed]. Figure 15 The example shown.

[0304] The application process of the facial image generation model is explained below.

[0305] S1: Obtain the facial image dataset and prompt text.

[0306] The facial image dataset includes multiple facial images, each corresponding to a different face, and each facial image corresponds to a prompt image. The prompt text is used to describe the modification requirements for the face. Different faces may correspond to different prompt texts, or different faces may correspond to the same prompt text. This application embodiment does not specifically limit this.

[0307] S2: Extract model parameter values ​​and keypoint parameter values, and convert them into first attribute features and first pose features.

[0308] For each facial image in the facial image dataset, model parameter values ​​are extracted to obtain the model parameter values ​​corresponding to each facial image. These model parameter values ​​are then converted into first attribute features. See details below. Figure 5 and Figure 6 The example shown.

[0309] S3: Extract the prompt text and convert it into the first text feature.

[0310] The first text feature, the first attribute feature, and the first pose feature all belong to the same text feature space.

[0311] S4: Generate a first facial image based on the first text features, the first attribute features, and the first pose features.

[0312] The process of fusing the first text feature, the first attribute feature, and the first pose feature through the cross-attention mechanism can be found in D1-D3. The process of denoising the fused features can be found in H1-H4.

[0313] S5: Get the edited text.

[0314] The edit text includes text for modifying model parameter values ​​and text for modifying prompts.

[0315] S6: Extract the edited text to obtain the modified text for the model parameter values ​​and the modified text for the prompt text.

[0316] S7: Obtain the updated model parameter values ​​and convert them into the updated first attribute features, and obtain the updated prompt text and convert it into the updated first text features.

[0317] The modified model parameter values ​​are obtained based on the modified text of the model parameter values, and the previously obtained model parameter values ​​are updated to obtain the updated model parameter values, which are then converted into the updated first attribute feature. Similarly, the updated prompt text is obtained based on the modified prompt text and converted into the updated first text feature.

[0318] S8: Generate a second facial image based on the first pose features, the updated first attribute features, and the updated first text features.

[0319] For details regarding S5-S8, please refer to A1-A4.

[0320] To evaluate the performance of this application's embodiments in editing facial images, taking faces as an example, three methods were compared: 1) a concept learning-based facial attribute editing method (Imagic: Text-Based Real Image Editing with Diffusion Models); 2) an identity-based facial attribute editing method (InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing); and 3) PreciseControl (PreciseControl: Enhancing Text-to-Image Diffusion Models with Fine-Grained Attribute Control), which performs attribute editing by adding a global latent editing direction to the W+ space of the input image's w-encoding. Quantitative results are shown in Table 1.

[0321] Table 1

[0322] Imagic InterfaceGAN <![CDATA[PreciseControl+W + ]]> This application index ΔCLIP LPIPS CS ΔCLIP LPIPS CS ΔCLIP LPIPS CS ΔCLIP LPIPS CS beard 4.184 0.533 0.492 2.644 0.221 0.732 2.473 0.185 0.731 2.307 0.184 0.763 age 4.432 0.571 0.474 1.359 0.22 0.744 1.777 0.209 0.698 2.617 0.126 0.885 smile 1.945 0.499 0.666 1.8 0.188 0.804 1.104 0.19 0.779 2.48 0.187 0.808

[0323] The evaluation metric is CLIP, which measures the alignment between the prompt text and the generated facial image. ΔCLIP ); Use cosine similarity of facial features to measure identity similarity (CS) between the input and generated images; Use LPIPS and identity similarity to measure changes in the image during editing.

[0324] For ideal fine-grained attribute editing, a higher ΔCLIP indicates meaningful editing, while lower LPIPS and higher CS indicate preservation of source identity. As shown in Table 1, the embodiments of this application achieve the lowest LPIPS score while maintaining a high ΔCLIP, indicating highly decoupled attribute editing. Neither of the two text-based editing methods preserved image regions. The embodiments of this application achieved a high CS score in editing with a high ΔCLIP, indicating identity-preserving attribute editing.

[0325] In response to the facial image generation method described above, this application also provides a corresponding facial image generation apparatus so that the above facial image generation method can be applied and implemented in practice.

[0326] See Figure 17 This figure is a schematic diagram of the structure of a facial image generation device provided in an embodiment of this application. Figure 17 As shown, the facial image generation device 1700 includes: an acquisition unit 1701, an extraction unit 1702, a conversion unit 1703, and a generation unit 1704;

[0327] The acquisition unit 1701 is used to acquire prompt text and prompt image, wherein the prompt text is text describing the modification requirements and the prompt image is an image including the target face;

[0328] The extraction unit 1702 is used to extract parameters from the prompt image to obtain model parameter values ​​for characterizing the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters are used to describe different inherent facial attributes of the target face.

[0329] The conversion unit 1703 is used to convert the model parameter values ​​to obtain a first attribute feature that characterizes the intrinsic facial attributes.

[0330] The extraction unit 1702 is also used to extract features from the prompt text to obtain first text features;

[0331] The generation unit 1704 is used to generate a first facial image based on the first text features and the first attribute features. The first facial image is an image obtained by modifying the target face based on the prompt text.

[0332] As can be seen from the above technical solution, the facial image generation apparatus 1700 provided in this application includes an acquisition unit, an extraction unit, a conversion unit, and a generation unit. The acquisition unit acquires prompt text and a prompt image. The extraction unit extracts features from the prompt text to obtain first text features. Model parameters are used to construct a parameterized model of the face; different model parameter values ​​can represent different faces. The extraction unit extracts parameters from the prompt image to obtain model parameter values ​​representing the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters describe different inherent facial attributes of the target face, and different attribute parameter values ​​of the same attribute parameter describe different degrees of the corresponding inherent facial attributes. Therefore, subsequent modifications to the face can be achieved based on any attribute parameter in the model parameters. The conversion unit converts the model parameter values ​​into first attribute features representing the inherent facial attributes. The generation unit then generates a first facial image based on the first text features and the first attribute features.

[0333] Therefore, by introducing a first attribute feature to characterize intrinsic facial attributes, facial attributes are refined into intrinsic and extrinsic facial attributes. During the generation of the first facial image, greater attention is paid to the intrinsic facial attributes that need to be preserved and the intrinsic or extrinsic facial attributes that need to be modified. This allows for the retention of desired intrinsic facial attributes while meeting modification requirements, improving the accuracy of the generated facial image. Furthermore, different attribute parameters in the model are used to modify different intrinsic facial attributes, enabling modifications to various parts of the face and decoupling the modification of facial parameters, further improving the accuracy of the generated facial image.

[0334] As one possible implementation, the device further includes: an editing unit, used for:

[0335] Obtain the value of the first target attribute parameter corresponding to the first target attribute parameter, wherein the first target attribute parameter is one of the plurality of attribute parameters;

[0336] The model parameter values ​​are updated based on the first target attribute parameter values ​​to obtain the first updated model parameter values;

[0337] The parameter values ​​of the first updated model are transformed to obtain the second attribute features;

[0338] A second facial image is obtained based on the first text feature and the second attribute feature. The second facial image is an image obtained by modifying the target face based on the prompt text and the first target attribute parameter value.

[0339] As one possible implementation, the acquisition unit 1701 is further configured to acquire the value of the second target attribute parameter corresponding to the second target attribute parameter, wherein the second target attribute parameter is one of the plurality of attribute parameters;

[0340] The conversion unit 1703 is specifically used for:

[0341] The model parameter values ​​are updated based on the second target attribute parameter values ​​to obtain the second updated model parameter values;

[0342] The parameter values ​​of the second updated model are transformed to obtain the first attribute feature.

[0343] As one possible implementation, the extraction unit 1702 is further configured to:

[0344] Key points are extracted from the prompt image to obtain key point parameter values, which are used to characterize the positions of multiple key points included in the target face.

[0345] The key point parameter values ​​are transformed to obtain a first pose feature used to characterize the pose of the target face;

[0346] The generation unit 1704 is specifically used for:

[0347] A first facial image is generated based on the first pose feature, the first text feature, and the first attribute feature.

[0348] As one possible implementation, the generation unit 1704 is specifically used for:

[0349] By concatenating the first pose feature and the first attribute feature, a first concatenated feature is obtained;

[0350] Based on the first splicing feature and the first text feature, the second fused feature is obtained by fusing them through the cross-attention mechanism.

[0351] The first facial image is obtained by denoising the noisy image based on the second fusion feature, wherein the noisy image is obtained based on the prompt text and noise.

[0352] As one possible implementation, the generation unit 1704 is specifically used for:

[0353] Based on the first text features and the first attribute features, a first fused feature is obtained by fusing them through a cross-attention mechanism;

[0354] The first facial image is obtained by denoising the noisy image based on the first fusion feature, wherein the noisy image is obtained based on the prompt text and noise.

[0355] As one possible implementation, the generation unit 1704 is specifically used for:

[0356] Based on the first fusion feature, the noisy image is denoised multiple times to obtain a first facial image;

[0357] The (n+1)th denoising step is as follows, where n is a positive integer:

[0358] A first undetermined facial image is obtained, which is obtained by performing the nth denoising on the noisy image based on the first fusion feature;

[0359] Determine the similarity between the model parameter values ​​and the first undetermined facial image;

[0360] The degree of denoising in the (n+1)th denoising step is determined based on the similarity.

[0361] Based on the first fusion feature and the denoising degree of the (n+1)th denoising, the first undetermined facial image is denoised to obtain the second undetermined facial image.

[0362] As one possible implementation, the generation unit 1704 is specifically used for:

[0363] If n is greater than the first threshold and n is less than or equal to the second threshold, then the similarity between the model parameter value and the first undetermined facial image is determined by the first identity measurement model, where the first threshold is less than the second threshold.

[0364] If n is greater than the second threshold and n is less than or equal to the third threshold, then the similarity between the model parameter value and the first undetermined facial image is determined by the second identity measurement model. The second threshold is less than the third threshold, and the first identity measurement model and the second identity measurement model are different models.

[0365] As one possible implementation, the apparatus further includes a determining unit for:

[0366] Get the total number of noise reduction attempts;

[0367] The first threshold, the second threshold, and the third threshold are determined based on the total number of denoising attempts. The first threshold is one-third of the total number of denoising attempts, the second threshold is two-thirds of the total number of denoising attempts, and the third threshold is equal to the total number of denoising attempts.

[0368] As one possible implementation, the generation unit 1704 is specifically used for:

[0369] Based on the first encoding model, feature extraction is performed on the model parameter values ​​to obtain parameter features;

[0370] Based on the second coding model, feature extraction is performed on the first undetermined facial image to obtain noise features;

[0371] Determine the similarity between the parameter features and the noise features;

[0372] The method further includes:

[0373] Obtain parameter image sample pairs, wherein the parameter image sample pairs include model parameter value samples and image samples used to describe the same face;

[0374] Based on the image sample and the number of times noise was added, a noisy image sample is obtained;

[0375] Based on the initial first encoding model, feature extraction is performed on the model parameter value samples to obtain parameter feature samples;

[0376] Based on the initial second coding model, feature extraction is performed on the noisy image samples to obtain noise feature samples;

[0377] Determine the predicted similarity between the parameter feature samples and the noise feature samples;

[0378] Based on the training direction that maximizes predicted similarity, the model parameters of the initial first encoding model and the model parameters of the initial second encoding model are adjusted to obtain the first encoding model and the second encoding model.

[0379] As one possible implementation, the cross-attention mechanism is implemented through a first attention network and a second attention network, and the generation unit 1704 is specifically used for:

[0380] The first attribute feature and the image feature are fused according to the first attention network to obtain the first sub-fusion feature, wherein the image feature is a feature obtained based on the prompt image;

[0381] The second attention network is used to fuse the first text features and the image features to obtain the second sub-fused features;

[0382] The first fusion feature is obtained based on the first sub-fusion feature and the second sub-fusion feature.

[0383] As one possible implementation, the device further includes a training unit for:

[0384] Obtain the first noisy image sample and the prompt image sample;

[0385] Feature extraction is performed on the prompt image sample to obtain image feature samples;

[0386] Parameters are extracted from the prompt image samples to obtain model parameter value samples;

[0387] The model parameter value samples are transformed to obtain attribute feature samples;

[0388] The image feature samples and the attribute feature samples are fused according to the initial first attention network to obtain the first sub-feature sample;

[0389] The first noisy image sample is denoised based on the first sub-feature sample to obtain the first predicted face image;

[0390] Based on the difference between the first predicted facial image and the prompt image sample, the model parameters of the initial first attention network are adjusted to obtain the first attention network.

[0391] As one possible implementation, the device further includes a training unit for:

[0392] Obtain labeled cue sample pairs and second image noise samples, wherein the cue sample pairs include cue text samples and cue image samples, and the labels are used to describe the image obtained by modifying the faces included in the cue image samples based on the cue text samples;

[0393] Feature extraction is performed on the prompt text sample to obtain text feature samples;

[0394] The image feature samples and the attribute feature samples are fused according to the first attention network to obtain the second sub-feature samples;

[0395] The image feature samples and the text feature samples are fused according to the initial second attention network to obtain the third sub-feature sample;

[0396] Based on the second sub-feature sample and the third sub-feature sample, a fused feature sample is obtained;

[0397] The second noisy image sample is denoised based on the fused feature samples to obtain the second predicted face image;

[0398] Based on the difference between the second predicted facial image and the label, the model parameters of the initial second attention network are adjusted to obtain the second attention network.

[0399] This application also provides a computer device, which can be a server or a terminal device. The computer device provided in this application will be described below from a hardware implementation perspective. Figure 18 The diagram shown is a schematic of the server's structure. Figure 19 The diagram shown is a structural schematic of the terminal device.

[0400] See Figure 18This figure is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more processors 1422, such as a central processing unit (CPU), memory 1432, and one or more application programs 1442 or data storage media 1430 (e.g., one or more mass storage devices). The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the server 1400.

[0401] Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0402] The steps performed by the server in the above embodiments can be based on this Figure 18 The server structure shown.

[0403] The processor 1422 is used to perform the following steps:

[0404] Obtain prompt text and prompt image, wherein the prompt text is text describing the modification requirements, and the prompt image is an image including the target face;

[0405] The prompt image is subjected to parameter extraction to obtain model parameter values ​​that characterize the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters are used to describe different inherent facial attributes of the target face.

[0406] The model parameter values ​​are transformed to obtain a first attribute feature used to characterize the intrinsic facial attributes;

[0407] Feature extraction is performed on the prompt text to obtain the first text feature;

[0408] A first facial image is generated based on the first text feature and the first attribute feature. The first facial image is an image obtained by modifying the target face based on the prompt text.

[0409] Optionally, the processor 1422 may also execute method steps of any specific implementation of the facial image generation method in the embodiments of this application.

[0410] See Figure 19 This figure is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The description will be based on a smartphone as an example. Figure 19 The diagram shown is a partial structural block diagram of the smartphone, which includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a Wi-Fi module 1570, a processor 1580, and a power supply 1590, among other components. Those skilled in the art will understand that... Figure 19 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0411] The following is combined Figure 19 A detailed introduction to the various components of a smartphone:

[0412] The RF circuit 1510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1580; in addition, it transmits uplink data to the base station.

[0413] The memory 1520 can be used to store software programs and modules, and the processor 1580 runs the software programs and modules stored in the memory 1520 to realize various functions and data processing of the smartphone.

[0414] Input unit 1530 can be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the smartphone. Specifically, input unit 1530 may include touch panel 1531 and other input devices 1532. Touch panel 1531, also known as a touch screen, can collect touch operations on or near the user and drive corresponding connected devices according to a pre-set program. In addition to touch panel 1531, input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0415] The display unit 1540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 1540 may include a display panel 1541, which may optionally be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0416] Smartphones may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Other sensors that smartphones may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be detailed here.

[0417] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, where speaker 1561 converts them into sound signals for output. On the other hand, microphone 1562 converts collected sound signals into electrical signals, which are received by audio circuit 1560, converted into audio data, and then processed by processor 1580 before being transmitted via RF circuit 1510 to, for example, another smartphone, or the audio data can be output to memory 1520 for further processing.

[0418] The processor 1580 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1520, and by calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units.

[0419] The smartphone also includes a power supply 1590 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0420] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0421] In this embodiment of the application, the memory 1520 included in the smartphone can store computer programs and transmit the computer programs to the processor.

[0422] The processor 1580 included in the smartphone can execute the facial image generation method provided in the above embodiments according to the instructions in the computer program.

[0423] This application also provides a computer-readable storage medium for storing a computer program for executing the facial image generation method provided in the above embodiments.

[0424] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the facial image generation method provided in various optional implementations of the above aspects.

[0425] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0426] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0427] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0428] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0429] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating facial images, characterized in that, The method includes: Obtain prompt text and prompt image, wherein the prompt text is text describing the modification requirements, and the prompt image is an image including the target face; The prompt image is subjected to parameter extraction to obtain model parameter values ​​that characterize the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters are used to describe different inherent facial attributes of the target face. The model parameter values ​​are transformed to obtain a first attribute feature used to characterize the intrinsic facial attributes; Feature extraction is performed on the prompt text to obtain the first text feature; A first facial image is generated based on the first text feature and the first attribute feature. The first facial image is an image obtained by modifying the target face based on the prompt text.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the value of the first target attribute parameter corresponding to the first target attribute parameter, wherein the first target attribute parameter is one of the plurality of attribute parameters; The model parameter values ​​are updated based on the first target attribute parameter values ​​to obtain the first updated model parameter values; The parameter values ​​of the first updated model are transformed to obtain the second attribute features; A second facial image is obtained based on the first text feature and the second attribute feature. The second facial image is an image obtained by modifying the target face based on the prompt text and the first target attribute parameter value.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the value of the second target attribute parameter corresponding to the second target attribute parameter, wherein the second target attribute parameter is one of the plurality of attribute parameters; The process of transforming the model parameter values ​​to obtain a first attribute feature used to characterize the intrinsic facial attributes includes: The model parameter values ​​are updated based on the second target attribute parameter values ​​to obtain the second updated model parameter values; The parameter values ​​of the second updated model are transformed to obtain the first attribute feature.

4. The method according to claim 1, characterized in that, The method further includes: Key points are extracted from the prompt image to obtain key point parameter values, which are used to characterize the positions of multiple key points included in the target face. The key point parameter values ​​are transformed to obtain a first pose feature used to characterize the pose of the target face; The step of generating the first facial image based on the first text feature and the first attribute feature includes: A first facial image is generated based on the first pose feature, the first text feature, and the first attribute feature.

5. The method according to claim 4, characterized in that, The step of generating a first facial image based on the first pose feature, the first text feature, and the first attribute feature includes: By concatenating the first pose feature and the first attribute feature, a first concatenated feature is obtained; Based on the first splicing feature and the first text feature, the second fused feature is obtained by fusing them through the cross-attention mechanism. The first facial image is obtained by denoising the noisy image based on the second fusion feature, wherein the noisy image is obtained based on the prompt text and noise.

6. The method according to claim 1, characterized in that, The step of generating the first facial image based on the first text feature and the first attribute feature includes: Based on the first text features and the first attribute features, a first fused feature is obtained by fusing them through a cross-attention mechanism; The first facial image is obtained by denoising the noisy image based on the first fusion feature, wherein the noisy image is obtained based on the prompt text and noise.

7. The method according to claim 6, characterized in that, The step of denoising the noisy image based on the first fusion feature to obtain the first facial image includes: Based on the first fusion feature, the noisy image is denoised multiple times to obtain a first facial image; The (n+1)th denoising step is as follows, where n is a positive integer: A first undetermined facial image is obtained, which is obtained by performing the nth denoising on the noisy image based on the first fusion feature; Determine the similarity between the model parameter values ​​and the first undetermined facial image; The degree of denoising in the (n+1)th denoising step is determined based on the similarity. Based on the first fusion feature and the denoising degree of the (n+1)th denoising, the first undetermined facial image is denoised to obtain the second undetermined facial image.

8. The method according to claim 7, characterized in that, Determining the similarity between the model parameter values ​​and the first undetermined facial image includes: If n is greater than the first threshold and n is less than or equal to the second threshold, then the similarity between the model parameter value and the first undetermined facial image is determined by the first identity measurement model, where the first threshold is less than the second threshold. If n is greater than the second threshold and n is less than or equal to the third threshold, then the similarity between the model parameter value and the first undetermined facial image is determined by the second identity measurement model. The second threshold is less than the third threshold, and the first identity measurement model and the second identity measurement model are different models.

9. The method according to claim 8, characterized in that, The method further includes: Get the total number of noise reduction attempts; The first threshold, the second threshold, and the third threshold are determined based on the total number of denoising attempts. The first threshold is one-third of the total number of denoising attempts, the second threshold is two-thirds of the total number of denoising attempts, and the third threshold is equal to the total number of denoising attempts.

10. The method according to claim 7, characterized in that, Determining the similarity between the model parameter values ​​and the first undetermined facial image includes: Based on the first encoding model, feature extraction is performed on the model parameter values ​​to obtain parameter features; Based on the second coding model, feature extraction is performed on the first undetermined facial image to obtain noise features; Determine the similarity between the parameter features and the noise features; The method further includes: Obtain parameter image sample pairs, wherein the parameter image sample pairs include model parameter value samples and image samples used to describe the same face; Based on the image sample and the number of times noise was added, a noisy image sample is obtained; Based on the initial first encoding model, feature extraction is performed on the model parameter value samples to obtain parameter feature samples; Based on the initial second coding model, feature extraction is performed on the noisy image samples to obtain noise feature samples; Determine the predicted similarity between the parameter feature samples and the noise feature samples; Based on the training direction that maximizes predicted similarity, the model parameters of the initial first encoding model and the model parameters of the initial second encoding model are adjusted to obtain the first encoding model and the second encoding model.

11. The method according to claim 6, characterized in that, The cross-attention mechanism is implemented through a first attention network and a second attention network. Based on the first text feature and the first attribute feature, they are fused through the cross-attention mechanism to obtain the first fused feature, including: The first attribute feature and the image feature are fused according to the first attention network to obtain the first sub-fusion feature, wherein the image feature is a feature obtained based on the prompt image; The second attention network is used to fuse the first text features and the image features to obtain the second sub-fused features; The first fusion feature is obtained based on the first sub-fusion feature and the second sub-fusion feature.

12. The method according to claim 11, characterized in that, The method further includes: Obtain the first noisy image sample and the prompt image sample; Feature extraction is performed on the prompt image sample to obtain image feature samples; Parameters are extracted from the prompt image samples to obtain model parameter value samples; The model parameter value samples are transformed to obtain attribute feature samples; The image feature samples and the attribute feature samples are fused according to the initial first attention network to obtain the first sub-feature sample; The first noisy image sample is denoised based on the first sub-feature sample to obtain the first predicted face image; Based on the difference between the first predicted facial image and the prompt image sample, the model parameters of the initial first attention network are adjusted to obtain the first attention network.

13. The method according to claim 12, characterized in that, The method further includes: Obtain labeled cue sample pairs and second image noise samples, wherein the cue sample pairs include cue text samples and cue image samples, and the labels are used to describe the image obtained by modifying the faces included in the cue image samples based on the cue text samples; Feature extraction is performed on the prompt text sample to obtain text feature samples; The image feature samples and the attribute feature samples are fused according to the first attention network to obtain the second sub-feature samples; The image feature samples and the text feature samples are fused according to the initial second attention network to obtain the third sub-feature sample; Based on the second sub-feature sample and the third sub-feature sample, a fused feature sample is obtained; The second noisy image sample is denoised based on the fused feature samples to obtain the second predicted face image; Based on the difference between the second predicted facial image and the label, the model parameters of the initial second attention network are adjusted to obtain the second attention network.

14. A facial image generation device, characterized in that, The device includes: an acquisition unit, an extraction unit, a conversion unit, and a generation unit; The acquisition unit is used to acquire prompt text and prompt image, wherein the prompt text is text describing the modification requirements and the prompt image is an image including the target face; The extraction unit is used to extract parameters from the prompt image to obtain model parameter values ​​that characterize the target face. The model parameter values ​​include attribute parameter values ​​corresponding to multiple attribute parameters. Different attribute parameters are used to describe different inherent facial attributes of the target face. The conversion unit is used to convert the model parameter values ​​to obtain a first attribute feature that characterizes the intrinsic facial attributes. The extraction unit is also used to extract features from the prompt text to obtain first text features; The generation unit is configured to generate a first facial image based on the first text features and the first attribute features, wherein the first facial image is an image obtained by modifying the target face based on the prompt text.

15. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-13 according to the computer program.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-13.

17. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-13.

Images