Head portrait generation method, system and device and computer readable storage medium
By obtaining and translating the text description information of the target and reference avatar, and extracting facial features, generating target avatars that maintain identity characteristics similar to the reference image, the problem of identity inconsistency when generating cross-attribute avatars is solved and the visual effect of the avatar is improved.
Patent Information
- Application Number
- CN202411851496.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-03-11
AI Technical Summary
When generating cross-attribute avatars, the prior art can easily lead to identity inconsistency and poor visual effects of the generated avatars.
By obtaining the target text description information and reference avatar of the target avatar, translating and feature extraction are performed, and target avatar corresponding to the target text description information, reference text description information, overall facial features and facial attribute features are generated.
Improve the visual effect of generating avatars to ensure that the generated avatars remain similar to the reference image.
Smart Images
Figure CN119600146A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and more specifically, to an avatar generation method, system, device and computer-readable storage medium. Background Art
[0002] Currently, corresponding avatars can be generated based on text descriptions, such as using generative artificial intelligence to generate avatars with cross-attributes such as gender, age, and race. However, the avatars generated across attributes have inconsistent identities, resulting in poor visual effects of the generated avatars.
[0003] In summary, how to improve the visual effect of generating avatars is a problem that needs to be solved urgently by those skilled in the art. Summary of the invention
[0004] The purpose of the present application is to provide an avatar generation method, which can solve the technical problem of how to improve the visual effect of generating an avatar to a certain extent. The present application also provides an avatar generation system, an electronic device and a computer-readable storage medium.
[0005] In order to achieve the above objectives, this application provides the following technical solutions:
[0006] A head portrait generation method, comprising:
[0007] Obtain target text description information of the target avatar to be generated;
[0008] Obtaining a reference avatar for generating the target avatar;
[0009] Translating the reference avatar to obtain reference text description information;
[0010] Extracting overall facial features and facial attribute features of the reference avatar;
[0011] The target avatar is generated corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features.
[0012] Preferably, extracting the overall facial features and facial attribute features of the reference avatar includes:
[0013] Performing facial cropping and alignment on the reference head portrait to obtain a reference facial image;
[0014] Encoding the reference facial image to obtain facial image features;
[0015] Performing feature projection on the facial image features to obtain facial feature word units;
[0016] Taking the facial feature word as the overall facial feature;
[0017] The facial attribute features of the reference avatar are extracted, wherein the facial attribute features include facial feature point coordinates, gender feature information, skin color feature information, body shape feature information, and age feature information.
[0018] Preferably, the generating of the target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features comprises:
[0019] Encoding the target text description information to obtain target text features;
[0020] Encoding the reference text description information to obtain reference text features;
[0021] Concatenating the target text feature and the reference text feature to obtain text feature word-units;
[0022] Performing cross-attention processing on the text feature words to obtain a text processing result;
[0023] Performing cross-attention processing on the facial feature word-units to obtain an image processing result;
[0024] Performing feature projection and splicing on the facial image features and the facial attribute features to obtain attribute feature word-units;
[0025] Get random noise;
[0026] The random noise is processed according to the text processing result, the image processing result and the attribute feature word to generate the target avatar.
[0027] Preferably, the translating the reference avatar to obtain reference text description information includes:
[0028] The reference avatar is translated by a pre-trained diffusion model image interpreter to obtain reference text description information;
[0029] The step of encoding the target text description information to obtain target text features, encoding the reference text description information to obtain reference text features, and concatenating the target text features and the reference text features to obtain text feature word units includes:
[0030] The target text description information and the reference text description information are respectively encoded by the text compiler of the diffusion model to obtain corresponding target text features and reference text features;
[0031] The target text feature and the reference text feature are concatenated through the text feature concatenation layer of the diffusion model to obtain text feature word-units;
[0032] Among them, the text compiler of the diffusion model is connected to the image interpreter and the text feature splicing layer respectively.
[0033] Preferably, encoding the reference facial image to obtain facial image features includes:
[0034] Encoding the reference facial image by using the facial encoder of the diffusion model to obtain facial image features;
[0035] The extracting facial attribute features of the reference avatar includes:
[0036] Extracting facial attribute features of the reference avatar through the attribute control network of the diffusion model;
[0037] The cross-attention processing is performed on the text feature word to obtain a text processing result, and the cross-attention processing is performed on the facial feature word to obtain an image processing result, including:
[0038] Performing cross-attention processing on the text feature word-units through the text cross-attention module of the diffusion model to obtain a text processing result;
[0039] Performing cross-attention processing on the facial feature word-units through the image cross-attention module of the diffusion model to obtain an image processing result;
[0040] The step of projecting and concatenating the facial image features and the facial attribute features to obtain attribute feature word-units includes:
[0041] Performing feature projection and concatenation on the facial image features and the facial attribute features through the attribute control network to obtain attribute feature word-units;
[0042] Wherein, the attribute control network is connected to the facial encoder.
[0043] Preferably, the processing of the random noise according to the text processing result, the image processing result and the attribute feature word to generate the target avatar includes:
[0044] Processing the attribute feature word-element through the identity-preserving network of the diffusion model to obtain an attribute feature processing result;
[0045] The text processing result, the image processing result, the attribute feature processing result and the random noise are processed by the image generation network of the diffusion model to generate the target avatar;
[0046] Among them, the identity preserving network and the image generation network are both U-shaped network structures; the output layer of the attribute control network is connected to the seventh layer of the identity preserving network, and the fourth to seventh layers of the identity preserving network are respectively connected to the same layers in the image generation network; in the identity preserving network, the first layer is connected to the seventh layer, the second layer is connected to the sixth layer, and the third layer is connected to the fifth layer.
[0047] Preferably, the training process of the diffusion model includes:
[0048] Get training text description information and training avatar;
[0049] Apply the training text description information and the training avatar to train the initial diffusion model, and control the timestamp, the denoising result at the current timestamp, the text feature word unit, the facial feature word unit and the attribute feature word unit during the training process;
[0050] Acquiring target noise generated by the diffusion model;
[0051] generating a loss function value of the diffusion model according to a Euclidean norm between the target noise and the random noise;
[0052] The diffusion model is adjusted according to the loss function value to obtain a trained diffusion model.
[0053] A head portrait generation system, comprising:
[0054] A first acquisition module is used to acquire target text description information of a target avatar to be generated;
[0055] A second acquisition module, used to acquire a reference avatar for generating the target avatar;
[0056] A first translation module, used to translate the reference avatar to obtain reference text description information;
[0057] A first extraction module, used to extract the overall facial features and facial attribute features of the reference avatar;
[0058] The first generating module is used to generate the target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features.
[0059] An electronic device, comprising:
[0060] Memory for storing computer programs;
[0061] A processor is used to implement the steps of any of the above-mentioned avatar generation methods when executing the computer program.
[0062] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned avatar generation methods are implemented.
[0063] The present application provides a method for generating an avatar, which includes obtaining target text description information of a target avatar to be generated; obtaining a reference avatar for reference in generating the target avatar; translating the reference avatar to obtain reference text description information; extracting the overall facial features and facial attribute features of the reference avatar; and generating a target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features. The present application translates the reference avatar to obtain reference text description information, describes the reference avatar from a text perspective, can provide prompts for the avatar generation process, and extracts the overall facial features and facial attribute features of the reference avatar. Since the overall facial features can supplement the identity information that cannot be fully described in the text from an image perspective, and the facial attribute features can capture the facial detail features in the reference image in a fine-grained manner, the reference text description information, the overall facial features and the facial attribute features can enhance the prompt information to generate a target avatar that maintains identity features similar to the reference image, thereby improving the visual effect of the generated avatar. The present application provides an avatar generation system, an electronic device and a computer-readable storage medium that also solve corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0065] Figure 1 A flowchart of a method for generating an avatar provided in an embodiment of the present application;
[0066] Figure 2 Schematic diagram of generating avatars for applying diffusion model;
[0067] Figure 3 A schematic diagram of the structure of an avatar generation system provided in an embodiment of the present application;
[0068] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0069] Figure 5 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] See also Figure 1 , Figure 1 A flowchart of a method for generating an avatar provided in an embodiment of the present application.
[0072] An avatar generation method provided in an embodiment of the present application may include the following steps:
[0073] Step S101: obtaining target text description information of the target avatar to be generated.
[0074] Step S102: Acquire a reference avatar for generating a target avatar.
[0075] In actual applications, in the process of generating avatars, it is necessary to use the reference avatar as a benchmark and adjust the reference avatar according to the text description information of the avatar to be generated. Therefore, it is necessary to obtain the target text description information of the target avatar to be generated and the reference avatar for reference in generating the target avatar. The target text description information refers to the description information of the generated target avatar. The target text description information, reference avatar, type and content of the target avatar can all be flexibly determined according to the application scenario.
[0076] Step S103: Translate the reference avatar to obtain reference text description information.
[0077] Step S104: extracting the overall facial features and facial attribute features of the reference avatar.
[0078] Step S105: Generate a target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features.
[0079] In practical applications, although the reference avatar can be modified directly according to the target text description information, the adaptability of the generated result across attributes cannot be well controlled, and it is easy to produce a generated result that is not similar to the original. In order to avoid this situation, the reference avatar can be translated first to obtain the reference text description information to describe the reference avatar from a textual perspective, and then the overall facial features and facial attribute features of the reference avatar can be extracted to reflect the facial features of the reference avatar from an overall perspective, that is, from a coarse-grained perspective, and the facial attribute features can be used to reflect the facial features of the avatar from an attribute perspective, that is, from a fine-grained perspective. Finally, the reference avatar only needs to be adjusted according to the target text description information, the reference text description information, the overall facial features and the facial attribute features, so that a target avatar similar to the reference avatar and corresponding to the target text description information can be generated. It should be noted that facial attribute features refer to the features of a certain attribute of the face, and their types can be determined according to actual needs.
[0080] In a specific application scenario, in the process of extracting the overall facial features and facial attribute features of a reference avatar, the reference avatar can be cropped and aligned to obtain a reference facial image to highlight the facial image information in the reference image; the reference facial image can be encoded to obtain facial image features; the facial image features can be feature projected to obtain facial feature words; the facial feature words are used as overall facial features; and the facial attribute features of the reference avatar are extracted, the facial attribute features including facial feature point coordinates, gender feature information, skin color feature information, body shape feature information, and age feature information. Among them, the coordinates of facial feature points may include the coordinates of key positions such as eyes, nose, mouth, eyebrows, and jaw, so as to control the alignment of the facial key point positions of the generated target avatar with the reference avatar, and ensure the similarity between the target avatar and the reference avatar in facial features and face shape; gender feature information may include gender labels, segmentation maps of hair, beard, etc., so as to handle the generation of avatars in cross-gender scenarios, such as in the female-to-male scenario, generating short hair in the hair segmentation area, and filling the remaining area with context-adaptive filling; in the male-to-female scenario, eliminating the beard segmentation area, and naturally blending it with the skin area, etc.; skin color feature information can be used to process the scene where the skin color changes from yellowish to darker, and not only the skin color adaptation of the facial skin is required, but also the generation of the avatar needs to be paid attention to. The skin color of the exposed face can be adapted, especially the natural connection between the neck skin color and the facial skin color; the body feature information can include contour information to handle the avatar generation in the scene of body shape change. For example, in the generation scenario of weight gain / weight loss, the contour information provides the basis for the change of face shape and body shape, and the reference avatar can be modified with the minimum cost to better maintain the identity of the generated result; the age feature information can include facial wrinkle and line information to handle the avatar generation in the scene of age change. For example, a universal wrinkle and line template can be used to support the aging generation in the scene of excessive facial skin refining. In addition, the hair segmentation result in the gender feature information can also be reused in the age feature information to support hair color adaptation in the scene of extreme age change.
[0081] In a specific application scenario, in the process of generating a target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features, the target text description information can be encoded to obtain the target text features, the reference text description information can be encoded to obtain the reference text features, the target text features and the reference text features can be spliced to obtain text feature words, so as to fuse the semantic features of the target text description information and the reference image and enhance the image prompting ability; the text feature words are cross-attention processed to obtain the text processing results, so as to add an internal self-attention mechanism to guide the avatar generation process from the target text description information to the reference text description information; the facial feature words are cross-attention processed to obtain the image processing results, so as to supplement the identity information that cannot be fully described in the text features, that is, the appearance information, and enhance the prompting information so that the generated target avatar keeps the identity features similar to the reference image; the facial image features and the facial attribute features are feature projected and spliced to obtain the attribute feature words; random noise is obtained; the random noise is processed according to the text processing results, the image processing results and the attribute feature words to generate the target avatar.
[0082] In practical applications, the head portrait generation method provided by the present application can be implemented with the help of a neural network model. For example, the existing diffusion model can be improved to implement the head portrait generation method of the present application. Taking portrait generation as an example, this process can be as follows: Figure 2 As shown, random noise is input through multiple rounds of controllable information under the diffusion model, and the denoising process is continuously learned to generate images. The controllable information mainly includes two parts of input: a reference image and a text description of the required generated avatar. The output is a generated avatar that satisfies the identity similarity with the reference avatar and the attribute consistency of the text description. At this time, in the process of translating the reference avatar to obtain the reference text description information, the reference avatar can be translated by the pre-trained diffusion model image interpreter to obtain the reference text description information; the target text description information is encoded to obtain the target text feature, the reference text description information is encoded to obtain the reference text feature, and the target text feature and the reference text feature are spliced to obtain the text feature word unit. The text compiler of the diffusion model can encode the target text description information and the reference text description information respectively to obtain the corresponding target text feature and reference text feature; the target text feature and the reference text feature are spliced by the text feature splicing layer of the diffusion model to obtain the text feature word unit; wherein the text compiler of the diffusion model is connected to the image interpreter and the text feature splicing layer respectively.
[0083] Correspondingly, in the process of encoding the reference facial image to obtain facial image features, the reference facial image can be encoded by the facial encoder of the diffusion model to obtain facial image features; in the process of extracting facial attribute features of the reference avatar, the facial attribute features of the reference avatar can be extracted by the attribute control network of the diffusion model; cross-attention processing is performed on text feature words to obtain text processing results, and cross-attention processing is performed on facial feature words to obtain image processing results. The text cross-attention module of the diffusion model can be used to perform cross-attention processing on text feature words to obtain text processing results, and the image cross-attention module of the diffusion model can be used to perform cross-attention processing on facial feature words to obtain image processing results; feature projection and splicing are performed on facial image features and facial attribute features. In the process of obtaining attribute feature words, the attribute control network can be used to project and concatenate facial image features and facial attribute features to obtain attribute feature words; wherein, the attribute control network is connected to the facial encoder, and the attribute control network can be implemented by a convolutional neural network, which analyzes the input reference image, outputs facial spatial position information, and attribute information such as gender, age, race, and body shape, and captures the human attribute characteristics of the input image in a finer granularity. These attributes plus the facial features obtained by the facial encoder are used as conditional control inputs, and after encoding, they pass through the feature projection layer to obtain attribute feature words that are sent to the identity preservation network, which focuses on specific attribute features under different generation tasks to ensure that the attributes of the generation results under the same attribute generation task remain consistent, and optimize the adaptability of attribute transformation under cross-attribute generation tasks.
[0084] Correspondingly, in the process of processing random noise according to text processing results, image processing results and attribute feature words to generate a target avatar, the attribute feature words can be processed by the identity preservation network of the diffusion model to obtain the attribute feature processing results; the text processing results, image processing results, attribute feature processing results and random noise are processed by the image generation network of the diffusion model to generate the target avatar; wherein the identity preservation network and the image generation network can both be existing U-shaped network structures; the output layer of the attribute control network is connected to the seventh layer of the identity preservation network, and the fourth to seventh layers of the identity preservation network are respectively connected to the same layers in the image generation network, that is, the fourth layer of the identity preservation network is connected to the fourth layer of the image generation network, and the remaining layers are connected by analogy; in the identity preservation network, the first layer is connected to the seventh layer, the second layer is connected to the sixth layer, and the third layer is connected to the fifth layer. It should be noted that the identity-preserving network uses additional conditional control to encode the complex features in the input image as conditional input to guide the pre-trained diffusion model to generate the content of the image, so as to ensure that the image generation network takes into account the control of both the image and the text, and the identity-preserving network is completely guided by the identity features without any text information. Therefore, the identity-preserving network can focus on the identity feature representation in the reference image and is not affected by the general description of the background, environment, and text, so that the generated results can more accurately reflect the identity characteristics of the individual.
[0085] In specific application scenarios, the loss function of the diffusion model of the present application can be flexibly determined as needed. For example, the diffusion model can be trained by attribute condition control so that the network learns in a direction consistent with the random image noise at the current timestamp and the image noise generated by the parameters under conditional control, and finally learns the model denoising process to generate better image results. That is, the training process of the diffusion model includes: obtaining training text description information and training avatars; using the training text description information and training avatars to train the initial diffusion model, and controlling the timestamp, denoising results at the current timestamp, text feature words, facial feature words and attribute feature words during the training process; obtaining the target noise generated by the diffusion model; generating the loss function value of the diffusion model based on the Euclidean norm between the target noise and the random noise, and the generation formula of the loss function value can be Among them, Loss represents the loss function value, t represents the timestamp, z t represents the denoising result at the current timestamp, C text Represents the text feature word, C face Represents facial feature words, C attr represents the attribute feature word, ∈ represents random noise, which can be Gaussian noise with (0,1) as the noise, etc., ∈ θrepresents the noise generated by the network parameters under conditional control; ||||2 represents the Euclidean norm; the diffusion model is adjusted according to the loss function value to obtain the trained diffusion model.
[0086] It should be noted that in the process of training the diffusion model, considering that the present application separates the cross-attention layer of image prompts and text prompts, the framework no longer updates the parameters of the two synchronously, but adjusts the weights of the image control conditions more flexibly, making the network training objectives more targeted and controllable, that is, only the image-related modules can be updated during the training process, and the parameters of the identity-preserving network and the image cross-attention module based on attribute control are continuously optimized; the facial encoder, text encoder, and text-to-image image generation model can all use pre-trained models, which are frozen during the training process to ensure flexibility. And the attribute control network can be updated in combination with specific scenarios. It can be a plug-and-play pre-trained model or a de novo training based on scene data, which is not limited here.
[0087] The present application provides a method for generating an avatar, which includes obtaining target text description information of a target avatar to be generated; obtaining a reference avatar for reference in generating the target avatar; translating the reference avatar to obtain reference text description information; extracting the overall facial features and facial attribute features of the reference avatar; and generating a target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features. The present application translates the reference avatar to obtain reference text description information, describes the reference avatar from a textual perspective, can provide prompts for the avatar generation process, and extracts the overall facial features and facial attribute features of the reference avatar. Since the overall facial features can supplement the identity information that cannot be fully described in the text from an image perspective, and the facial attribute features can capture the facial detail features in the reference image in a fine-grained manner, the reference text description information, the overall facial features and the facial attribute features can enhance the prompt information to generate a target avatar that maintains identity features similar to the reference image, thereby improving the visual effect of the generated avatar.
[0088] See also Figure 3 , Figure 3 A schematic diagram of the structure of an avatar generation system provided in an embodiment of the present application.
[0089] An avatar generation system provided in an embodiment of the present application may include:
[0090] The first acquisition module 101 is used to acquire target text description information of the target avatar to be generated;
[0091] The second acquisition module 102 is used to acquire a reference avatar for generating a target avatar;
[0092] A first translation module 103 is used to translate the reference avatar to obtain reference text description information;
[0093] A first extraction module 104, used to extract the overall facial features and facial attribute features of the reference avatar;
[0094] The first generating module 105 is used to generate a target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features.
[0095] In an avatar generation system provided in an embodiment of the present application, a first extraction module may include:
[0096] A first processing unit is used to perform facial cropping and alignment on the reference head portrait to obtain a reference facial image;
[0097] A first encoding unit, used for encoding a reference facial image to obtain facial image features;
[0098] A first projection unit, used for performing feature projection on facial image features to obtain facial feature word units;
[0099] A second processing unit is used to use facial feature words as overall facial features;
[0100] The first extraction unit is used to extract facial attribute features of the reference avatar, where the facial attribute features include facial feature point coordinates, gender feature information, skin color feature information, body shape feature information, and age feature information.
[0101] An embodiment of the present application provides an avatar generation system, wherein the first generation module may include:
[0102] The second encoding unit is used to encode the target text description information to obtain the target text features; encode the reference text description information to obtain the reference text features;
[0103] A first concatenation unit is used to concatenate the target text feature and the reference text feature to obtain a text feature word unit;
[0104] The third processing unit is used to perform cross-attention processing on the text feature word unit to obtain a text processing result; and perform cross-attention processing on the facial feature word unit to obtain an image processing result;
[0105] A second concatenation unit is used to perform feature projection and concatenation on facial image features and facial attribute features to obtain attribute feature word-units;
[0106] A first acquisition unit, used for acquiring random noise;
[0107] The fourth processing unit is used to process the random noise according to the text processing result, the image processing result and the attribute feature word to generate a target avatar.
[0108] In an avatar generation system provided by an embodiment of the present application, a first translation module may be used to: translate a reference avatar through a pre-trained image interpreter of a diffusion model to obtain reference text description information;
[0109] The first generation module can be used to: respectively encode the target text description information and the reference text description information through the text compiler of the diffusion model to obtain corresponding target text features and reference text features; splice the target text features and the reference text features through the text feature splicing layer of the diffusion model to obtain text feature word units;
[0110] Among them, the text compiler of the diffusion model is connected to the image interpreter and the text feature concatenation layer respectively.
[0111] In an avatar generation system provided by an embodiment of the present application, a first extraction module can be used to: encode a reference facial image through a facial encoder of a diffusion model to obtain facial image features; extract facial attribute features of the reference avatar through an attribute control network of the diffusion model;
[0112] The first generation module can be used to: perform cross-attention processing on text feature words through the text cross-attention module of the diffusion model to obtain text processing results; perform cross-attention processing on facial feature words through the image cross-attention module of the diffusion model to obtain image processing results; perform feature projection and splicing on facial image features and facial attribute features through the attribute control network to obtain attribute feature words; wherein the attribute control network is connected to the facial encoder.
[0113] An avatar generation system provided by an embodiment of the present application, wherein a first generation module can be used to: process attribute feature words through an identity preservation network of a diffusion model to obtain an attribute feature processing result; process a text processing result, an image processing result, an attribute feature processing result and random noise through an image generation network of a diffusion model to generate a target avatar;
[0114] Among them, both the identity preserving network and the image generation network have U-shaped network structures; the output layer of the attribute control network is connected to the seventh layer of the identity preserving network, and the fourth to seventh layers of the identity preserving network are respectively connected to the same layers in the image generation network; in the identity preserving network, the first layer is connected to the seventh layer, the second layer is connected to the sixth layer, and the third layer is connected to the fifth layer.
[0115] The avatar generation system provided in the embodiment of the present application may further include:
[0116] The first training module is used to obtain training text description information and training avatars during the training process of the diffusion model; use the training text description information and training avatars to train the initial diffusion model, and control the timestamp, denoising results at the current timestamp, text feature words, facial feature words and attribute feature words during the training process; obtain the target noise generated by the diffusion model; generate the loss function value of the diffusion model according to the Euclidean norm between the target noise and the random noise; adjust the diffusion model according to the loss function value to obtain a trained diffusion model.
[0117] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the head portrait generation method provided in the embodiment of the present application. Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0118] An electronic device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the avatar generation method described in any of the above embodiments are implemented.
[0119] See also Figure 5 , another electronic device provided in the embodiment of the present application may also include: an input port 203 connected to the processor 202, for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, for displaying the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, for realizing communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-Definition Multimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.
[0120] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the avatar generation method described in any of the above embodiments are implemented.
[0121] The computer-readable storage medium involved in this application includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.
[0122] For the description of the relevant parts of the avatar generation system, electronic device, and computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in the avatar generation method provided in the embodiments of the present application, which will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.
[0123] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0124] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for generating an avatar, characterized in that: include: Obtain target text description information of the target avatar to be generated; Acquire a reference avatar for generating the target avatar; Translating the reference avatar to obtain reference text description information; Extracting overall facial features and facial attribute features of the reference avatar; The target avatar is generated corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features.
2. The method according to claim 1, characterized in that The extracting of the overall facial features and facial attribute features of the reference avatar includes: Performing facial cropping and alignment on the reference head portrait to obtain a reference facial image; Encoding the reference facial image to obtain facial image features; Performing feature projection on the facial image features to obtain facial feature word units; Taking the facial feature word as the overall facial feature; The facial attribute features of the reference avatar are extracted, wherein the facial attribute features include facial feature point coordinates, gender feature information, skin color feature information, body shape feature information, and age feature information.
3. The method according to claim 2, characterized in that The generating the target head portrait corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features comprises: Encoding the target text description information to obtain target text features; Encoding the reference text description information to obtain reference text features; Concatenating the target text feature and the reference text feature to obtain text feature word-units; Performing cross-attention processing on the text feature words to obtain a text processing result; Performing cross-attention processing on the facial feature word units to obtain an image processing result; Performing feature projection and concatenation on the facial image features and the facial attribute features to obtain attribute feature word-units; Get random noise; The random noise is processed according to the text processing result, the image processing result and the attribute feature word to generate the target avatar.
4. The method according to claim 3, characterized in that: The translating of the reference avatar to obtain reference text description information includes: The reference avatar is translated by a pre-trained diffusion model image interpreter to obtain reference text description information; The step of encoding the target text description information to obtain target text features, encoding the reference text description information to obtain reference text features, and concatenating the target text features and the reference text features to obtain text feature word units includes: The target text description information and the reference text description information are respectively encoded by the text compiler of the diffusion model to obtain corresponding target text features and reference text features; The target text feature and the reference text feature are concatenated through the text feature concatenation layer of the diffusion model to obtain text feature word-units; Wherein, the text compiler of the diffusion model is connected to the image interpreter and the text feature splicing layer respectively.
5. The method according to claim 4, characterized in that The step of encoding the reference facial image to obtain facial image features comprises: Encoding the reference facial image by using the facial encoder of the diffusion model to obtain facial image features; The extracting facial attribute features of the reference avatar includes: Extracting facial attribute features of the reference avatar through the attribute control network of the diffusion model; The cross-attention processing is performed on the text feature word to obtain a text processing result, and the cross-attention processing is performed on the facial feature word to obtain an image processing result, including: Performing cross-attention processing on the text feature word-units through the text cross-attention module of the diffusion model to obtain a text processing result; Performing cross-attention processing on the facial feature word-units through the image cross-attention module of the diffusion model to obtain an image processing result; The step of projecting and concatenating the facial image features and the facial attribute features to obtain attribute feature word-units includes: Performing feature projection and concatenation on the facial image features and the facial attribute features through the attribute control network to obtain attribute feature word-units; Wherein, the attribute control network is connected to the facial encoder.
6. The method according to claim 5, characterized in that The step of processing the random noise according to the text processing result, the image processing result and the attribute feature word to generate the target avatar includes: Processing the attribute feature word-element through the identity-preserving network of the diffusion model to obtain an attribute feature processing result; The text processing result, the image processing result, the attribute feature processing result and the random noise are processed by the image generation network of the diffusion model to generate the target avatar; Among them, the identity preserving network and the image generation network are both U-shaped network structures; the output layer of the attribute control network is connected to the seventh layer of the identity preserving network, and the fourth to seventh layers of the identity preserving network are respectively connected to the same layers in the image generation network; in the identity preserving network, the first layer is connected to the seventh layer, the second layer is connected to the sixth layer, and the third layer is connected to the fifth layer.
7. The method according to claim 6, characterized in that The training process of the diffusion model includes: Get training text description information and training avatar; Apply the training text description information and the training avatar to train the initial diffusion model, and control the timestamp, the denoising result at the current timestamp, the text feature word unit, the facial feature word unit and the attribute feature word unit during the training process; Acquiring target noise generated by the diffusion model; generating a loss function value of the diffusion model according to a Euclidean norm between the target noise and the random noise; The diffusion model is adjusted according to the loss function value to obtain a trained diffusion model.
8. A head portrait generation system, characterized in that: include: A first acquisition module is used to acquire target text description information of a target avatar to be generated; A second acquisition module, used to acquire a reference avatar for generating the target avatar; A first translation module, used to translate the reference avatar to obtain reference text description information; A first extraction module, used to extract the overall facial features and facial attribute features of the reference avatar; The first generating module is used to generate the target avatar corresponding to the target text description information, the reference text description information, the overall facial features and the facial attribute features.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the avatar generation method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the avatar generation method according to any one of claims 1 to 7 are implemented.