Digital human generation method and device

Abstract

The invention discloses a digital human generation method and device, and belongs to the technical field of image processing. The digital human generation method comprises the following steps: processing a collected face image corresponding to a target object to obtain a face feature image corresponding to the target object and at least one mouth mask image; processing the acquired audio data corresponding to the target object to obtain mouth shape information corresponding to the target object; and processing the facial feature image, the at least one mouth mask image and the mouth shape information to obtain a digital portrait corresponding to the target object. According to the digital human generation method, the identity information difference between the generated image and the original image is reduced, the training model is prevented from being finely adjusted again on a new reference image, and the generalization ability and universality of the model are improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method and device for generating a digital human. Background Art

[0002] In scenarios such as news broadcasting, film and television dubbing, or interactive dialogue, it is necessary to synchronize the voice and lip shape of the digital human image. In the related art, there is a method of extracting voice features from audio and combining them with the original image features to synthesize lip shape. The identity information of the image generated by this method is quite different from that of the original image. Usually, fine-tuning training is required to improve the similarity between the generated image and the original image, and the generalization ability is poor. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a method and device for generating a digital human, which reduces the identity information difference between the generated image and the original image, avoids re-fine-tuning the training model on a new reference image, and improves the generalization ability and versatility of the model.

[0004] In a first aspect, the present application provides a method for generating a digital human, the method comprising:

[0005] Processing the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object; the facial feature image includes all identity features of the target object, and the mouth mask image includes part of the identity features of the target object;

[0006] Processing the collected audio data corresponding to the target object to obtain lip shape information corresponding to the target object;

[0007] The facial feature image, the at least one mouth mask image and the lip shape information are processed to obtain a digital portrait corresponding to the target object.

[0008] According to the digital human generation method provided in the embodiment of the present application, a facial image is processed to obtain a facial feature image and at least one mouth mask image, and the facial feature image, at least one mouth mask image and lip shape information are processed in a latent space, that is, feature editing is completed in the latent space, so that the synthesized digital human portrait is not affected by the distribution of training data, thereby being able to better retain the identity information of the reference image, reducing the difference in identity information between the generated image and the original image, avoiding re-fine-tuning the training model on a new reference image, and improving the generalization ability and versatility of the model.

[0009] In one embodiment of the present application, a method for generating a digital human being, wherein the facial feature image, the at least one mouth mask image, and the lip shape information are processed to obtain a digital human portrait corresponding to the target object, including:

[0010] Processing the mouth feature included in the facial feature image and the head posture feature included in the mouth mask image to obtain a first fusion feature corresponding to the target object;

[0011] Processing the first fused feature or the lip shape information based on a target linear transformation matrix in at least one linear transformation matrix to obtain a second fused feature corresponding to the target object;

[0012] The digital portrait is obtained based on the facial feature image, the at least one mouth mask image and at least one second fused feature.

[0013] In a digital human generation method according to an embodiment of the present application, the mouth feature included in the facial feature image and the head posture feature included in the mouth mask image are processed to obtain a first fusion feature corresponding to the target object, including:

[0014] Performing a linear transformation on the head posture feature to obtain a linear feature corresponding to the head posture feature;

[0015] The linear feature and the mouth feature are fused to obtain the first fused feature.

[0016] In a digital human generation method according to an embodiment of the present application, the linear transformation of the head posture feature to obtain a linear feature corresponding to the head posture feature includes:

[0017] Based on multiple orthogonal bases and the amplitudes corresponding to each of the orthogonal bases, the head posture feature is linearly transformed to obtain the linear feature; the amplitude is obtained by inputting the head posture feature into a perception network.

[0018] In a digital human generation method according to an embodiment of the present application, the first fusion feature or the lip shape information is processed based on a target linear transformation matrix in at least one linear transformation matrix to obtain a second fusion feature corresponding to the target object, including:

[0019] Based on a first linear transformation matrix in the at least one linear transformation matrix, the first fused feature is processed to obtain a first feature corresponding to the first fused feature;

[0020] Based on a second linear transformation matrix in the at least one linear transformation matrix, the first fused feature is processed to obtain a second feature corresponding to the first fused feature;

[0021] Based on a third linear transformation matrix in the at least one linear transformation matrix, the lip shape information is processed to obtain a third feature corresponding to the lip shape information;

[0022] The first feature, the second feature and the third feature are fused to obtain the second fused feature.

[0023] In one embodiment of the present application, the method for generating a digital human being, obtaining the digital human portrait based on the facial feature image, the at least one mouth mask image and the at least one second fusion feature, comprises:

[0024] Performing feature extraction on the facial feature image to obtain reference frame features corresponding to the facial feature image; performing feature extraction on each of the mouth mask images to obtain mask frame features corresponding to each of the mouth mask images;

[0025] Performing a linear transformation on each of the second fused features to obtain an affine transformation matrix corresponding to each of the second fused features;

[0026] Based on each of the affine transformation matrices, performing an affine transformation on the reference frame feature to obtain a fourth feature corresponding to the reference frame feature;

[0027] The fourth feature and each of the mask frame features are spliced ​​together to obtain the digital portrait.

[0028] A digital human generation method according to an embodiment of the present application, wherein the facial image corresponding to the acquired target object is processed to obtain a facial feature image and at least one mouth mask image corresponding to the target object, comprises:

[0029] Processing the facial image based on a key point detection algorithm to obtain at least one facial key point position of the target object;

[0030] Based on the at least one facial key point position, the facial feature image and the at least one mouth mask image are obtained.

[0031] In a second aspect, the present application provides a digital human generation device, the device comprising:

[0032] A first processing module is used to process the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object; the facial feature image includes all identity features of the target object, and the mouth mask image includes part of the identity features of the target object;

[0033] A second processing module is used to process the collected audio data corresponding to the target object to obtain the lip shape information corresponding to the target object;

[0034] The third processing module is used to process the facial feature image, the mouth mask image and the lip shape information to obtain a digital portrait corresponding to the target object.

[0035] According to the digital human generation device provided in the embodiment of the present application, by processing the facial image, a facial feature image and at least one mouth mask image are obtained, and the facial feature image, at least one mouth mask image and lip shape information are processed in the latent space, that is, feature editing is completed in the latent space, so that the synthesized digital human portrait is not affected by the distribution of training data, thereby being able to better retain the identity information of the reference image, reducing the difference in identity information between the generated image and the original image, avoiding re-fine-tuning the training model on the new reference image, and improving the generalization ability and versatility of the model.

[0036] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating a digital human as described in the first aspect above is implemented.

[0037] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a digital human as described in the first aspect above is implemented.

[0038] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the digital human generation method as described in the first aspect above.

[0039] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0040] By processing the facial image, a facial feature image and at least one mouth mask image are obtained, and the facial feature image, at least one mouth mask image and lip shape information are processed in a latent space, that is, feature editing is completed in the latent space, so that the synthesized digital portrait is not affected by the distribution of training data, thereby being able to better retain the identity information of the reference image, reducing the difference in identity information between the generated image and the original image, avoiding re-fine-tuning the training model on a new reference image, and improving the generalization ability and versatility of the model.

[0041] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 This is one of the flow charts of the method for generating a digital human provided in the embodiment of the present application;

[0044] Figure 2 This is one of the principle schematic diagrams of the digital human generation method provided in the embodiment of the present application;

[0045] Figure 3 This is the second principle schematic diagram of the digital human generation method provided in the embodiment of the present application;

[0046] Figure 4 This is the third principle schematic diagram of the digital human generation method provided in the embodiment of the present application;

[0047] Figure 5 This is the second flow chart of the method for generating a digital human provided in the embodiment of the present application;

[0048] Figure 6 is a structural schematic diagram of a digital human generation device provided in an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0051] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0052] The digital human generation method, digital human generation device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0053] The digital human generation method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0054] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0055] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0056] The digital human generation method provided in the embodiment of the present application can be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the digital human generation method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras and wearable devices. The digital human generation method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0057] like Figure 5 As shown, the digital human generation method includes: step 110, step 120 and step 130.

[0058] Step 110: Process the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object.

[0059] In this step, the target object may be an object or a person.

[0060] The facial image may be an image containing a human face area, and the facial image may be a static picture, or may be one or more frames in a video.

[0061] Facial images include biometric and identity information.

[0062] Facial images may include eyes, nose, mouth, chin, cheeks, and facial texture, etc.

[0063] Processing facial images may include image processing and feature extraction, etc. Processing facial images may realize face recognition, expression analysis, gender recognition, etc.

[0064] The facial feature image is a reference frame, and the facial feature image includes the complete identity features of the target object. For example, the facial feature image includes the mouth features and facial muscle texture of the target object, wherein the mouth features may include the beard, lip shape and teeth of the target object.

[0065] The facial image can be processed based on the trained model or algorithm to obtain a facial feature image.

[0066] By processing the facial image, the specific position of the mouth can be obtained, for example, the mouth coordinates can be obtained, and then a mouth mask image is generated according to the position of the mouth.

[0067] The number of at least one mouth mask image can be multiple, for example, 5 or 6, and can be user-defined, which is not limited in this application.

[0068] The mouth mask image includes some identity features of the target object, for example, the mouth mask image includes the head posture, eye movements, eye angles and face proportions of the target object, while the mouth mask image does not include the mouth features of the target object.

[0069] In the actual implementation process, a mask (which can be a rectangle or other polygon) covering the mouth area can be drawn on the original image, and the mouth area can be segmented using image segmentation technology.

[0070] In some embodiments, step 110 may further include:

[0071] Processing the facial image based on a key point detection algorithm to obtain at least one facial key point position of the target object;

[0072] Based on the position of at least one facial key point, a facial feature image and at least one mouth mask image are obtained.

[0073] In this embodiment, the key point detection algorithm can be a face key point detection algorithm, wherein the face key point detection algorithm includes traditional methods, such as ASM (Active Shape Model) and AAM (Active Appearnce Model), and deep learning-based methods, for example, DCNN (Deep Convolution Network) and TCDCN (Tasks-Constrained Deep Convolutional Network) and other algorithms can be used to locate the key points of the face through a convolutional neural network structure.

[0074] Facial landmarks may include eyebrows, eyes, nose, and mouth, among others.

[0075] The face region can be determined based on a key point detection algorithm, and then specific facial key point positions can be located within the region. For example, the positions of the key points can be represented in the form of coordinates to reflect the geometric features of the face.

[0076] Based on the detected positions of facial key points, facial feature information can be further extracted. For example, the shape of the eyebrows, the outline of the eyes, the height and width of the nose, and the shape of the mouth can be extracted.

[0077] These features can be enhanced through image processing techniques (such as edge detection and texture analysis) to make them more obvious and easier to identify.

[0078] Based on the extracted facial feature information, a facial feature image can be generated.

[0079] In actual implementation, the facial feature image may be generated by fusing the original facial image with the extracted feature information, or by redrawing the facial image using the feature information.

[0080] Based on the key point detection algorithm, the key point positions of the mouth of the target object can be obtained, such as the corners of the mouth and the center of the lip.

[0081] According to the position information of the key points of the mouth, the outline of the mouth can be drawn.

[0082] Image processing techniques (such as morphological operations or region filling, etc.) can then be used to generate a mouth mask image.

[0083] The mouth mask image is an image that does not contain the mouth area.

[0084] Step 120: Process the collected audio data corresponding to the target object to obtain lip shape information corresponding to the target object;

[0085] In this step, the parameters of the audio data may be user-defined. For example, the audio data may be PCM (Pulse Code Modulation) data with a speech length of 200 milliseconds, 16 bits, a single channel, and a sampling frequency of 16000 Hz.

[0086] Processing the audio data may include performing speech recognition or speech analysis on the audio data, and mapping the recognized or analyzed speech to a visual image.

[0087] For example, speech recognition technology can be used to convert audio into text or to identify specific speech units (such as phonemes).

[0088] The recognized speech units can be further analyzed to obtain the pronunciation characteristics and timing of the speech units.

[0089] According to the correspondence between speech units and mouth shapes, the recognized speech units are mapped to the corresponding mouth shapes.

[0090] For example, a speech-to-vision mapping table or model may be predefined, and then speech units may be mapped to mouth shapes through the mapping table or model.

[0091] Based on the mapping relationship, a lip animation corresponding to the target object audio can be generated.

[0092] For example, the lip shape information can be generated by adjusting the parameters of the facial model (such as the shape and degree of opening of the lips).

[0093] Step 130: Process the facial feature image, at least one mouth mask image and the lip shape information to obtain a digital portrait corresponding to the target object.

[0094] In this step, the digital portrait may be an image, a video, or a dynamic model.

[0095] The digital portrait may include multiple frames of images, and the digital portrait may be a sequence of frames. The number of images included in the digital portrait is consistent with the number of the at least one mouth mask image.

[0096] In actual implementation, digital portraits can be applied to a variety of scenarios, such as virtual anchors, movie special effects or game characters.

[0097] The mouth features of the image identity information in the facial feature image and the head posture information in the mouth mask image can be fused, and then the fused image and the mouth shape information are fused to obtain a digital portrait of the target object.

[0098] During the research and development process, the inventors discovered that in the related technology, when using voice-driven generation of lip-sync videos, the generated digital portrait is affected by the distribution of training data, and the identity information of the generated digital portrait is quite different from that of the reference image, for example, the beard and facial muscle texture in the synthesized lip-sync area. When facing different reference images, this method needs to re-fine-tune the training model to eliminate the identity information difference.

[0099] In the present application, the audio-driven lip-sync model can complete feature editing in the latent space, where the latent space is a high-dimensional space used to represent data in the deep learning model. In the audio-driven lip-sync model, the latent space is usually used to capture facial features, speech features and the mapping relationship between them.

[0100] Feature editing is completed in the latent space, that is, the facial features generated by the model can be fine-tuned without directly modifying the input data or model parameters, so that the model can retain or modify the identity information of the reference image according to user needs. For example, when it is necessary to generate facial images with different mouth shapes but the same identity information, only the feature dimensions related to the mouth shape can be adjusted in the latent space, while the feature dimensions related to the identity information remain unchanged.

[0101] In this application, if Figure 1 As shown, a speech and image encoding module, a latent space editing module and a feature decoding module can be set.

[0102] The speech and image encoding module may be used to extract audio features and image features, and the speech and image encoding module may include an audio encoder and an image encoder.

[0103] The latent space editing module is used to fuse and edit the feature information extracted from the encoder.

[0104] The feature decoding module can be used to decode implicit features through a decoder.

[0105] In the actual implementation process, audio data and facial images are input into the model, and a digital portrait with speech and lip synchronization output by the model can be obtained.

[0106] According to the digital human generation method provided in the embodiment of the present application, a facial image is processed to obtain a facial feature image and at least one mouth mask image, and the facial feature image, at least one mouth mask image and lip shape information are processed in a latent space, that is, feature editing is completed in the latent space, so that the synthesized digital human portrait is not affected by the distribution of training data, thereby being able to better retain the identity information of the reference image, reducing the difference in identity information between the generated image and the original image, avoiding re-fine-tuning the training model on a new reference image, and improving the generalization ability and versatility of the model.

[0107] In some embodiments, step 130 may include:

[0108] Processing the mouth features included in the facial feature image and the head posture features included in the mouth mask image to obtain a first fusion feature corresponding to the target object;

[0109] The first fusion features and the lip shape information are processed to obtain a digital portrait corresponding to the target object.

[0110] In this embodiment, the mouth mask image does not include mouth features, and the mouth mask image includes head posture features corresponding to the target object.

[0111] The facial feature image is a reference frame, and the mouth mask image is a mask frame. The head posture features in the mask frame are linearly fused with the image identity information (ie, mouth features) in the reference frame to obtain a first fused feature corresponding to the target object.

[0112] The first fused features include the head posture features and the mouth features of the target object.

[0113] For each mask frame, the head posture features included therein are linearly fused with the mouth features included in the reference frame, and at least one first fused feature can be obtained.

[0114] By fusing the first fusion features and the lip shape information, a digital portrait can be obtained.

[0115] Among them, the first fusion feature and the lip shape information can be fused based on a fusion algorithm or a fusion mechanism. For example, the first fusion feature and the lip shape information can be fused through a Cross-Attention mechanism, or can be processed based on other fusion mechanisms, and can be based on user customization, which is not limited in this application.

[0116] In the actual implementation process, Figure 2 As shown, the audio data is input to the audio encoder E a In the above example, the lip shape information Z can be extracted. a .

[0117] Input the facial feature image to the image encoder E i In the above example, the implicit feature of the reference frame can be extracted, namely the mouth feature Z r→c .

[0118] Input the mouth mask image to the image encoder E i In the above example, the implicit features of the mask frame can be extracted, namely the head posture feature Z s .

[0119] In some embodiments, processing the mouth features included in the facial feature image and the head posture features included in the mouth mask image to obtain the first fusion feature corresponding to the target object may include:

[0120] Performing linear transformation on the head posture features to obtain linear features corresponding to the head posture features;

[0121] The linear features and mouth features are fused to obtain the first fused features.

[0122] In this embodiment, a linear transformation is performed on the head posture feature. The linear feature can be obtained by multiplying the head posture feature by a transformation matrix, so that the head posture feature can be mapped to a new feature space.

[0123] For example, if you need to normalize or transform the rotation angle of the head into a coordinate system that is easier to handle, you can use a rotation matrix to do this.

[0124] The linear features and the mouth features are fused, and the linear features and the mouth features can be simply concatenated or weighted summed to obtain the first fused features.

[0125] For example, the feature vector corresponding to the linear feature and the feature vector corresponding to the mouth feature can be directly concatenated into a longer vector.

[0126] Alternatively, we can assign a weight to each eigenvector and then add them together.

[0127] Alternatively, you can use a neural network (such as a fully connected layer) to automatically learn the relationship between features and fuse them.

[0128] Based on the fused features, image generation technology, such as generative adversarial networks (GANs), conditional generation models or image deformation technology, can be used to obtain the first fused features.

[0129] like Figure 3 As shown, the reference frame and the mask frame share an image encoder, so the head posture feature Z s and mouth feature Z r→c In the same latent space, the head posture feature can be superimposed on the mouth feature to obtain the first fusion feature. After superimposing the head posture information, the implicit feature of the reference frame can be expressed as:

[0130] Z r→s =Z r→c +Z c→s

[0131] Among them, Z r→c is the mouth feature, Z c→s is a linear feature.

[0132] In some embodiments, performing a linear transformation on the head posture feature to obtain a linear feature corresponding to the head posture feature may include:

[0133] Based on multiple orthogonal bases and the amplitudes corresponding to each orthogonal base, the head posture features are linearly transformed to obtain linear features.

[0134] In this embodiment, the orthogonal basis can cover multiple aspects of the head posture, such as yaw angle, pitch angle, and roll angle.

[0135] The orthogonal basis may be a vector based on physical or geometric meaning, or may be a feature vector obtained through training.

[0136] Among them, the training process of the orthogonal basis needs to follow the following restrictions:

[0137]

[0138] The amplitude is obtained by inputting the head posture features into the perception network.

[0139] For each orthogonal basis, it is necessary to calculate the projection of the original head posture feature on the basis, that is, the amplitude.

[0140] The calculated amplitudes and the corresponding orthogonal basis are used for linear combination to generate new linear features.

[0141] like Figure 3 As shown, the linear feature can be expressed by the following formula:

[0142]

[0143] Among them, Z c→s is the linear characteristic, M is the number of orthogonal bases, a i Yes i The amplitude of .

[0144] The amplitude corresponding to the orthogonal basis is obtained by linearly transforming the head posture feature. For example, the head posture feature can be input into a multi-layer perception network to obtain the amplitude output by the multi-layer perception network.

[0145] The amplitude can be Z s After a 5-layer perception network E m get:

[0146] E m (Z s )={a1,Λ,a M}

[0147] Among them, E m It is similar to the structure of the multi-layer perception network MLP.

[0148] In some embodiments, processing each first fusion feature and the lip shape information to obtain a digital portrait corresponding to the target object may include:

[0149] Processing the first fused feature or the lip shape information based on a target linear transformation matrix in at least one linear transformation matrix to obtain a second fused feature corresponding to the target object;

[0150] A digital portrait is obtained based on the facial feature image, at least one mouth mask image and at least one second fused feature.

[0151] In this embodiment, the target linear transformation matrix is ​​any linear transformation matrix among the at least one linear transformation matrix.

[0152] The first fused feature is processed based on the target linear transformation matrix, and the first fused feature can be spatially transformed, for example, the head posture in the first fused feature can be adjusted based on the target linear transformation matrix.

[0153] By processing the lip shape information based on the target transformation matrix, lip shape features that match the target speech can be generated.

[0154] In the actual implementation process, the first fusion feature or lip shape information is processed through a linear transformation matrix to obtain multiple features, and then the multiple features are fused to obtain a second fusion feature corresponding to the target object.

[0155] The second fused feature is an implicit feature that integrates information such as the reference image identity, head posture, and lip shape.

[0156] The second fused feature, the facial feature image and the mouth mask image are combined to obtain a digital portrait.

[0157] In some embodiments, processing the first fused feature or the lip shape information based on the target linear transformation matrix in at least one linear transformation matrix to obtain the second fused feature corresponding to the target object may include:

[0158] Based on a first linear transformation matrix in at least one linear transformation matrix, the first fused feature is processed to obtain a first feature corresponding to the first fused feature;

[0159] Based on a second linear transformation matrix in at least one linear transformation matrix, the first fused feature is processed to obtain a second feature corresponding to the first fused feature;

[0160] Based on a third linear transformation matrix in the at least one linear transformation matrix, the lip shape information is processed to obtain a third feature corresponding to the lip shape information;

[0161] The first feature, the second feature and the third feature are fused to obtain a second fused feature.

[0162] In this embodiment, if Figure 3 As shown, at least one linear transformation matrix may include E q , E k and E v .

[0163] Each linear transformation matrix can process different types of vectors and has different parameters. The parameters of the linear transformation matrix can be learned through training based on training data. For example, the linear transformation matrix can be trained based on audio data and lip-sync videos corresponding to the audio data.

[0164] The linear transformation matrix can be a convolution kernel or a fully connected layer, etc., which is not limited in this application.

[0165] At least one linear transformation matrix may be used to process the query vector, the key vector, and the value vector.

[0166] For example, the first linear transformation matrix E v Can be used to process value vectors and to convert value vectors into vectors in value space.

[0167] The second linear transformation matrix E k Can be used to process key vectors and convert key vectors into vectors in the key space.

[0168] The third linear transformation matrix E q It can be used to process the query vector and convert the query vector into a vector in the query space.

[0169] The first linear transformation matrix is ​​E v , based on E v By processing the first fusion feature, we can get the first feature:

[0170] v=E v (Z r→s )

[0171] Among them, v is the first feature, E v is the first linear transformation matrix, Z r→s It is the implicit feature of the reference frame after superimposing the head posture information.

[0172] The second linear transformation matrix is ​​E k , based on E k By processing the first fusion feature, the second feature can be obtained:

[0173] k=E k (Z r→s )

[0174] Among them, k is the second feature, E k is the second linear transformation matrix, Z r→s It is the implicit feature of the reference frame after superimposing the head posture information.

[0175] The third linear transformation matrix is ​​E q , based on E q By processing the lip shape information, we can get the third feature:

[0176] q=E q (Z a )

[0177] Among them, q is the third characteristic, E q is the third linear transformation matrix, Za It is the lip shape information.

[0178] The first feature, the second feature and the third feature can be fused based on the following formula to obtain the second fused feature:

[0179]

[0180] Among them, Z m is the second fusion feature, q is the third feature, k is the second feature, and v is the first feature.

[0181] The softmax function is an activation function that can convert a real vector into a probability distribution. Given a vector containing any real value, the softmax function can calculate the exponent of each element, and then normalize the exponents of multiple elements so that they add up to 1, thus forming a valid probability distribution.

[0182] like Figure 4 As shown, in some embodiments, obtaining a digital portrait based on a facial feature image, at least one mouth mask image, and at least one second fused feature may include:

[0183] Perform feature extraction on the facial feature image to obtain reference frame features corresponding to the facial feature image; perform feature extraction on each mouth mask image to obtain mask frame features corresponding to each mouth mask image;

[0184] Performing a linear transformation on each second fused feature to obtain an affine transformation matrix corresponding to each second fused feature;

[0185] Based on each affine transformation matrix, an affine transformation is performed on the reference frame feature to obtain a fourth feature corresponding to the reference frame feature;

[0186] The fourth feature and the features of each mask frame are spliced ​​together to obtain a digital portrait.

[0187] In this embodiment, the implicit features can be decoded by a decoder through a feature decoding module. The decoder can be a StyleGAN, a convolutional decoder (Convolutional Decoder), a deconvolutional network (Deconvolutional Network) or a recurrent neural network (RNN) decoder, etc., which can be selected based on user needs and is not limited in this application.

[0188] like Figure 2 As shown, by extracting features from the facial feature image, the reference frame feature F corresponding to the facial feature image can be obtained. r; Perform feature extraction on the mouth mask image to obtain the mask frame feature F corresponding to the mouth mask image s .

[0189] The reference frame features and the mask frame features may be input to the decoder in a skip-connection manner.

[0190] By performing a linear transformation on the second fusion feature, the affine transformation matrix corresponding to the second fusion feature can be obtained, so that the reference frame feature can be transformed to obtain the fourth feature F corresponding to the reference frame feature. r ', then the fourth feature and the mask frame feature are concatenated and input into the decoder, so as to obtain the target sequence frame output by the decoder, that is, the digital portrait.

[0191] The digital human generating device provided by the present application is described below. The digital human generating device described below and the digital human generating method described above can be referenced to each other.

[0192] The digital human generation method provided in the embodiment of the present application can be executed by a digital human generation device. In the embodiment of the present application, the digital human generation device executing the digital human generation method is taken as an example to illustrate the digital human generation device provided in the embodiment of the present application.

[0193] The embodiment of the present application also provides a digital human generation device.

[0194] like Figure 6 As shown, the digital human generating device includes: a first processing module 610, a second processing module 620 and a third processing module 630.

[0195] The first processing module 610 is used to process the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object; the facial feature image includes all identity features of the target object, and the mouth mask image includes part of the identity features of the target object;

[0196] The second processing module 620 is used to process the collected audio data corresponding to the target object to obtain the lip shape information corresponding to the target object;

[0197] The third processing module 630 is used to process the facial feature image, the mouth mask image and the lip shape information to obtain a digital portrait corresponding to the target object.

[0198] According to the digital human generation device provided in the embodiment of the present application, by processing the facial image, a facial feature image and at least one mouth mask image are obtained, and the facial feature image, at least one mouth mask image and lip shape information are processed in the latent space, that is, feature editing is completed in the latent space, so that the synthesized digital human portrait is not affected by the distribution of training data, thereby being able to better retain the identity information of the reference image, reducing the difference in identity information between the generated image and the original image, avoiding re-fine-tuning the training model on the new reference image, and improving the generalization ability and versatility of the model.

[0199] In some embodiments, the third processing module 630 may also be used to:

[0200] Processing the mouth features included in the facial feature image and the head posture features included in the mouth mask image to obtain a first fusion feature corresponding to the target object;

[0201] Processing the first fused feature or the lip shape information based on a target linear transformation matrix in at least one linear transformation matrix to obtain a second fused feature corresponding to the target object;

[0202] A digital portrait is obtained based on the facial feature image, at least one mouth mask image and at least one second fused feature.

[0203] In some embodiments, the third processing module 630 may also be used to:

[0204] Performing linear transformation on the head posture features to obtain linear features corresponding to the head posture features;

[0205] The linear features and mouth features are fused to obtain the first fused features.

[0206] In some embodiments, the third processing module 630 may also be used to:

[0207] Based on multiple orthogonal bases and the amplitudes corresponding to each orthogonal base, the head posture features are linearly transformed to obtain linear features; the amplitudes are obtained by inputting the head posture features into the perception network.

[0208] In some embodiments, the third processing module 630 may also be used to:

[0209] Based on a first linear transformation matrix in at least one linear transformation matrix, the first fused feature is processed to obtain a first feature corresponding to the first fused feature;

[0210] Based on a second linear transformation matrix in at least one linear transformation matrix, the first fused feature is processed to obtain a second feature corresponding to the first fused feature;

[0211] Based on a third linear transformation matrix in the at least one linear transformation matrix, the lip shape information is processed to obtain a third feature corresponding to the lip shape information;

[0212] The first feature, the second feature and the third feature are fused to obtain a second fused feature.

[0213] In some embodiments, the third processing module 630 may also be used to:

[0214] Perform feature extraction on the facial feature image to obtain reference frame features corresponding to the facial feature image; perform feature extraction on each mouth mask image to obtain mask frame features corresponding to each mouth mask image;

[0215] Performing a linear transformation on each second fused feature to obtain an affine transformation matrix corresponding to each second fused feature;

[0216] Based on each affine transformation matrix, an affine transformation is performed on the reference frame feature to obtain a fourth feature corresponding to the reference frame feature;

[0217] The fourth feature and the features of each mask frame are spliced ​​together to obtain a digital portrait.

[0218] In some embodiments, the first processing module 610 may also be used to:

[0219] Processing the facial image based on a key point detection algorithm to obtain at least one facial key point position of the target object;

[0220] Based on the position of at least one facial key point, a facial feature image and at least one mouth mask image are obtained.

[0221] The digital human generation device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0222] The digital human generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0223] The digital human generation device provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0224] In some embodiments, Figure 7 As shown, the embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned digital human generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0225] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0226] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the various processes of the above-mentioned digital human generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0227] On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to perform the various processes of the above-mentioned digital human generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0228] On the other hand, an embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned digital human generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0229] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0230] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0231] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a digital human, characterized in that: include: Processing the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object; The facial feature image includes all identity features of the target object, and the mouth mask image includes part of the identity features of the target object; Processing the collected audio data corresponding to the target object to obtain lip shape information corresponding to the target object; The facial feature image, the at least one mouth mask image and the lip shape information are processed to obtain a digital portrait corresponding to the target object.

2. The method for generating a digital human according to claim 1, characterized in that: The step of processing the facial feature image, the at least one mouth mask image and the lip shape information to obtain a digital portrait corresponding to the target object includes: Processing the mouth feature included in the facial feature image and the head posture feature included in the mouth mask image to obtain a first fusion feature corresponding to the target object; Processing the first fused feature or the lip shape information based on a target linear transformation matrix in at least one linear transformation matrix to obtain a second fused feature corresponding to the target object; The digital portrait is obtained based on the facial feature image, the at least one mouth mask image and at least one second fused feature.

3. The method for generating a digital human according to claim 2, characterized in that: The processing of the mouth feature included in the facial feature image and the head posture feature included in the mouth mask image to obtain a first fusion feature corresponding to the target object includes: Performing a linear transformation on the head posture feature to obtain a linear feature corresponding to the head posture feature; The linear feature and the mouth feature are fused to obtain the first fused feature.

4. The method for generating a digital human according to claim 3, characterized in that: The linear transformation of the head posture feature to obtain a linear feature corresponding to the head posture feature includes: Based on multiple orthogonal bases and the amplitudes corresponding to each of the orthogonal bases, the head posture feature is linearly transformed to obtain the linear feature; the amplitude is obtained by inputting the head posture feature into a perception network.

5. The method for generating a digital human according to claim 2, characterized in that: The processing of the first fused feature or the lip shape information based on a target linear transformation matrix in at least one linear transformation matrix to obtain a second fused feature corresponding to the target object includes: Based on a first linear transformation matrix in the at least one linear transformation matrix, the first fused feature is processed to obtain a first feature corresponding to the first fused feature; Based on a second linear transformation matrix in the at least one linear transformation matrix, the first fused feature is processed to obtain a second feature corresponding to the first fused feature; Based on a third linear transformation matrix in the at least one linear transformation matrix, the lip shape information is processed to obtain a third feature corresponding to the lip shape information; The first feature, the second feature and the third feature are fused to obtain the second fused feature.

6. The method for generating a digital human according to claim 2, characterized in that: The step of obtaining the digital portrait based on the facial feature image, the at least one mouth mask image and the at least one second fusion feature comprises: Performing feature extraction on the facial feature image to obtain reference frame features corresponding to the facial feature image; performing feature extraction on each of the mouth mask images to obtain mask frame features corresponding to each of the mouth mask images; Performing a linear transformation on each of the second fused features to obtain an affine transformation matrix corresponding to each of the second fused features; Based on each of the affine transformation matrices, performing an affine transformation on the reference frame feature to obtain a fourth feature corresponding to the reference frame feature; The fourth feature and each of the mask frame features are spliced ​​together to obtain the digital portrait.

7. The method for generating a digital human according to any one of claims 1 to 6, characterized in that: The step of processing the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object includes: Processing the facial image based on a key point detection algorithm to obtain at least one facial key point position of the target object; Based on the at least one facial key point position, the facial feature image and the at least one mouth mask image are obtained.

8. A digital human generation device, characterized in that: include: A first processing module is used to process the acquired facial image corresponding to the target object to obtain a facial feature image and at least one mouth mask image corresponding to the target object; the facial feature image includes all identity features of the target object, and the mouth mask image includes part of the identity features of the target object; A second processing module is used to process the collected audio data corresponding to the target object to obtain the lip shape information corresponding to the target object; The third processing module is used to process the facial feature image, the mouth mask image and the lip shape information to obtain a digital portrait corresponding to the target object.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the digital human generation method as described in any one of claims 1-7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a digital human as described in any one of claims 1 to 7 is implemented.

Images