Speech-driven lip shape generation method, device and storage medium
By decoupling the motion information of the face image in the voice-driven lip shape generation method, and using the target texture features to generate the lip shape, the problem of low lip shape synchronization accuracy in the prior art is solved, and higher lip shape generation and Synchronous accuracy.
Patent Information
- Application Number
- CN202310856520.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-07-12
AI Technical Summary
In the existing speech-driven lip shape generation methods, the accuracy of lip shape synchronization is low, which is mainly due to the coupling of apparent prior information and motion information of the lip, resulting in insufficient accuracy and robustness of the generation results.
The initial texture features of the face image are extracted through a texture encoder and the motion information is decoupled to obtain the target texture features. The lip shape is generated based on the target texture features and speech input to avoid the impact of the motion information on the generated results.
提高了唇形生成及同步的准确性和鲁棒性,提升了唇形同步的准确率,确保了唇形与语音输入的高度同步性。
Smart Images

Figure CN117115315B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for voice-driven lip shape generation. Background Art
[0002] Speech-driven lip generation is an advanced AI technology used to generate high-quality lip images from speech input. Speech-driven lip generation technology can achieve highly realistic lip animation, so that the mouth shape of virtual characters or AI assistants perfectly matches the speech input. This technology has a wide range of application potentials in film, animation, virtual reality, games, speech synthesis and human-computer interaction.
[0003] The vivid and realistic speaking ability of digital humans is a key factor in demonstrating their expressiveness, and the core issue is to maintain the synchronization of lip shape and speech content. Some existing speech-driven lip shape generation methods use deep neural networks to predict lip movements corresponding to audio input, but the extracted lip appearance prior information is not accurate enough, resulting in inaccurate lip shape synchronization. Summary of the invention
[0004] The embodiments of the present application provide a voice-driven lip shape generation method, device and storage medium to solve the technical problem of low accuracy of lip shape synchronization in the prior art.
[0005] In a first aspect, an embodiment of the present application provides a method for generating lip shape driven by speech, comprising:
[0006] Input multiple frames of face images into a texture encoder to extract texture features, and obtain initial texture features of each face image;
[0007] Decoupling motion information from the initial texture features to obtain target texture features of the face image;
[0008] The face lip shape is generated based on the target texture features of the face image and the voice input.
[0009] In some embodiments, decoupling the motion information from the initial texture features to obtain the target texture features of the face image includes:
[0010] Decoupling the motion information from the initial texture features based on a contrast constraint; the contrast constraint is used to maximize the similarity of the texture features of each face image with the same identification code and the texture features of the paired images corresponding to each face image;
[0011] The average value of the decoupled initial texture features of all face images is calculated to obtain the target texture features of the face images.
[0012] In some embodiments, the method further comprises:
[0013] Input multiple frames of face images into a motion encoder to extract motion features, and obtain initial motion features of each face image;
[0014] Decoupling texture information from the initial motion features to obtain target motion features of the face image;
[0015] It is determined whether the face can be restored according to the target texture features of the face image and the target motion features of the face image, and the validity of the target texture features of the face image is verified according to the determination result.
[0016] In some embodiments, decoupling the texture information from the initial motion features to obtain the target motion features of the face image includes:
[0017] Decoupling texture information from the initial motion features based on an orthogonal constraint; the orthogonal constraint is used to orthogonalize the texture features and motion features of each face image;
[0018] The target motion feature of the face image is determined to be any decoupled initial motion feature.
[0019] In some embodiments, the generating the lip shape of a human face based on the target texture feature of the human face image and the voice input includes:
[0020] Acquiring expression features of the facial image and determining lip movement features based on voice input;
[0021] The face lip shape is generated based on the target texture feature of the face image, the expression feature of the face image and the lip shape movement feature.
[0022] In some embodiments, the generating of the face lip shape based on the target texture feature of the face image, the expression feature of the face image and the lip shape movement feature comprises:
[0023] Fusing the facial expression features and the lip motion features to obtain fused motion features;
[0024] Based on the target texture features of the facial image and the fused motion features, a generative adversarial network StyleGAN is used to generate the lip shape of the facial image.
[0025] In some embodiments, the method further comprises:
[0026] Similarity constraints are performed on the fused motion features according to the target motion features of the face image.
[0027] In a second aspect, an embodiment of the present application provides a speech-driven lip shape generation device, comprising:
[0028] The first acquisition module is used to input multiple frames of face images into a texture encoder to extract texture features and obtain initial texture features of each face image;
[0029] A first decoupling module, used for decoupling motion information from the initial texture features to obtain target texture features of the face image;
[0030] A generation module is used to generate a lip shape of a human face based on the target texture features of the human face image and the voice input.
[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating lip shapes driven by speech as described in the first aspect above is implemented.
[0032] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech-driven lip shape generation method as described in the first aspect above.
[0033] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the voice-driven lip shape generation method as described in the first aspect above.
[0034] The voice-driven lip shape generation method, device and storage medium provided in the embodiments of the present application obtain the initial texture features of each facial image through a texture encoder, and decouple the motion information from the initial texture features to obtain the target texture features of the facial image, thereby generating the facial lip shape based on the target texture features of the facial image and the voice input, avoiding the influence of motion information on the generation result and improving the accuracy of lip shape generation and synchronization. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0036] Figure 1 It is a flowchart of the voice-driven lip shape generation method provided in an embodiment of the present application;
[0037] Figure 2This is a feature decoupling framework diagram of an example scenario of the voice-driven lip shape generation method provided in an embodiment of the present application;
[0038] Figure 3 It is a lip shape generation framework diagram of an example scenario of a voice-driven lip shape generation method provided in an embodiment of the present application;
[0039] Figure 4 It is a structural schematic diagram of a speech-driven lip shape generation device provided in an embodiment of the present application;
[0040] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] Speech-driven lip shape generation technology uses deep learning algorithms to model the complex relationship between speech signals and lip shape images, and achieve a high degree of synchronization when generating lip shapes.
[0042] Some existing methods can already achieve voice-driven. For example, Wav2Lip is a method for generating talking head videos from audio input and a single still image of a face. It uses a deep neural network to predict lip movements corresponding to the audio input, but it can only synthesize the mouth and has a low resolution. For another example, some methods use a high-resolution lip motion generation model to first spatially deform the feature map of the reference image to better preserve high-frequency details. In order to control the speaking style, StyleTalk extracts motion information from the driving audio and style information from the style video to extract its speaking style encoding such as emotions. It can control the speaking style or emotion while synthesizing the speaking lip shape. StyleTalker proposed a lip shape discriminator based on contrastive learning, which improved the lip synchronization of the speaking video. In addition, the proposed autoregressive prior and conditional sequential variational autoencoder enable it to be driven by both video and audio.
[0043] However, all of the above methods have a significant problem, which is that they do not explicitly decouple motion and appearance. Since lip movement accounts for a relatively low proportion of the overall facial movement, the details are easily ignored in the optimization, and the coupling of the lip appearance and motion features in the existing algorithms can easily interfere with the final lip driving effect. That is, when using the speech lip synchronization algorithm, the input image that provides the lip appearance prior is mixed with the lip movement state information. The existing lip synchronization algorithm does not decouple the appearance information and motion information when extracting the prior information, which affects the accuracy of the final lip synchronization.
[0044] Based on the above technical problems, an embodiment of the present application proposes a voice-driven lip shape generation method, which obtains the initial texture features of each facial image through a texture encoder, and decouples the motion information from the initial texture features to obtain the target texture features of the facial image, thereby generating facial lip shape based on the target texture features of the facial image and voice input, avoiding the influence of motion information on the generation results and improving the accuracy of lip shape generation and synchronization.
[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0046] Figure 1 is a flow chart of the method for generating lip shape driven by speech provided in the embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a method for generating lip shape driven by speech, including:
[0047] Step 101: Input multiple frames of face images into a texture encoder to extract texture features, and obtain initial texture features of each face image.
[0048] Specifically, the multiple frames of face images are images corresponding to different frames of a speech video of the same person. The texture encoder is used to extract texture features in the face images. The multiple frames of face images are input into the texture encoder, and the initial texture features of each face image are output.
[0049] For example, get the images corresponding to the first frame to the tth frame in a speech video of the same person, denoted as Using texture encoder E t The texture features of these face images are extracted respectively to obtain multiple initial texture features.
[0050] Step 102: decouple the motion information from the initial texture features to obtain target texture features of the face image.
[0051] Specifically, in the embodiment of the present application, the motion information refers to information such as facial expression and posture, etc. The motion information is decoupled from the initial texture features, thereby obtaining target texture features that contain almost no motion information.
[0052] Optionally, since the motion information between frames is different, the inter-frame similarity constraint can avoid the inclusion of motion information in the texture features as much as possible, and use the similarity measurement method to constrain the consistency of multi-frame face texture encoding, decouple the motion information from the initial texture features, and obtain the target texture features of the face image.
[0053] For example, multiple frames of face images And multiple pairs of images corresponding to multiple frames of face images Input texture encoder E t In the above example, we obtain the initial texture features of the face image and the initial texture features of the paired images, where images with the same index value are in a corresponding relationship, such as and Correspondingly, a contrastive learning algorithm is used to constrain the initial texture features of each face image and the initial texture features of its corresponding paired image to be as similar as possible, and the similarity between the initial texture features of face images of different frames should be reduced as much as possible. Through this constraint, the motion information can be decoupled from the initial texture features to obtain accurate target texture features of the face image.
[0054] For another example, multiple frames of face images And multiple pairs of images corresponding to multiple frames of face images Input texture encoder E t In the method, the initial texture features of the face image and the initial texture features of the paired images are obtained, and then the similarity between the initial texture features of the face image and the initial texture features of the corresponding paired images is measured by calculating the cosine value of the angle between the two features. The larger the cosine value of the angle, the smaller the angle between the two features, and the more similar the two features are. By maximizing the similarity between the initial texture features of the face image and the initial texture features of the corresponding paired images, the motion information is decoupled from the initial texture features, and the target texture features of the face image are obtained.
[0055] For another example, multiple frames of face images And multiple pairs of images corresponding to multiple frames of face images Input texture encoder E t In the method, the initial texture features of the face image and the initial texture features of the paired images are obtained, and then the similarity between the initial texture features of the face image and the initial texture features of the corresponding paired images is measured by calculating the correlation coefficient between the two. The larger the absolute value of the correlation coefficient, the higher the similarity between the two features. By maximizing the similarity between the initial texture features of the face image and the initial texture features of the corresponding paired images, the motion information is decoupled from the initial texture features, and the target texture features of the face image are obtained.
[0056] Step 103: Generate the lip shape of the face based on the target texture features of the face image and the voice input.
[0057] Specifically, after obtaining the target texture features of the face image, the lip shape generation is driven by voice input based on the target texture features.
[0058] For example, lip movement features are extracted from speech input, and expression features are obtained based on information other than lips in the face image. Then, the target texture features, expression features, and lip movement features of the face image are used as input to the face generator to generate lip shapes.
[0059] For another example, lip movement features are extracted from speech input, and expression features are obtained based on information other than lips in the face image. Then, based on the target texture features, expression features and lip movement features of the face image, a more advanced generative adversarial network (Style Generative Adversarial Networks, StyleGAN) is used to generate lip shapes.
[0060] The speech-driven lip shape generation method provided in the embodiment of the present application decouples the motion information from the initial texture features and only uses the target texture features obtained after decoupling as the reference prior for lip shape generation, thereby avoiding the influence of motion information on the generation result, improving the accuracy and robustness of the generated lip shape, and improving the accuracy of lip synchronization.
[0061] In some embodiments, decoupling the motion information from the initial texture features to obtain the target texture features of the face image includes:
[0062] Decoupling the motion information from the initial texture features based on a contrast constraint; the contrast constraint is used to maximize the similarity of the texture features of each face image with the same identification code and the texture features of the paired images corresponding to each face image;
[0063] The average value of the decoupled initial texture features of all face images is calculated to obtain the target texture features of the face images.
[0064] Specifically, in order to avoid the inclusion of motion information in the initial texture features as much as possible, the motion information is decoupled from the initial texture features based on contrast constraints. The contrast constraints are designed based on a contrast learning algorithm. First, a corresponding paired image is determined for each face image. The paired image can be an image of the same person (same ID) with the same texture information and different motion information as the face image, or an image of another person (different ID) with the same texture information and motion information as the face image.
[0065] A plurality of initial texture feature pairs are obtained through a texture encoder, each of which includes an initial texture feature of a face image and an initial texture feature of a paired image corresponding to the face image, that is, the initial texture feature of the face image and the initial texture feature of the paired image with the same index value; and then, the motion information is decoupled from the initial texture feature by maximizing the similarity of the initial texture feature pairs with the same identification code (ID).
[0066] Then, the average value of the decoupled initial texture features of all face images is calculated, and the average value is used as the final target texture feature of the face image.
[0067] For example, Figure 2 is a feature decoupling framework diagram of an example scenario of the voice-driven lip shape generation method provided in an embodiment of the present application, such as Figure 2 As shown, input multiple frames of face images And multiple pairs of images corresponding to multiple frames of face images To Texture Encoder E t In the texture encoder E t Encode these face images as That is, the set of initial texture features of the face image; texture encoder E t The corresponding pairs of images are encoded as That is, the set of initial texture features of paired images. Then, contrast constraints are used to decouple motion information from the initial texture features to ensure that the distance between the initial texture feature pairs of the same ID of the face image and its corresponding paired image is close. In addition, the texture feature pairs of different face IDs should be as far apart as possible. Then, the average value of the decoupled initial texture features of all face images is used as the target texture feature of the face image. In the above process, the texture encoder E t The network parameters are fixed.
[0068] For another example, input multiple frames of face images And multiple pairs of images corresponding to multiple frames of face images To Texture Encoder E t Among them, and For face images and paired images with the same ID, the encoder outputs the initial texture features of the face image And the initial texture features of the paired images Based on the similarity measurement of the initial texture features of the face image and the initial texture features of the paired image, similarity constraints are performed based on the contrastive learning algorithm. The expression of the designed contrast constraint is:
[0069]
[0070] Among them, L decou represents the contrast constraint function; τ represents an adjustable scaling parameter; k represents the index of the initial texture feature pair with the same ID; Represents the initial texture features of the face image with index value k; Represents the initial texture features of the paired images with index value k; t is the number of face images; Represents the initial texture features of the i-th face image; Represents the initial texture features of the i-th paired image.
[0071] The motion information is decoupled from the initial texture feature by the contrast constraint, and then the average value of the decoupled initial texture features of all face images is used as the target texture feature of the face image.
[0072] The speech-driven lip shape generation method provided in the embodiment of the present application performs similarity constraints on the texture features of facial images based on contrastive learning and designs contrast constraints, thereby avoiding the inclusion of motion information in the features as much as possible, thereby achieving the decoupling of motion information from texture features, and obtaining more accurate target texture features as a reference prior for lip shape generation, thereby improving the accuracy of lip shape generation and synchronization.
[0073] In some embodiments, the method further comprises:
[0074] Input multiple frames of face images into a motion encoder to extract motion features, and obtain initial motion features of each face image;
[0075] Decoupling texture information from the initial motion features to obtain target motion features of the face image;
[0076] It is determined whether the face can be restored according to the target texture features of the face image and the target motion features of the face image, and the validity of the target texture features of the face image is verified according to the determination result.
[0077] Specifically, the motion encoder is used to extract motion features of facial images. Multiple frames of facial images are input into the motion encoder to extract motion features, and the initial motion features of each facial image are obtained. Then, the texture information is decoupled from the motion features, that is, the texture information is prevented from being included in the motion features. First, multiple frames of facial images are input into the motion encoder to extract motion features, and the initial motion features of each facial image are obtained. Then, the texture information is decoupled from the initial motion features to obtain the target motion features of the facial images.
[0078] Since the target motion features and target texture features of the obtained facial image are not necessarily complete, for example, when the target texture features only include partial cheek texture information, or the target motion features only include eyebrow and eye movements, the target texture features and target motion features cannot be used to successfully restore the face, and the obtained target motion features and target texture features are invalid and cannot be used.
[0079] Therefore, after obtaining the target texture features and target motion features of the face image, it is still necessary to perform face restoration based on the target texture features and target motion features of the face image. If the restoration is successful, it indicates that the target motion features and target texture features are complete and effective, and can continue to be used in subsequent steps.
[0080] For example, the target texture features and target motion features of the facial image are taken as input, and a face generator is used to determine whether the face can be successfully restored. If the face can be successfully restored, the target texture features and target motion features of the facial image are valid. If the face restoration fails, the obtained target texture features and target motion features of the facial image are discarded, and the texture features and motion features are re-extracted based on the texture encoder and the motion encoder, and the texture information and motion information are decoupled.
[0081] The voice-driven lip shape generation method provided in the embodiment of the present application determines whether the target texture features and target motion features obtained can successfully restore the face, determines whether the target texture features and target motion features are complete and effective, thereby ensuring the effectiveness and success rate of face restoration and improving the accuracy of lip shape generation.
[0082] In some embodiments, decoupling the texture information from the initial motion features to obtain the target motion features of the face image includes:
[0083] Decoupling texture information from the initial motion features based on an orthogonal constraint; the orthogonal constraint is used to orthogonalize the texture features and motion features of each face image;
[0084] The target motion feature of the face image is determined to be any decoupled initial motion feature.
[0085] Specifically, after obtaining the initial motion features of each face image output by the motion encoder, the texture information is decoupled from the motion features based on the orthogonal constraint, that is, the initial motion features of each face image and the corresponding initial texture features are made as orthogonal (independent) as possible to prevent the texture information from being contained in the motion features.
[0086] For example, Figure 2 As shown, input multiple frames of face images To motion encoder Em Extract motion features from the image to obtain the initial motion features of each face image, which can be formally expressed as And input multiple frames of face images To Texture Encoder E t In the above example, the initial texture features of each face image are obtained, which can be formally expressed as The motion features are constrained based on orthogonal constraints, that is, to ensure that f m and T x The corresponding feature points are multiplied by 0, and the loss function expression corresponding to the orthogonal constraint is:
[0087]
[0088] Among them, L org represents the loss function based on orthogonal constraints; t is the number of face images; Represents the initial motion features of the i-th face image; represents the initial texture features of the i-th face image; ‖ ‖ represents the modulus of the vector, and · represents the vector dot product.
[0089] After decoupling the texture information from the initial motion feature, any decoupled initial motion feature is selected as the target motion feature of the face image.
[0090] The speech-driven lip shape generation method provided in the embodiment of the present application decouples texture information from initial motion features by means of orthogonal constraints, obtains accurate motion features, and provides accuracy guarantee for subsequent face generation and lip shape generation.
[0091] In some embodiments, the generating the lip shape of a human face based on the target texture feature of the human face image and the voice input includes:
[0092] Acquiring expression features of the facial image and determining lip movement features based on voice input;
[0093] The face lip shape is generated based on the target texture feature of the face image, the expression feature of the face image and the lip shape movement feature.
[0094] Specifically, the target texture features of the face image, the expression features of the face image and the lip shape movement features are used as inputs of a face generator to achieve lip shape generation.
[0095] The lip motion feature is extracted from the voice input. The expression feature of the face image includes the eyebrow expression and facial posture information of the original face image. Since the lip shape generation only needs to edit the mouth and other parts remain unchanged, the information of the face image other than the lips is used as input to provide facial posture and eyebrow expression, etc., to obtain the expression feature of the face image.
[0096] For example, Figure 3 is a lip shape generation framework diagram of an example scenario of a voice-driven lip shape generation method provided in an embodiment of the present application, such as Figure 3 As shown, based on the target texture features of the face image, the expression features of the face image and the lip shape movement features, the corresponding speaking lip shape is decoded and generated.
[0097] First, a texture encoder is used to extract the face image Extracting target texture features Then, the mean texture feature of t faces is calculated as the final target texture feature. Then, the mouth of each face image is masked and other parts are kept unchanged, which is recorded as Then through the face encoding network E f Map it into expression features (expression encoding) f e =E f (x s ). And extract lip movement information from the speech input. In order to avoid the influence of timbre on lip shape generation, a deep speech recognition network can be used to extract the deep features of speech. In order to describe the lip shape more accurately, the speech features of a time window are used as input each time. The time-sensitive network (TSN) based on one-dimensional convolution is used to extract the temporal features and obtain the lip motion feature f a =E a (x a ).
[0098] The target texture features of the face image, the expression features of the face image and the lip shape movement features are used as inputs of a face generator to achieve lip shape generation.
[0099] The voice-driven lip shape generation method provided in the embodiment of the present application generates facial lip shape based on multiple features including the target texture features of the facial image, the expression features of the facial image and the lip shape movement features, thereby improving the accuracy of generated lip shape.
[0100] In some embodiments, the generating of the face lip shape based on the target texture feature of the face image, the expression feature of the face image and the lip shape movement feature comprises:
[0101] Fusing the facial expression features and the lip motion features to obtain fused motion features;
[0102] Based on the target texture features of the facial image and the fused motion features, a generative adversarial network StyleGAN is used to generate the lip shape of the facial image.
[0103] Specifically, the expression features of the facial image and the lip shape motion features are first fused to obtain fused motion features, and then the target texture features of the facial image and the fused motion features are used as input, and the advanced StyleGAN is used as a model for face / lip shape generation to generate lip shape.
[0104] For example, the facial expression feature f e and lip movement characteristics f a Fusion is performed to obtain the fused motion feature f′ m , and then the facial expression feature f e and fused motion feature f′ m After splicing, it passes through a conversion network, which maps the input information to the w space of StyleGAN, and then StyleGAN takes w as input to generate the final lip shape.
[0105] The voice-driven lip shape generation method provided in the embodiment of the present application adopts a more advanced StyleGAN generation model as the final lip shape generator, which can obtain high-quality and high-resolution facial images and further improve the accuracy of lip shape generation and synchronization.
[0106] In some embodiments, the method further comprises:
[0107] Similarity constraints are performed on the fused motion features according to the target motion features of the face image.
[0108] Specifically, in order to reduce the difficulty of training, similarity constraints are also performed on the fused motion features according to the target motion features of the face image, so as to improve the convergence speed of training.
[0109] For example, firstly transform the facial expression feature f e and lip movement characteristics f a Fusion is performed to obtain the fused motion feature f′ m , and then fuse the motion features f′ m and the target motion feature f of the face image m =E m (x s ) to perform similarity constraints.
[0110] The voice-driven lip shape generation method provided in the embodiment of the present application performs similarity constraints on the fused motion features according to the target motion features of the facial image, thereby improving the convergence speed of training and reducing the difficulty of training.
[0111] Figure 4 is a structural diagram of a speech-driven lip shape generation device provided in an embodiment of the present application, such as Figure 4 As shown, an embodiment of the present application provides a speech-driven lip shape generation device, including a first acquisition module 401, a first decoupling module 402 and a generation module 403.
[0112] The first acquisition module 401 is used to input multiple frames of face images into a texture encoder to extract texture features, so as to obtain initial texture features of each face image.
[0113] The first decoupling module 402 is used to decouple the motion information from the initial texture features to obtain the target texture features of the face image.
[0114] The generating module 403 is used to generate the lip shape of the face based on the target texture features of the face image and the voice input.
[0115] In some embodiments, the first decoupling module comprises:
[0116] A first decoupling unit, configured to decouple the motion information from the initial texture features based on a contrast constraint, wherein the contrast constraint is configured to maximize the similarity between the texture features of each face image having the same identification code and the texture features of a paired image corresponding to each face image;
[0117] The calculation unit is used to calculate the average value of the decoupled initial texture features of all face images to obtain the target texture features of the face images.
[0118] In some embodiments, it also includes:
[0119] The second acquisition module inputs multiple frames of face images into a motion encoder to extract motion features, and obtains initial motion features of each face image;
[0120] The second decoupling module decouples the texture information from the initial motion feature to obtain the target motion feature of the face image;
[0121] The judgment module is used to judge whether the face can be restored according to the target texture features of the face image and the target motion features of the face image, and to verify the validity of the target texture features of the face image according to the judgment result.
[0122] In some embodiments, the second decoupling module comprises:
[0123] A second decoupling unit, used for decoupling texture information from the initial motion features based on an orthogonal constraint; the orthogonal constraint is used for orthogonalizing the texture features and motion features of each face image;
[0124] The determination unit is used to determine that the target motion feature of the face image is any decoupled initial motion feature.
[0125] In some embodiments, the generating module comprises:
[0126] An acquisition unit, used to acquire the expression features of the face image and determine the lip movement features based on the voice input;
[0127] A generating unit is used to generate a lip shape of a face based on a target texture feature of the face image, an expression feature of the face image and a lip shape movement feature.
[0128] In some embodiments, the generating unit comprises:
[0129] A feature fusion subunit, used for fusing the expression feature of the face image and the lip motion feature to obtain a fused motion feature;
[0130] The generating subunit is used to generate the lip shape of the face by using the generative adversarial network StyleGAN based on the target texture features of the face image and the fused motion features.
[0131] In some embodiments, it also includes:
[0132] The constraint subunit is used to perform similarity constraints on the fused motion features according to the target motion features of the face image.
[0133] Specifically, the above-mentioned voice-driven lip shape generation device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned voice-driven lip shape generation method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.
[0134] It should be noted that the division of units / modules in the above embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated units may be implemented in the form of hardware or in the form of software functional units.
[0135] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5As shown, the electronic device may include: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 communicate with each other via the communication bus 504. The processor 501 may call the logic instructions in the memory 503 to execute the voice-driven lip shape generation method, which includes:
[0136] Input multiple frames of face images into a texture encoder to extract texture features, and obtain initial texture features of each face image;
[0137] Decoupling motion information from the initial texture features to obtain target texture features of the face image;
[0138] The face lip shape is generated based on the target texture features of the face image and the voice input.
[0139] Specifically, the processor 501 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD), and the processor can also adopt a multi-core architecture.
[0140] The logic instructions in the memory 503 can be implemented in the form of software functional units and can be stored in a processor-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0141] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the voice-driven lip shape generation method provided by the above-mentioned method embodiments, the method comprising:
[0142] Input multiple frames of face images into a texture encoder to extract texture features, and obtain initial texture features of each face image;
[0143] Decoupling motion information from the initial texture features to obtain target texture features of the face image;
[0144] The face lip shape is generated based on the target texture features of the face image and the voice input.
[0145] Specifically, the above-mentioned computer program product provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0146] In some embodiments, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is used to enable a computer to execute the voice-driven lip shape generation method provided by the above-mentioned method embodiments, the method comprising:
[0147] Input multiple frames of face images into a texture encoder to extract texture features, and obtain initial texture features of each face image;
[0148] Decoupling motion information from the initial texture features to obtain target texture features of the face image;
[0149] The face lip shape is generated based on the target texture features of the face image and the voice input.
[0150] Specifically, the above-mentioned computer-readable storage medium provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0151] It should be noted that the computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.
[0152] It should also be noted that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.
[0153] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0154] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.
[0155] In this application, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for example, "when A meets the second condition, determine B", etc.; for example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.
[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0158] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0159] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0160] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A speech-driven lip shape generation method, characterized in that: include: Input multiple frames of face images into a texture encoder to extract texture features, and obtain initial texture features of each face image; Decoupling motion information from the initial texture features to obtain target texture features of the face image; The face lip shape is generated based on the target texture features of the face image and the voice input.
2. The method for generating lip shape driven by speech according to claim 1, characterized in that: The step of decoupling the motion information from the initial texture features to obtain target texture features of the face image includes: Decoupling the motion information from the initial texture features based on a contrast constraint; the contrast constraint is used to maximize the similarity of the texture features of each face image with the same identification code and the texture features of the paired images corresponding to each face image; The average value of the decoupled initial texture features of all face images is calculated to obtain the target texture features of the face images.
3. The method for generating lip shape driven by speech according to claim 1, characterized in that: The method further comprises: Input multiple frames of face images into a motion encoder to extract motion features, and obtain initial motion features of each face image; Decoupling texture information from the initial motion features to obtain target motion features of the face image; It is determined whether the face can be restored according to the target texture features of the face image and the target motion features of the face image, and the validity of the target texture features of the face image is verified according to the determination result.
4. The method for generating lip shape driven by speech according to claim 3, characterized in that: The decoupling of texture information from the initial motion features to obtain target motion features of the face image includes: Decoupling texture information from the initial motion features based on an orthogonal constraint; the orthogonal constraint is used to orthogonalize the texture features and motion features of each face image; The target motion feature of the face image is determined to be any decoupled initial motion feature.
5. The method for generating lip shape driven by speech according to claim 1, characterized in that: The generating of the face lip shape based on the target texture feature of the face image and the voice input comprises: Acquiring expression features of the facial image and determining lip movement features based on voice input; The face lip shape is generated based on the target texture feature of the face image, the expression feature of the face image and the lip shape movement feature.
6. The method for generating lip shape driven by speech according to claim 5, characterized in that: The generating of the face lip shape based on the target texture feature of the face image, the expression feature of the face image and the lip shape movement feature comprises: Fusing the facial expression features and the lip motion features to obtain fused motion features; Based on the target texture features of the facial image and the fused motion features, a generative adversarial network StyleGAN is used to generate the lip shape of the facial image.
7. The method for generating lip shape driven by speech according to claim 6, characterized in that: The method further comprises: The fused motion features are similarly constrained according to target motion features of the face image; the target motion features of the face image are obtained by decoupling texture information from initial motion features, and the initial motion features are obtained by inputting multiple frames of face images into a motion encoder for motion feature extraction.
8. A speech-driven lip shape generation device, characterized in that: include: The first acquisition module is used to input multiple frames of face images into a texture encoder to extract texture features and obtain initial texture features of each face image; A first decoupling module, used for decoupling motion information from the initial texture features to obtain target texture features of the face image; A generation module is used to generate a lip shape of a human face based on the target texture features of the human face image and the voice input.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the speech-driven lip shape generation method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating lip shapes driven by speech according to any one of claims 1 to 7 is implemented.
