Generation method and device for driving expression coefficient of digital human, electronic equipment and storage medium

By extracting the characteristics of audio and control information, and combining the cross attention unit to generate expression coefficients, the problem that the digital human expression driving results do not meet expectations are solved, and high expressive digital human expression and multimodal control are achieved.

CN119007755APending Publication Date: 2024-11-22BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410931290.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When driving digital human expressions, it is difficult to obtain expected and highly expressive results, mainly because the voice input driving results are too dependent on audio signals and lack effective support for digital human emotions and multimodal control.

Method used

By obtaining audio data, speaking style information and multimodal control information, extracting the characteristics of each dimension and inputting the expression coefficient generation module to generate expression coefficients for driving digital people, combining the voice driver module, the first control module and the second control module The feature extraction of the feature, using the cross attention unit to generate expression coefficients, realize the modeling of digital human emotions and multimodal control.

Benefits of technology

The high expressiveness and emotional characteristics of digital human expressions are realized, and the driving methods for digital humans are enriched through multimodal control, and the driving results that meet expectations are simply obtained.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007755A_ABST
    Figure CN119007755A_ABST
Patent Text Reader

Abstract

The invention relates to an expression coefficient generation method and device for driving a digital person, electronic equipment and a storage medium. The method comprises the steps that audio data for driving the digital person, first control information including speaking style information and speaking emotion information and second control information including control information of multiple modes are acquired; performing feature extraction processing on the audio data through a voice driving module to obtain audio features; performing feature extraction processing on the first control information through the first control module to obtain a speaking style feature and a speaking emotion feature; performing feature extraction processing on the second control information through a second control module to obtain time sequence text features; and inputting the audio features, the speaking style features, the speaking emotion features and the time sequence text features into an expression coefficient generation module for processing to obtain an expression coefficient for driving the digital human. According to the method, the control modes of the digital human can be enriched, and a driving result which accords with expectation and is high in expressive force is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, storage medium and program product for driving a digital human's expression coefficient generation method. Background Art

[0002] It is an important task in the field of digital humans to drive the head of 3D digital humans based on speech, such as lip-shaping and expression generation, to achieve the ability of 3D digital humans to express emotions with high expressiveness. This technology is often used to drive 3D characters in the film and game industries, or to drive the head of digital human agents in AR / VR scenes.

[0003] In the related art, when performing expression driving, the audio is generally encoded using an existing audio encoder, and the encoded audio features are injected into the expression coefficient generation module. After receiving the audio features, the expression coefficient generation module performs model training and reasoning to generate expression coefficients, thereby driving the digital human to perform expression movements.

[0004] However, the driving results of existing solutions are completely dependent on voice input, which makes it difficult to obtain expected and highly expressive results. Summary of the invention

[0005] The present disclosure provides a method, device, electronic device, storage medium and program product for driving the expression coefficient of a digital human, so as to at least solve the problem in the related art that it is difficult to obtain expected and highly expressive results. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a method for generating expression coefficients for driving a digital human is provided, comprising:

[0007] Acquire audio data, first control information, and second control information for driving the digital human; the first control information includes speech style information and speech emotion information, and the second control information includes control information of multiple modes, and the control information of each mode corresponds to controlling the facial movements of the digital human within a time period;

[0008] The audio data is subjected to feature extraction processing by the voice driving module to obtain audio features; the first control information is subjected to feature extraction processing by the first control module to obtain speech style features and speech emotion features; the second control information is subjected to feature extraction processing by the second control module to obtain time series text features;

[0009] The audio features, the speaking style features, the speaking emotion features, and the time-series text features are input into an expression coefficient generation module for processing to obtain an expression coefficient for driving the digital human.

[0010] In an exemplary embodiment, the performing feature extraction processing on the audio data by the voice driving module to obtain the audio features includes:

[0011] In the voice driving module, the audio data is encoded to obtain audio encoding information;

[0012] Perform lip shape coefficient prediction based on the audio coding information to obtain a lip shape regression coefficient;

[0013] The audio feature is generated according to the audio encoding information and the lip shape regression coefficient.

[0014] In an exemplary embodiment, the speech emotion information includes emotion category and emotion intensity; the first control module performs feature extraction processing on the first control information to obtain speech style features and speech emotion features, including:

[0015] In the first control module, feature extraction processing is performed on the speaking style information in the first control information to obtain the speaking style feature;

[0016] Furthermore, the emotion intensity of the emotion category is adjusted by using the trained adjustment parameters corresponding to the emotion category and the emotion intensity to obtain the speech emotion feature.

[0017] In an exemplary embodiment, the step of performing feature extraction processing on the second control information by the second control module to obtain the time series text feature includes:

[0018] In the second control module, action units corresponding to the control information of each mode in the second control information are respectively determined;

[0019] Combine the action units corresponding to the control information of each mode in time sequence to obtain time sequence action combination control information;

[0020] According to a preset sentence template, assembling the time sequence action combination control information into time sequence text control information;

[0021] The assembled time-series text control information is subjected to feature extraction processing to obtain the time-series text feature.

[0022] In an exemplary embodiment, the expression coefficient generation module includes a first cross-attention unit, a second cross-attention unit, and a third cross-attention unit;

[0023] The step of inputting the audio features, the speaking style features, the speaking emotion features, and the time-series text features into an expression coefficient generation module for processing to obtain an expression coefficient for driving the digital human comprises:

[0024] Inputting the audio feature into the first cross-attention unit to obtain a first attention feature; inputting the speaking style feature and the speaking emotion feature into the second cross-attention unit to obtain a second attention feature; and inputting the temporal text feature into the third cross-attention unit to obtain a third attention feature;

[0025] Based on the first attention feature, the second attention feature and the third attention feature, an expression coefficient for driving the digital human is generated.

[0026] In an exemplary embodiment, the speech emotion information includes an emotion category and an emotion intensity, and the emotion intensity is controlled by an adjustment parameter corresponding to the emotion category; the method further includes:

[0027] Obtaining sample audio data, sample first control information, sample second control information and true expression coefficient;

[0028] The sample audio data is subjected to feature extraction processing by the speech driving module to obtain sample audio features; the speaking style information in the sample first control information is subjected to feature extraction processing by the first control module to obtain sample speaking style features, and the speaking emotion information in the sample first control information is converted into sample speaking emotion features by initializing the adjustment parameters; the sample second control information is subjected to feature extraction processing by the second control module to obtain sample time series text features;

[0029] Taking the sample audio features, the sample speech style features, the sample speech emotion features, and the sample time-series text features as input variables and the real expression coefficient as supervision information, the expression coefficient generation module to be trained and the initialized adjustment parameters are trained to obtain the expression coefficient generation module and the adjustment parameters.

[0030] In an exemplary embodiment, the performing feature extraction processing on the sample second control information by the second control module to obtain the sample time-series text feature includes:

[0031] In the second control module, the action units corresponding to the control information of each mode in the sample second control information are combined in time sequence to obtain time sequence action combination control information;

[0032] Performing mask processing on the timing action combination control information to obtain mask control information;

[0033] According to the sample first control information, a preset sentence template is adjusted to obtain an adjusted sentence template; according to the adjusted sentence template, the action units in the mask control information are randomly sorted and assembled into time-sequence text control information;

[0034] The assembled time-series text control information is subjected to feature extraction processing to obtain the sample time-series text features.

[0035] In an exemplary embodiment, performing mask processing on the timing action combination control information to obtain mask control information includes:

[0036] Masking is performed on a single action unit and / or a combination of action units in the sequential action combination control information to obtain the mask control information.

[0037] According to a second aspect of an embodiment of the present disclosure, there is provided a device for generating expression coefficients for driving a digital human, comprising:

[0038] A data acquisition unit is configured to acquire audio data, first control information, and second control information for driving a digital human; the first control information includes speech style information and speech emotion information, and the second control information includes control information of multiple modes, and each mode of control information corresponds to controlling the facial movements of the digital human within a time period;

[0039] The feature extraction unit is configured to perform feature extraction processing on the audio data through the voice driving module to obtain audio features; perform feature extraction processing on the first control information through the first control module to obtain speaking style features and speaking emotion features; perform feature extraction processing on the second control information through the second control module to obtain time series text features;

[0040] The coefficient generating unit is configured to input the audio feature, the speaking style feature, the speaking emotion feature, and the temporal text feature into the expression coefficient generating module for processing to obtain the expression coefficient for driving the digital human.

[0041] In an exemplary embodiment, the feature extraction unit is further configured to execute in the voice driving module, encode the audio data to obtain audio encoding information; predict the lip shape coefficient based on the audio encoding information to obtain the lip shape regression coefficient; and generate the audio feature according to the audio encoding information and the lip shape regression coefficient.

[0042] In an exemplary embodiment, the speaking emotion information includes emotion category and emotion intensity; the feature extraction unit is further configured to execute in the first control module, perform feature extraction processing on the speaking style information in the first control information to obtain the speaking style feature; and adjust the emotion intensity of the emotion category through trained adjustment parameters corresponding to the emotion category and the emotion intensity to obtain the speaking emotion feature.

[0043] In an exemplary embodiment, the feature extraction unit is also configured to execute in the second control module, respectively determine the action units corresponding to the control information of each mode in the second control information; combine the action units corresponding to the control information of each mode in chronological order to obtain time-series action combination control information; assemble the time-series action combination control information into time-series text control information according to a preset sentence template; perform feature extraction processing on the assembled time-series text control information to obtain the time-series text feature.

[0044] In an exemplary embodiment, the expression coefficient generation module includes a first cross-attention unit, a second cross-attention unit and a third cross-attention unit; the coefficient generation unit is also configured to execute inputting the audio feature into the first cross-attention unit to obtain a first attention feature; inputting the speaking style feature and the speaking emotion feature into the second cross-attention unit to obtain a second attention feature; and inputting the temporal text feature into the third cross-attention unit to obtain a third attention feature; based on the first attention feature, the second attention feature and the third attention feature, generate an expression coefficient for driving the digital human.

[0045] In an exemplary embodiment, the speech emotion information includes emotion category and emotion intensity, and the emotion intensity is controlled by the adjustment parameter corresponding to the emotion category; the device also includes a model training unit, which is configured to execute the acquisition of sample audio data, sample first control information, sample second control information and real expression coefficient; the sample audio data is subjected to feature extraction processing by the voice driving module to obtain sample audio features; the speech style information in the sample first control information is subjected to feature extraction processing by the first control module to obtain sample speaking style features, and the speech emotion information in the sample first control information is converted into sample speaking emotion features by the initialized adjustment parameters; the sample second control information is subjected to feature extraction processing by the second control module to obtain sample time-series text features; the expression coefficient generation module to be trained and the initialized adjustment parameters are trained with the sample audio features, the sample speaking style features, the sample speech emotion features and the sample time-series text features as input variables and the real expression coefficient as supervision information to obtain the expression coefficient generation module and the adjustment parameters.

[0046] In an exemplary embodiment, the model training unit is also configured to execute in the second control module, combine the action units corresponding to the control information of each mode in the sample second control information in chronological order to obtain time-series action combination control information; perform mask processing on the time-series action combination control information to obtain masked control information; adjust the preset sentence template according to the sample first control information to obtain an adjusted sentence template; according to the adjusted sentence template, randomly sort the action units in the masked control information and assemble them into time-series text control information; perform feature extraction processing on the assembled time-series text control information to obtain the sample time-series text feature.

[0047] In an exemplary embodiment, the model training unit is further configured to perform mask processing on a single action unit and / or a combination of action units in the timing action combination control information to obtain the mask control information.

[0048] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0049] processor;

[0050] a memory for storing instructions executable by the processor;

[0051] The processor is configured to execute the instructions to implement the method as described in any one of the above items.

[0052] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods described above.

[0053] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods described above.

[0054] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0055] The audio data, the first control information including the speaking style information and the speaking emotion information, and the multi-modal second control information are used as the driving data, and the features of the data in each dimension are extracted, and then input into the expression coefficient generation module for processing to obtain the expression coefficient for driving the digital human. When generating the expression coefficient, in addition to the audio signal, the method also introduces the first control information and the second control information. Through the speaking style and speaking emotion in the first control information, the emotional modeling of the digital human can be realized, so that the action expression made by the digital human has emotional characteristics. Through the multi-modal timing control information in the second control information, the control of the digital human at different granularities such as the whole, part, and timing can be realized, and the control method of the digital human can be enriched, so that the driving result that meets the expectations and is highly expressive can be simply obtained.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0058] Figure 1 The figure is a flow chart of a method for generating expression coefficients for driving a digital human according to an exemplary embodiment.

[0059] Figure 2 The figure is a schematic diagram showing a method of controlling a digital human through multi-dimensional control information according to an exemplary embodiment.

[0060] Figure 3 is a schematic diagram showing a process of generating an expression coefficient according to an exemplary embodiment.

[0061] Figure 4 It is a structural diagram of a voice drive module according to an exemplary embodiment.

[0062] Figure 5 The figure is a schematic diagram showing a timing action combination control information according to an exemplary embodiment.

[0063] Figure 6 The diagram is a schematic diagram showing how to assemble sequential action combination control information into text according to an exemplary embodiment.

[0064] Figure 7 It is a structural diagram of an expression coefficient generation module according to an exemplary embodiment.

[0065] Figure 8 The figure is a schematic diagram showing an expression coefficient generation module and a training step of adjusting parameters according to an exemplary embodiment.

[0066] Fig. 9 The diagram is a schematic diagram showing a decoupling and enhancement process of sample second control information according to an exemplary embodiment.

[0067] Fig.10 It is a schematic diagram of the overall framework of a generation model for driving the expression coefficient of a digital human according to an exemplary embodiment.

[0068] Fig.11 The present invention is a structural block diagram of a device for driving expression coefficients of a digital human according to an exemplary embodiment.

[0069] Fig.12 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0070] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0072] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties.

[0073] In an exemplary embodiment, if Figure 1 As shown, a method for generating expression coefficients for driving a digital human is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0074] In step S110, audio data, first control information and second control information for driving the digital human are obtained; the first control information includes speaking style information and speaking emotion information, and the second control information includes control information of multiple modes, and the control information of each mode corresponds to controlling the facial movements of the digital human within a time period.

[0075] The second control information and the audio data are consistent in time length, that is, the playback duration of the audio data is the same as the control duration of the second control information. The first control information represents a single control signal that maintains time consistency in the generated expression coefficient sequence, that is, the first control information remains unchanged within the playback duration corresponding to the audio data.

[0076] The speaking style information is information used to control the speaking characteristics of the digital person, and can be characterized by age, gender, identity, region, etc. The speaking style information can be represented by a picture or text. For example, the speaking style information can be a picture of a young man, or a text constructed based on age, identity, gender, etc., such as a 30-year-old female lawyer.

[0077] The speech emotion information includes emotion category and emotion intensity. The emotion category may include anger, contempt, disgust, fear, happiness, sadness, surprise, neutrality, etc. The emotion intensity indicates the degree or intensity of the emotion. For example, the emotion intensity can be divided into strong, medium, weak, etc.

[0078] The control information of multiple modes included in the second control information may be action unit (AU) information, reference images, text descriptions based on natural language, etc. The control information of each mode corresponds to controlling the facial movements of the digital human within a time period, which can be understood as fine-grained and multi-modal control of the digital human in terms of time sequence.

[0079] refer to Figure 2 , is a schematic diagram of controlling a digital human through multi-dimensional control information shown in an embodiment. The driving of a digital human can be realized by combining voice driving, first control information and second control information. Among them, the first control information includes speaking style information and speaking emotion information. The speaking style information can be a picture or text, and the speaking emotion information includes emotion category (emotion label) and emotion intensity. The second control information includes information of multiple time periods, each time period corresponds to controlling different expression action frames, and inputs control information of different modes. For example, 10-15 frames are controlled by given AU information, 30-45 frames are controlled by given reference pictures, and 60-80 frames are controlled by given natural language (text description). The figure also shows a schematic diagram of expression sequences generated by controlling only with the second control information, controlling only with the first control information, and controlling with the first control information and the second control information at the same time.

[0080] In step S120, the audio data is subjected to feature extraction processing by the voice driving module to obtain audio features; the first control information is subjected to feature extraction processing by the first control module to obtain speaking style features and speaking emotion features; the second control information is subjected to feature extraction processing by the second control module to obtain time series text features.

[0081] Specifically, the voice drive module, the first control module and the second control module can be understood as three feature extraction modules, which are used to extract the features of audio data, the first control information and the second control information respectively. The three modules can be set in parallel, that is, the feature extraction process of the three modules can be carried out simultaneously to ensure the efficiency of feature extraction.

[0082] For example, refer to Figure 3The schematic diagram of the expression coefficient generation process shown in FIG. 1 shows that after obtaining audio data, first control information and second control information, the audio data is input into the voice driving module for processing, the first control information is input into the first control module for processing, and the second control information is input into the second control module for processing. Among them, the second control information needs to be converted into an AU sequence before being input into the second control module, and then the AU sequence is converted into a text sequence, and the text sequence is input into the second control module for processing.

[0083] In step S130, the audio features, speaking style features, speaking emotion features, and temporal text features are input into an expression coefficient generation module for processing to obtain expression coefficients for driving the digital human.

[0084] Among them, the expression coefficients include multiple groups of expression coefficients, each group of expression coefficients controls an expression movement of the digital human. The control of the digital human by each group of expression coefficients in chronological order can drive the digital human to make a corresponding expression sequence, forming an expression animation effect.

[0085] Specifically, after obtaining the output results of each module, the output results of each module are further input into the expression coefficient generation module for processing, for example, Figure 3 In the process, the audio features output by the audio driving module, the speech style features and speech emotion features output by the first control module, and the temporal text features output by the second control module are input into the expression coefficient generation module for processing, and the expression coefficients for driving the digital human are output, so as to drive the digital human to make corresponding expression movements based on the expression coefficients.

[0086] Among them, the expression coefficient generation module can be regarded as a diffusion model using the Transformer architecture, which trains and infers the model in a diffusion manner to generate expression coefficients.

[0087] In the above-mentioned method for generating expression coefficients for driving digital humans, audio data, first control information including speech style information and speech emotion information, and multi-modal second control information are used as driving data. After extracting the features of the data in each dimension, the data is input into the expression coefficient generation module for processing to obtain the expression coefficients for driving the digital humans. When generating expression coefficients, in addition to the audio signal, the method also introduces the first control information and the second control information. Through the speech style and speech emotion in the first control information, the emotional modeling of the digital human can be realized, so that the action expressions made by the digital human have emotional characteristics. Through the multi-modal timing control information in the second control information, the control of the digital human at different granularities such as the whole, part, and timing can be realized, and the control method of the digital human can be enriched, so that the driving results that meet expectations and are highly expressive can be simply obtained.

[0088] In an exemplary embodiment, in step S120, feature extraction is performed on the audio data through a voice driving module to obtain audio features, which can be specifically achieved in the following manner: in the voice driving module, the audio data is encoded to obtain audio coding information; lip shape coefficients are predicted based on the audio coding information to obtain lip shape regression coefficients; and audio features are generated based on the audio coding information and the lip shape regression coefficients.

[0089] Specifically, when extracting vital signs from audio data in the speech-driven module, the audio data can first be encoded through an audio encoder, such as Wave2Vec2.0 (a self-supervised speech representation learning model based on deep learning) or whisper (a series of pre-trained models for automatic speech recognition), to obtain audio coding information; then, the lip shape coefficient is predicted based on the audio coding information to obtain the lip shape regression coefficient; the audio coding information and the lip shape regression coefficient are mapped together through a linear layer to obtain audio features.

[0090] refer to Figure 4 , is a schematic diagram of the structure of a voice drive module shown in an embodiment, such as Figure 4 As shown, the speech-driven module is a small lip shape regressor, which is also based on the Transformer structure. This regressor predicts lip shape coefficients based on audio features in a regression manner. First, the audio data is encoded by the audio encoder to obtain the audio encoding information, and then the position encoding (Posion Embedding, PE) unit is used for position encoding. Then, it is processed by the Transformer encoder (Trans.Encoder) and decoder (Trans.Decoder) to obtain the decoded data. While processing the audio data, the lip shape regression coefficient is initialized to obtain the lip shape regression coefficient zeros that are all 0. The initialized lip shape regression coefficient is successively decoded by position encoding and decoder to obtain the lip shape regression coefficient. The decoded data and the lip shape regression coefficient are input into the linear layer (Proj layer) for mapping, and the audio features are output. .

[0091] In this embodiment, the audio data is encoded to obtain audio encoding information; the lip shape coefficient is predicted based on the audio encoding information to obtain the lip shape regression coefficient, and audio features are generated according to the audio encoding information and the lip shape regression coefficient. This can ensure that the generated lip shape result is consistent with the input audio data, so that the digital human can make lip shape movements consistent with the audio data.

[0092] In an exemplary embodiment, in step S120, the first control information is subjected to feature extraction processing by the first control module to obtain speech style features and speech emotion features, which can be specifically achieved in the following manner: in the first control module, the speech style information in the first control information is subjected to feature extraction processing to obtain speech style features; and, by adjusting the emotion intensity of the emotion category through trained adjustment parameters corresponding to the emotion category and emotion intensity, the speech emotion features are obtained.

[0093] The adjustment parameters are obtained through training and may include adjustment parameters corresponding to different emotion intensities, and each emotion category may have corresponding adjustment parameters. For example, for the emotion category: happy, assuming that the emotion intensity has 3 levels, the corresponding adjustment parameters can be regarded as a 3×N matrix, including 3 1×N vectors, each 1×N vector corresponding to an adjustment parameter under an emotion intensity.

[0094] In a specific implementation, since the speaking style information in the first control information can be a picture or a text, the speaking style information can be subjected to feature extraction processing through the image-text alignment model to obtain the speaking style feature. For example, the speaking style information is converted into a CLIP feature using a CLIP model (Contrastive Language-Image Pre-Training, a text-image pre-training model using contrastive learning), and the CLIP feature is used as the speaking style feature to realize the control of the speaking style of the digital human based on the two modalities of picture and text.

[0095] Each emotion category has a corresponding token (adjustment parameter). After determining the emotion category and emotion intensity in the first control information, the adjustment parameter corresponding to the emotion category can be first determined from a plurality of trained adjustment parameters, and then the target adjustment parameter corresponding to the emotion intensity can be determined from the adjustment parameters corresponding to the emotion category as the adjustment parameter corresponding to both the emotion category and the emotion intensity, so as to adjust and control the emotion category according to the emotion intensity, and use the obtained adjustment result as the speech emotion feature.

[0096] The above-mentioned speech style feature and speech emotion feature are used as the first control signal , thereby guiding the relatively coarse-grained generation of expressions of speaker style and emotions.

[0097] In this embodiment, the speaking style information in the first control information is subjected to feature extraction processing to obtain the speaking style feature; and the emotion intensity of the emotion category is adjusted by the trained adjustment parameters corresponding to the emotion category and the emotion intensity to obtain the speaking emotion feature. The two features are used together as a control signal, thereby realizing the control of the speaking style of the digital person by using text or pictures, and at the same time realizing the control of the speaking emotion and emotion intensity of the digital person, thereby improving the interactive effect of the head drive of the digital person.

[0098] In an exemplary embodiment, the second control information is subjected to feature extraction processing by the second control module to obtain a time-series text feature, including: in the second control module, respectively determining the action units corresponding to the control information of each mode in the second control information; combining the action units corresponding to the control information of each mode in chronological order to obtain time-series action combination control information; assembling the time-series action combination control information into text control information according to a preset sentence template; and performing feature extraction processing on the assembled text control information to obtain a time-series text feature.

[0099] The second control information refers to the timing control information that changes in timing. In order to simultaneously realize fine-grained control and multimodal control in timing, this embodiment uses AU (action units) signals as intermediate media signals, and obtains feature sequences after a series of processing as temporal text features. The AU signal refers to the signal of a single action unit of head movement, such as turning the head to the left, turning the head to the right, raising the head, lowering the head, opening the eyes wide, closing the eyes, etc.

[0100] It can be understood that multiple action units are used to express an emotion. For example, the action units corresponding to happy emotions may include cheek lifting actions and lip corner pulling actions; the action units corresponding to sad emotions may include inner eyebrow lifting actions, eyebrow lowering actions, and lip corner suppression actions. Therefore, the control information of each mode will also correspond to multiple action units, involving multiple AU signals. Therefore, the action units corresponding to the control information of each mode in the second control information can be determined first, and then the action units corresponding to the control information of each mode can be combined according to the time sequence corresponding to each mode to obtain the time sequence action combination control information. Figure 5 As shown in FIG. 1 , it is a schematic diagram of the sequential action combination control information. In the figure, AU a, AU b, and AU c represent three different action units, for example, corresponding to the actions of the forehead, chin, and lips, respectively. Figure 5 It can be clearly determined in terms of timing which AU signal will be activated and when.

[0101] Then, the timing action combination control information is Figure 6The sentence template "A person is talking with xx, and xx, and xx." is assembled into text to obtain time-series text control information. The time-series text control information is further feature extracted and converted into a time-series text control signal. The time-series text control signal is converted into a corresponding time-series text feature through a semantic extraction model, such as the CLIP model. .

[0102] In this embodiment, the action units corresponding to the control information of each mode in the second control information are combined in chronological order to obtain time-series action combination control information, and the time-series action combination control information is assembled into text control information according to a preset sentence template; feature extraction processing is performed on the assembled text control information to obtain time-series text features. Through this method, time-series text features at different times can be obtained, thereby realizing multi-modal and fine-grained control of the digital human in time.

[0103] In an exemplary embodiment, the expression coefficient generation module includes a first cross-attention unit, a second cross-attention unit, and a third cross-attention unit;

[0104] In step S130, the audio features, speaking style features, speaking emotion features, and time-series text features are input into the expression coefficient generation module for processing to obtain the expression coefficients used to drive the digital human, including:

[0105] Step S131, inputting the audio feature into the first cross attention unit to obtain the first attention feature; inputting the speaking style feature and the speaking emotion feature into the second cross attention unit to obtain the second attention feature; and inputting the temporal text feature into the third cross attention unit to obtain the third attention feature;

[0106] Step S132, generating an expression coefficient for driving the digital human based on the first attention feature, the second attention feature and the third attention feature.

[0107] Specifically, refer to Figure 7 , is a schematic diagram of the structure of an expression coefficient generation module shown in an embodiment, the expression coefficient generation module includes three cross attention units, denoted as a first cross attention unit, a second cross attention unit and a third cross attention unit, which are respectively used to process the output features of the speech driving module, the first control module and the second control module, for example, Figure 7 As shown, the audio features output by the voice driver module , the first control module outputs the speech style feature and the speech emotion feature , the time series text features output by the second control module The three cross-attention units are input for processing respectively, and three attention features are output. The three attention features are further fused to generate the expression coefficient for driving the digital human.

[0108] It can be understood that since the audio data and the second control information used to drive the digital human have corresponding time lengths, multiple sets of expression coefficients can be generated, and each set of expression coefficients is generated by the audio data of the corresponding number of frames, the first control information, and the second control information. Therefore, it is necessary to generate multiple expression coefficients, so the information input to the expression coefficient generation module also includes the parameters used to characterize the diffusion timestamp. , in order to change the parameters Into the expression coefficient generation module, a FiLM (feature-wise linear modulation) layer is used between the multi-head self-attention, multi-head cross-attention and feed-forward layers of the Transformer decoding block. Input the FiLM layer and introduce it into the expression coefficient generation module. Figure 7 The T in represents the total number of frames generated.

[0109] In this embodiment, three attention units are set in the expression coefficient generation module to facilitate processing of the output features of the three modules, namely the speech drive module, the first control module and the second control module, so that the generated expression coefficient integrates the speaking style, speaking emotion and fine-grained multimodal motion control, thereby improving the driving effect of the digital human.

[0110] In an exemplary embodiment, the speech emotion information includes an emotion category and an emotion intensity, and the emotion intensity is controlled by an adjustment parameter corresponding to the emotion category; Figure 8 As shown, the method also includes an expression coefficient generation module and a training process for adjusting parameters, including the following steps:

[0111] Step S810, obtaining sample audio data, sample first control information, sample second control information and true expression coefficient;

[0112] Step S820, performing feature extraction processing on the sample audio data through the voice driving module to obtain sample audio features; performing feature extraction processing on the speaking style information in the sample first control information through the first control module to obtain sample speaking style features, and converting the speaking emotion information in the sample first control information into sample speaking emotion features through the initialized adjustment parameters; performing feature extraction processing on the sample second control information through the second control module to obtain sample time series text features;

[0113] Step S830, using sample audio features, sample speech style features, sample speech emotion features, and sample time-series text features as input variables, and using real expression coefficients as supervision information, train the expression coefficient generation module to be trained and the initialized adjustment parameters to obtain the expression coefficient generation module and adjustment parameters.

[0114] Specifically, the training of the expression coefficient generation model includes the training of the expression coefficient generation module and the adjustment parameters for controlling the intensity of emotions. The models involved in the voice drive module, the image-text alignment model in the first control module, and the models involved in the second control module can all be pre-trained.

[0115] It should be noted that the expression coefficient generation module can be a denoising diffusion probabilistic model (DenoisingDiffusion Probabilistic Models, DDPM), and the model is trained in a diffusion manner. The training includes gradually adding noise to generate a series of noisy expression coefficients, using the noisy expression coefficients and the corresponding real expression coefficients to calculate the loss value to train the expression coefficient generation module and adjust the parameters, and the reverse process gradually eliminates the noise through learning denoising.

[0116] It can be understood that the training process is to add different degrees of noise to the sample, and then input it into the expression coefficient generation module to predict the corresponding level of noise, which can directly predict the approximate result of the sample. Therefore, the loss function of the training can be the mean square error between the predicted result and the corresponding level of noise or the mean square error between the predicted result and the true result. Later, during inference, the result is restored by denoising, and the results predicted by the expression coefficient generation module are continuously sent to the expression coefficient generation module. After predicting different levels of noise, they are added to the predicted result as a new predicted result until the final result is obtained.

[0117] In this embodiment, by combining multiple information such as audio features, speaking style features, speaking emotion features and temporal text features, the expression coefficient generation module can more comprehensively understand the speaker's emotions and intentions, thereby generating more realistic and natural expressions. Different expressions can be generated through different control information, for example, the speaker's emotions, style and tone can be controlled, and the speaking emotion features can be adjusted by adjusting parameters to achieve fine control of the expression.

[0118] Further, in an exemplary embodiment, in step S820, the second control module performs feature extraction processing on the sample second control information to obtain the sample time series text feature, including:

[0119] Step S821, in the second control module, combining the action units corresponding to the control information of each mode in the sample second control information in time sequence to obtain time sequence action combination control information;

[0120] Step S822, performing mask processing on the timing action combination control information to obtain mask control information;

[0121] Step S823, adjusting the preset sentence template according to the sample first control information to obtain an adjusted sentence template; and randomly sorting the action units in the mask control information according to the adjusted sentence template to assemble them into time-series text control information;

[0122] Step S824, performing feature extraction processing on the assembled time-series text control information to obtain sample time-series text features.

[0123] Specifically, in order to avoid excessive reliance on the second control information, this embodiment also proposes to effectively decouple the three control signals so that they can achieve the expected control purpose respectively. Specifically, the action units corresponding to the control information of each mode in the sample second control information can be combined in time sequence to obtain the time sequence action combination control information, and then the action units in the second control information can be sparsely processed by masking the time sequence action combination control information.

[0124] In addition, the second control information may be enhanced, specifically, it may include two aspects of enhancement mechanisms. On the one hand, at the level of assembling text, when assembling the time-series action combination control information into the time-series text control information through the preset sentence template, the activated AUs may be randomly sorted according to their facial regions (such as eyebrows, nose, etc.), and the randomly sorted time-series action combination control information may be assembled into the time-series text control information. Fig. 9 In Figure (c), the original order of forehead, lips, ..., chin is shuffled to the order of chin, forehead, lips. On the other hand, a large language model (such as GPT, Large Language Model (LLM)) can be used to modify the preset sentence template. The large language model will modify the sentence based on the preset sentence template according to the emotion category and emotion intensity of the first control information and the activated AU, further improving the diversity of the sentence. For example, in Fig. 9 In Figure (d), the initial sentence template “A person is talking with xx,and xx,and xx.” is modified to “A person is laughingheartly with xx,and xx,and xx.”.

[0125] It should be noted that in actual applications, one or more of the above mask strategies and two enhancement mechanisms can be selected according to needs for corresponding processing, without having to select all of them.

[0126] In this embodiment, the second control information is sparsely distributed through a masking strategy to avoid excessive reliance on high-frequency control signals. By adjusting the preset sentence template through sample first control information, the diversity of sentences can be further improved. By randomly sorting action units when assembling text, the model can be helped to fully extract control features.

[0127] In an exemplary embodiment, step S922, performing mask processing on the timing action combination control information to obtain the mask control information includes: performing mask processing on a single action unit and / or a combination of action units in the timing action combination control information to obtain the mask control information.

[0128] Specifically, the sparseness of the second control information by the mask strategy can be achieved in two ways. One is to perform a single action unit mask (single AU mask) at the Boolean action unit level, such as Fig. 9 As shown in Figure (a) in Figure 1, each mask completely masks a section of continuously activated AUs, so that the AUs in the second control information are sparse, which helps to decouple frequently co-activated AUs. The other is to perform a mask (group AUmask) on the temporal level, such as Fig. 9 As shown in Figure (b), each mask will completely mask out a section of all stable and unchanged AU combinations, so that all AUs in this time period are masked out, thereby making the high-frequency signal sparse in the time dimension.

[0129] In this embodiment, masking is performed by using these two masking strategies of a single action unit and a combination of action units, so that the second control information better meets the requirements of sparsity in time and space, and reduces the dependence of the model on the second control information.

[0130] In an exemplary embodiment, in order to facilitate those skilled in the art to understand the embodiment of the present disclosure, the following will be described with reference to specific examples of the accompanying drawings. Fig.10 , is a schematic diagram of the overall framework of a generation model for driving the expression coefficient of a digital human according to an exemplary embodiment. Fig.10As shown in the figure, the model adopts the transformer architecture, and performs model training and reasoning in a diffusion manner to generate expression coefficients. And three cross-attention units are adopted to inject audio signals, the first control signal (representing the low-frequency overall control signal, including the speaker's style, emotion category, and emotion intensity) and the second control signal (representing the high-frequency fine-grained control signal, the time-series muscle signal) to guide the generation of expression coefficients. The following describes how the control signals at different levels are designed and decoupled, so as to achieve effective control of the generation results in a multi-level and multi-modal manner.

[0131] (1) Speech drive module. The audio signal is encoded by the audio encoder to obtain audio encoding information, which is then processed by the lip encoder to obtain lip-type encoding information (lip-type regression coefficient). The audio encoding information and lip-type encoding information are mapped to obtain audio features. .

[0132] (2) The first control module. For the speaking style information in the form of pictures or texts, it is converted into speaking style features through the image-text alignment model; for the speaking emotion information, the speaking emotion features are obtained by multiplying the learnable token by the emotion intensity. The speaking style features and speaking emotion features are recorded as .

[0133] (3) The second control module. The input is multi-modal control information. The multi-modality may include AU tags, pictures, and natural language. The control information of different modes can be converted into a time-series AU combined control signal (intermediate medium signal) to achieve simple control of multiple modes. Specifically,

[0134] For AU tags: the corresponding AU signals can be directly activated to guide the corresponding facial muscles to make movements.

[0135] For pictures: You can use the existing Openface (a face recognition model) tool to detect the AU activation status of the reference picture, and generate head animation according to the guidance of the picture.

[0136] For natural language: natural language can be converted into corresponding AU activation state through a large language model, and head animation can be generated according to the guidance of natural language.

[0137] The activated AUs are assembled into text according to the preset sentence template and converted into a time-series text control signal. The time-series text control signal is converted into the corresponding time-series text features through the semantic extraction model. .

[0138] (4) Audio features , speaking style characteristics and speaking emotion characteristics , temporal text features Input the expression coefficient generation module for processing and output the expression coefficient sequence.

[0139] In this embodiment, on the basis of voice-driven 3D digital humans, control signals of different modalities (labels, pictures, natural languages) can be used to guide the generation of different control granularities (overall, local, timing) and control dimensions (style, emotion, muscle), such as using pictures or texts to control the speaker's style; the speaker's emotions and their intensity can be controlled; fine-grained and multi-modal facial muscle movement control can be performed in timing, such as 10-15 frames of given AU signals for control, 30-45 frames of given reference pictures for control, and 60-80 frames of given natural languages ​​for control.

[0140] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0141] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0142] Based on the same inventive concept, the embodiment of the present disclosure also provides a device for generating expression coefficients of a digital human for realizing the above-mentioned method for generating expression coefficients of a digital human.

[0143] Fig.11 is a structural block diagram of a device for driving a digital human's expression coefficient generation according to an exemplary embodiment. Fig.11 , the device comprises:

[0144] The data acquisition unit 1110 is configured to acquire audio data, first control information, and second control information for driving the digital human; the first control information includes speech style information and speech emotion information, and the second control information includes control information of multiple modes, and each mode of control information corresponds to controlling the facial movements of the digital human within a time period;

[0145] The feature extraction unit 1120 is configured to perform feature extraction processing on the audio data through the voice driving module to obtain audio features; perform feature extraction processing on the first control information through the first control module to obtain speaking style features and speaking emotion features; perform feature extraction processing on the second control information through the second control module to obtain time series text features;

[0146] The coefficient generating unit 1130 is configured to input the audio features, speaking style features, speaking emotion features, and temporal text features into the expression coefficient generating module for processing to obtain the expression coefficients for driving the digital human.

[0147] In an exemplary embodiment, the feature extraction unit 1120 is further configured to execute in the speech drive module, encode the audio data to obtain audio coding information; predict the lip shape coefficient based on the audio coding information to obtain the lip shape regression coefficient; and generate audio features based on the audio coding information and the lip shape regression coefficient.

[0148] In an exemplary embodiment, the speech emotion information includes emotion category and emotion intensity; the feature extraction unit 1120 is also configured to execute in the first control module, perform feature extraction processing on the speech style information in the first control information to obtain a speech style feature; and adjust the emotion intensity of the emotion category through trained adjustment parameters corresponding to the emotion category and emotion intensity to obtain a speech emotion feature.

[0149] In an exemplary embodiment, the feature extraction unit 1120 is also configured to execute in the second control module to respectively determine the action units corresponding to the control information of each mode in the second control information; combine the action units corresponding to the control information of each mode in chronological order to obtain time-series action combination control information; assemble the time-series action combination control information into time-series text control information according to a preset sentence template; perform feature extraction processing on the assembled time-series text control information to obtain time-series text features.

[0150] In an exemplary embodiment, the expression coefficient generation module includes a first cross-attention unit, a second cross-attention unit and a third cross-attention unit; the coefficient generation unit 1130 is also configured to execute inputting the audio feature into the first cross-attention unit to obtain the first attention feature; inputting the speaking style feature and the speaking emotion feature into the second cross-attention unit to obtain the second attention feature; and inputting the temporal text feature into the third cross-attention unit to obtain the third attention feature; based on the first attention feature, the second attention feature and the third attention feature, generate the expression coefficient for driving the digital human.

[0151] In an exemplary embodiment, the speech emotion information includes emotion category and emotion intensity, and the emotion intensity is controlled by the adjustment parameter corresponding to the emotion category; the device also includes a model training unit, which is configured to execute the acquisition of sample audio data, sample first control information, sample second control information and real expression coefficient; perform feature extraction processing on the sample audio data through the voice driving module to obtain sample audio features; perform feature extraction processing on the speech style information in the sample first control information through the first control module to obtain sample speaking style features, and convert the speech emotion information in the sample first control information into sample speaking emotion features through the initialized adjustment parameters; perform feature extraction processing on the sample second control information through the second control module to obtain sample time-series text features; use the sample audio features, sample speaking style features, the sample speech emotion features, and the sample time-series text features as input variables, and use the real expression coefficient as supervision information to train the expression coefficient generation module to be trained and the initialized adjustment parameters to obtain the expression coefficient generation module and the adjustment parameters.

[0152] In an exemplary embodiment, the model training unit is also configured to execute in the second control module, combine the action units corresponding to the control information of each mode in the sample second control information in chronological order to obtain the time-series action combination control information; perform mask processing on the time-series action combination control information to obtain the masked control information; adjust the preset sentence template according to the sample first control information to obtain the adjusted sentence template; according to the adjusted sentence template, randomly sort the action units in the masked control information and assemble them into the time-series text control information; perform feature extraction processing on the assembled time-series text control information to obtain the sample time-series text feature.

[0153] In an exemplary embodiment, the model training unit is further configured to perform mask processing on a single action unit and / or a combination of action units in the timing action combination control information to obtain mask control information.

[0154] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0155] Fig.11 1 is a block diagram of an electronic device 1100 for implementing a method for generating expression coefficients for driving a digital human according to an exemplary embodiment. For example, the electronic device 1100 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0156] Reference Fig.11 The electronic device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1110 , an input / output (I / O) interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0157] The processing component 1102 generally controls the overall operation of the electronic device 1100, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1102 may include one or more modules to facilitate the interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate the interaction between the multimedia component 1108 and the processing component 1102.

[0158] The memory 1104 is configured to store various types of data to support operations on the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, an optical disk, or a graphene memory.

[0159] The power supply component 1106 provides power to various components of the electronic device 1100. The power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1100.

[0160] The multimedia component 1108 includes a screen that provides an output interface between the electronic device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. When the electronic device 1100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0161] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC), and when the electronic device 1100 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1104 or sent via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting an audio signal.

[0162] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0163] The sensor assembly 1114 includes one or more sensors for providing various aspects of status assessment for the electronic device 1100. For example, the sensor assembly 1114 can detect the open / closed state of the electronic device 1100, the relative positioning of the components, such as the display and keypad of the electronic device 1100, and the sensor assembly 1114 can also detect the position change of the electronic device 1100 or the components of the electronic device 1100, the presence or absence of user contact with the electronic device 1100, the orientation or acceleration / deceleration of the device 1100, and the temperature change of the electronic device 1100. The sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1114 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0164] The communication component 1116 is configured to facilitate wired or wireless communication between the electronic device 1100 and other devices. The electronic device 1100 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0165] In an exemplary embodiment, the electronic device 1100 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0166] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the instructions can be executed by a processor 1120 of the electronic device 1100 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0167] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 1120 of the electronic device 1100 to complete the above method.

[0168] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0169] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0170] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating expression coefficients for driving digital humans, characterized in that: include: Acquiring audio data, first control information, and second control information for driving the digital human; The first control information includes speaking style information and speaking emotion information, and the second control information includes control information of multiple modes, and the control information of each mode corresponds to controlling the facial movements of the digital human within a time period; Performing feature extraction processing on the audio data through a voice driving module to obtain audio features; Performing feature extraction processing on the first control information by a first control module to obtain a speaking style feature and a speaking emotion feature; Performing feature extraction processing on the second control information by a second control module to obtain time series text features; The audio features, the speaking style features, the speaking emotion features, and the time-series text features are input into an expression coefficient generation module for processing to obtain an expression coefficient for driving the digital human.

2. The method according to claim 1, characterized in that The step of performing feature extraction processing on the audio data by the voice driving module to obtain audio features includes: In the voice driving module, the audio data is encoded to obtain audio encoding information; Perform lip shape coefficient prediction based on the audio coding information to obtain a lip shape regression coefficient; The audio feature is generated according to the audio encoding information and the lip shape regression coefficient.

3. The method according to claim 1, characterized in that The speech emotion information includes emotion category and emotion intensity; the first control module performs feature extraction processing on the first control information to obtain speech style features and speech emotion features, including: In the first control module, feature extraction processing is performed on the speaking style information in the first control information to obtain the speaking style feature; Furthermore, the emotion intensity of the emotion category is adjusted by using the trained adjustment parameters corresponding to the emotion category and the emotion intensity to obtain the speech emotion feature.

4. The method according to claim 1, characterized in that: The step of performing feature extraction processing on the second control information by the second control module to obtain time series text features includes: In the second control module, action units corresponding to the control information of each mode in the second control information are respectively determined; Combine the action units corresponding to the control information of each mode in time sequence to obtain time sequence action combination control information; According to a preset sentence template, assembling the time sequence action combination control information into time sequence text control information; The assembled time-series text control information is subjected to feature extraction processing to obtain the time-series text feature.

5. The method according to claim 1, characterized in that The expression coefficient generation module includes a first cross attention unit, a second cross attention unit and a third cross attention unit; The step of inputting the audio features, the speaking style features, the speaking emotion features, and the time-series text features into an expression coefficient generation module for processing to obtain an expression coefficient for driving the digital human comprises: Inputting the audio feature into the first cross-attention unit to obtain a first attention feature; inputting the speaking style feature and the speaking emotion feature into the second cross-attention unit to obtain a second attention feature; and inputting the temporal text feature into the third cross-attention unit to obtain a third attention feature; Based on the first attention feature, the second attention feature and the third attention feature, an expression coefficient for driving the digital human is generated.

6. The method according to any one of claims 1 to 5, characterized in that: The speech emotion information includes an emotion category and an emotion intensity, and the emotion intensity is controlled by an adjustment parameter corresponding to the emotion category; the method further includes: Obtaining sample audio data, sample first control information, sample second control information and true expression coefficient; The sample audio data is subjected to feature extraction processing by the speech driving module to obtain sample audio features; the speaking style information in the sample first control information is subjected to feature extraction processing by the first control module to obtain sample speaking style features, and the speaking emotion information in the sample first control information is converted into sample speaking emotion features by initializing the adjustment parameters; the sample second control information is subjected to feature extraction processing by the second control module to obtain sample time series text features; Taking the sample audio features, the sample speech style features, the sample speech emotion features, and the sample time-series text features as input variables and the real expression coefficient as supervision information, the expression coefficient generation module to be trained and the initialized adjustment parameters are trained to obtain the expression coefficient generation module and the adjustment parameters.

7. The method according to claim 6, characterized in that The step of performing feature extraction processing on the sample second control information by the second control module to obtain sample time series text features includes: In the second control module, the action units corresponding to the control information of each mode in the sample second control information are combined in time sequence to obtain time sequence action combination control information; Performing mask processing on the timing action combination control information to obtain mask control information; According to the sample first control information, a preset sentence template is adjusted to obtain an adjusted sentence template; according to the adjusted sentence template, the action units in the mask control information are randomly sorted and assembled into time-sequence text control information; The assembled time-series text control information is subjected to feature extraction processing to obtain the sample time-series text features.

8. The method according to claim 7, characterized in that The performing mask processing on the timing action combination control information to obtain mask control information includes: Masking is performed on a single action unit and / or a combination of action units in the sequential action combination control information to obtain the mask control information.

9. A device for generating expression coefficients for driving a digital human, characterized in that: include: A data acquisition unit, configured to acquire audio data, first control information, and second control information for driving the digital human; The first control information includes speaking style information and speaking emotion information, and the second control information includes control information of multiple modes, and the control information of each mode corresponds to controlling the facial movements of the digital human within a time period; A feature extraction unit is configured to perform feature extraction processing on the audio data through a voice driving module to obtain audio features; Performing feature extraction processing on the first control information through a first control module to obtain a speaking style feature and a speaking emotion feature; performing feature extraction processing on the second control information through a second control module to obtain a time series text feature; The coefficient generating unit is configured to input the audio feature, the speaking style feature, the speaking emotion feature, and the temporal text feature into the expression coefficient generating module for processing to obtain the expression coefficient for driving the digital human.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for generating expression coefficients for driving a digital human as claimed in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for generating expression coefficients for driving a digital human as claimed in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating expression coefficients for driving a digital human according to any one of claims 1 to 8 is implemented.

Citation Information

Cited By

  • Digital human expression driving method and device, storage medium and computer equipment

    CN119538011A