A method for generating personalized digital teachers

Abstract

The invention relates to the technical field of digital human generation, and discloses a method for generating a personalized digital teacher, comprising: preprocessing an input video; respectively sending audio data and text data into a style encoder and a phoneme encoder of a speech generation module to obtain style features and a phoneme sequence, and obtaining a new phoneme sequence through a variance adapter; sending the style features and the new phoneme sequence together into an audio decoder of the speech generation module to obtain generated audio; obtaining image features, motion features and audio features; adding a group of noises to the motion features to obtain noise features, splicing audio features onto the noise features to obtain noise features containing audio information, inputting the noise features into a denoising module, and outputting generated motion features; inputting the generated motion features and image features into a decoder of a video generation module, and merging them with the generated audio to obtain a personalized digital teacher video; the authenticity and vividness of the generated digital teacher can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a method for generating a personalized digital teacher. Background Art

[0002] Personalized digital human generation refers to the process of generating a digital virtual human with the corresponding image by analyzing the facial and voice features of the target in the input video. This method is mainly based on two key technologies: video generation and speech synthesis, to generate a personalized digital human with a specified image and voice. With the rapid development of artificial intelligence, personalized digital humans have shown broad application potential in many fields such as education, services and entertainment. In the field of education, personalized digital teacher generation can customize the exclusive teacher image according to the students' learning habits, which can effectively improve students' learning interest and participation. Therefore, generating a more realistic digital teacher that better meets the needs of students can effectively stimulate students' enthusiasm for learning and enhance their learning experience. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a method for generating a personalized digital teacher.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A method for generating a personalized digital teacher, wherein the generative model used can extract the facial features and voice features of a teacher in an input video, thereby generating a digital teacher with the same appearance and voice as the teacher in the video, and injecting personalization into the digital teacher through the facial change features of the teacher in the video; the construction and training process of the generative model includes the following steps:

[0006] S1, preprocessing the input video to obtain image data, video data, audio data and text data;

[0007] S2: Send the audio data and text data to the style encoder and phoneme encoder of the speech generation module respectively to obtain the style feature F t and phoneme sequence P0, and then adjust P0 to the speech frame length through the variance adapter of the speech generation module to obtain a new phoneme sequence P1;

[0008] S3, the style feature F t And the new phoneme sequence P1 are sent to the audio decoder of the speech generation module to obtain the generated audio. Based on the generated audio, the L1 distance is used as the reconstruction loss L c ;

[0009] S4, sending the image data, video data and generated audio to the image encoder, motion encoder and audio encoder of the video generation module respectively to obtain image features I, motion features M and audio features A;

[0010] S5, add a set of noise to the motion feature M to obtain the noise feature M0, and splice the audio feature A to the noise feature M t The noise characteristics containing audio information are obtained Will Input into the denoising module and output the generated motion feature M t ′, and construct the mean square error loss L m ;

[0011] S6, the generated motion feature M t ′ The image feature I is input into the decoder of the video generation module, and the generated video is output. After being merged with the generated audio, the final generated personalized digital teacher video is obtained. Based on the reconstruction loss L c and variance loss L m Train the generative model.

[0012] Furthermore, step S1 specifically includes:

[0013] A video with a teacher's image is input and preprocessed in the following way: first, the input video is divided into multiple video segments, and the video and audio are separated in each video segment to obtain video data VD and audio data AD; then, a frame in the input video is randomly selected as the image data ID; using speech conversion technology, the text data TD corresponding to the audio data is obtained.

[0014] Furthermore, step S2 specifically includes:

[0015] The input of the style encoder of the speech generation module is the audio data AD; the style encoder consists of a fully connected layer, a convolutional neural network CNN and a multi-head self-attention module; each frame of audio data AD is converted into a hidden sequence through the fully connected layer, and then the sequence information in the audio is extracted through the convolutional neural network CNN. Finally, the multi-head self-attention module is applied to encode the global information, and the output of the attention is averaged in the time dimension to obtain the style feature F t ;

[0016] The input of the phoneme encoder of the speech generation module is the text data TD; firstly, the text data TD is preprocessed, including text cleaning, word segmentation and text normalization operations; then, each character or word in the preprocessed text data is converted into the corresponding phoneme using the predefined phoneme library; finally, the phonemeized text sequence is input into the phoneme encoder and encoded into the phoneme sequence P0;

[0017] The input of the variance adapter of the speech generation module is the style feature F t And the phoneme sequence P0, the variance adapter predicts the variance of different phonemes and the duration of each phoneme, thereby adjusting the length of the phoneme sequence to the length of the speech frame and obtaining a new phoneme sequence P1.

[0018] Furthermore, step S3 specifically includes:

[0019] The input of the audio decoder is the style feature F t and the new phoneme sequence P1, which is converted through an audio decoder based on the Transformer architecture to obtain the generated audio AD′;

[0020] The L1 distance between the generated audio AD′ and the audio data AD is taken as the reconstruction loss L c :

[0021] L c =||AD′-AD||1.

[0022] Furthermore, step S4 specifically includes:

[0023] The input of the video generation module includes image data ID, video data VD and generated audio AD′; first, the image encoder is used to extract features from the input image data ID, model the appearance features of the teacher, and obtain image features I; then, the motion encoder is used to decouple the motion information from the input video data VD to obtain motion features M; finally, the pre-trained audio encoder is used to extract features from the generated audio AD′ to obtain audio features A.

[0024] Furthermore, step S5 specifically includes:

[0025] Based on the idea of ​​diffusion model, a motion generator module is constructed in the video generation module. During the training process, the input of the motion generator module is the motion feature M. First, a set of random noise N is added to the motion feature M to obtain the noise feature M. t During the test, random noise is directly used as the noise feature; therefore, the audio feature A is added element by element to the noise feature M0 to obtain the noise feature containing audio information Will Input into the denoising module to obtain the generated motion feature M t ′; The denoising module consists of 12 stacked denoising blocks, each of which includes a feedforward layer, a convolutional layer, a multi-head self-attention module, and a layer normalization operation;

[0026] Based on generating motion features M t ′ and motion feature M to construct mean square error loss L m:

[0027]

[0028] Further, in step S6, the reconstruction loss L c and variance loss L m Training the generative model includes:

[0029] Reconstruction loss L c and variance loss L m The total loss function L is constructed, and the generative model is trained based on the total loss function L:

[0030] L=L c +L m .

[0031] Compared with the prior art, the beneficial technical effects of the present invention are:

[0032] The present invention effectively combines the speech synthesis module and the video generation module to construct a personalized digital teacher generation model; the teacher characteristics are modeled from a multimodal perspective, firstly the appearance characteristics and the motion characteristics are decoupled to prevent information interference; at the same time, in the process of generating motion information, the audio characteristics are integrated as additional information to generate more accurate motion information. In this way, the authenticity and vividness of the generated digital teacher can be improved, and students can customize personalized digital teachers, which will effectively stimulate students' interest in learning and provide favorable support for the learning effect of digital teacher subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of model training of the present invention. DETAILED DESCRIPTION

[0034] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0035] The present invention performs multimodal information fusion and intelligent analysis modeling to obtain a generation model of a personalized digital teacher, which can mine the characteristics of the teacher from three dimensions of image, video and audio according to the teacher video input by the students, and is used to generate a personalized digital teacher teaching video.

[0036] like Figure 1 As shown, the present invention proposes a method for generating a personalized digital teacher, which constructs input data of four modalities through teacher videos input by students, and respectively utilizes a voice generation module and a video generation module to fuse multimodal information, thereby generating a real and vivid personalized digital teacher video.

[0037] like Figure 1As shown, a method for generating a personalized digital teacher in the present invention, the construction and training process of the generation model adopted includes the following steps:

[0038] S1, preprocessing the video data to obtain four types of input data, namely image data, video data, audio data and text data.

[0039] Step S1 specifically includes the following steps:

[0040] S11, students input the teacher video that needs to be customized, and the video is stored in mp4 format;

[0041] S12, dividing the video into multiple 10s video segments, and separating the video and audio in each video segment to obtain video data VD and audio data AD;

[0042] S13, randomly selecting a frame in the video as the image data ID;

[0043] S14, using Whisper voice conversion to obtain text data TD corresponding to the audio.

[0044] S2, send the audio data and text data to the style and phoneme encoders respectively to obtain the style feature F t And the phoneme sequence P0, then P0 is adjusted to the speech frame length through a variance adapter to obtain a new phoneme sequence P1.

[0045] Step S2 specifically includes the following steps:

[0046] S21, use the style encoder to extract features from the input audio data AD. The audio data AD is converted into a hidden sequence through a fully connected layer, and then the sequence information in the audio is extracted through a convolutional neural network CNN. Finally, a multi-head self-attention module is applied to encode the global information and average the output of the attention in the time dimension to obtain the style feature F t ;

[0047] S22, using the function provided by the Chinese word segmentation tool Jieba, to segment the text into independent vocabulary units;

[0048] S23, using a predefined phoneme library, converting each character or word in the text into a corresponding phoneme;

[0049] S24, using a phoneme encoder to encode the text sequence that has been phonemed, extract its inherent speech features, and obtain a phoneme sequence P0;

[0050] S25, using a variance adapter to predict different variances in the audio, and predicting the duration of each phoneme, thereby adjusting the length of the phoneme sequence to the length of the speech frame, and obtaining a new phoneme sequence P1.

[0051] S3, the style feature F t It is sent to the audio decoder together with the phoneme sequence P1, and the generated audio is output. The speech generation module is trained using the L1 distance as the reconstruction loss.

[0052] Step S3 specifically includes the following steps:

[0053] S31, using an audio decoder based on the Transformer architecture, t and the phoneme sequence P1 to obtain the corresponding generated audio AD′;

[0054] S32, performing L1 loss on the generated audio AD′ and the original audio data AD to train the parameters of the entire model. The specific L1 loss function is:

[0055]

[0056] S4, sending the image data, video data and generated audio to the image encoder, motion encoder and audio encoder respectively to obtain image features I, motion features M and audio features A.

[0057] Step S4 specifically includes the following steps:

[0058] S41, using an image encoder to extract features from the input image data ID, modeling the appearance features of the teacher, and obtaining image features I;

[0059] S42, decoupling motion information from the input video data VD using a motion encoder to obtain a motion feature M;

[0060] S43, using a pre-trained audio encoder Wave2Vec to perform feature extraction on the generated audio AD′ to obtain an audio feature A;

[0061] S5, add a set of noise to the motion feature M to obtain the noise feature M0, and splice the audio feature A to the noise feature M t Get features on Will Input into the denoising module and output the generated motion feature M t ′, and the model is trained using mean square error loss.

[0062] Step S5 specifically includes the following steps:

[0063] S51, during the training process, a set of random noise N is added to the motion feature M to obtain the noise feature M t ; In the test case, since there is no motion feature, the randomly initialized noise N is directly used as the noise feature;

[0064] S52, considering that the audio and motion sequences are semantically aligned in time, the audio feature A is added element by element to the noise feature M0 to obtain the noise feature containing the audio information

[0065] S53, using a denoising module consisting of 12 layers of denoising blocks to remove noise features The noise in the image is used to obtain the denoised motion feature M t ′, each denoising block consists of four parts: feedforward layer, convolutional layer, multi-head self-attention module and layer normalization;

[0066] S53, the finally obtained motion feature M t ′ and the motion feature M extracted by the original action encoder are subjected to mean square error loss to train the parameters of the entire model. The mean square error loss L m for:

[0067]

[0068] S6, the generated motion feature M t ′ The image features I are input into the decoder, the generated video is output, and after being merged with the audio generated by the speech generation module, the final generated personalized digital teacher video is obtained.

[0069] Step S6 specifically includes the following steps:

[0070] S61, using the decoder to decode the motion feature M t ′ Process the image feature I to generate a video result;

[0071] S62, merging the audio generated by the speech generation module with the video to obtain a final generated personalized digital teacher video;

[0072] S63, the parameters of the entire generation model are trained through the total loss function L, and the total loss function L can be expressed as:

[0073] L=L c +L m ;

[0074] The parameters of the generative model are updated to minimize the overall loss of the generative model. This process is repeated until the performance of the model reaches a satisfactory level or the preset number of training rounds is reached.

[0075] The trained generative model can extract the facial and voice features of the teacher in the input video, thereby generating a digital teacher with the same appearance and voice as the teacher in the video, and injecting personalization into the digital teacher through the facial change features of the teacher in the video.

[0076] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0077] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for generating a personalized digital teacher, characterized in that: The generative model used can extract the facial features and voice features of the teacher in the input video, thereby generating a digital teacher with the same appearance and voice as the teacher in the video, and injecting personalization into the digital teacher through the facial change features of the teacher in the video; The process of building and training a generative model consists of the following steps: S1, preprocessing the input video to obtain image data, video data, audio data and text data; S2: Send the audio data and text data to the style encoder and phoneme encoder of the speech generation module respectively to obtain the style feature F t and phoneme sequence P0, and then adjust P0 to the speech frame length through the variance adapter of the speech generation module to obtain a new phoneme sequence P1; S3, the style feature F t And the new phoneme sequence P1 are sent to the audio decoder of the speech generation module to obtain the generated audio. Based on the generated audio, the L1 distance is used as the reconstruction loss L c ; S4, sending the image data, video data and generated audio to the image encoder, motion encoder and audio encoder of the video generation module respectively to obtain image features I, motion features M and audio features A; S5, add a set of noise to the motion feature M to obtain the noise feature M0, and splice the audio feature A to the noise feature M t The noise characteristics containing audio information are obtained Will Input into the denoising module and output the generated motion feature M t ′, and construct the mean square error loss L m ; S6, the generated motion feature M′ t and image features, which are input into the decoder of the video generation module, and the generated video is output. After being merged with the generated audio, the final generated personalized digital teacher video is obtained. Based on the reconstruction loss L c and variance loss L m Train the generative model.

2. The method for generating a personalized digital teacher according to claim 1, characterized in that: Step S1 specifically includes: A video with a teacher's image is input and preprocessed in the following way: first, the input video is divided into multiple video segments, and the video and audio are separated in each video segment to obtain video data VD and audio data AD; then, a frame in the input video is randomly selected as the image data ID; using speech conversion technology, the text data TD corresponding to the audio data is obtained.

3. The method for generating a personalized digital teacher according to claim 1, characterized in that: Step S2 specifically includes: The input of the style encoder of the speech generation module is the audio data AD; the style encoder consists of a fully connected layer, a convolutional neural network CNN and a multi-head self-attention module; each frame of audio data AD is converted into a hidden sequence through the fully connected layer, and then the sequence information in the audio is extracted through the convolutional neural network CNN. Finally, the multi-head self-attention module is applied to encode the global information, and the output of the attention is averaged in the time dimension to obtain the style feature F t ; The input of the phoneme encoder of the speech generation module is the text data TD; firstly, the text data TD is preprocessed, including text cleaning, word segmentation and text normalization operations; then, each character or word in the preprocessed text data is converted into the corresponding phoneme using the predefined phoneme library; finally, the phonemeized text sequence is input into the phoneme encoder and encoded into the phoneme sequence P0; The input of the variance adapter of the speech generation module is the style feature F t And the phoneme sequence P0, the variance adapter predicts the variance of different phonemes and the duration of each phoneme, thereby adjusting the length of the phoneme sequence to the length of the speech frame and obtaining a new phoneme sequence P1.

4. The method for generating a personalized digital teacher according to claim 1, characterized in that: Step S3 specifically includes: The input of the audio decoder is the style feature F t and the new phoneme sequence P1, which is converted through an audio decoder based on the Transformer architecture to obtain the generated audio AD′; The L1 distance between the generated audio AD′ and the audio data AD is taken as the reconstruction loss L c : L c =||AD′-AD||1。 5. The method for generating a personalized digital teacher according to claim 1, characterized in that: Step S4 specifically includes: The input of the video generation module includes image data ID, video data VD and generated audio AD′; first, the image encoder is used to extract features from the input image data ID, model the appearance features of the teacher, and obtain image features I; then, the motion encoder is used to decouple the motion information from the input video data VD to obtain motion features M; finally, the pre-trained audio encoder is used to extract features from the generated audio AD′ to obtain audio features A.

6. The method for generating a personalized digital teacher according to claim 1, characterized in that: Step S5 specifically includes: Based on the idea of ​​diffusion model, a motion generator module is constructed in the video generation module. During the training process, the input of the motion generator module is the motion feature M. First, a set of random noise N is added to the motion feature M to obtain the noise feature M. t During the test, random noise is directly used as the noise feature; therefore, the audio feature A is added element by element to the noise feature M0 to obtain the noise feature containing audio information Will Input into the denoising module to obtain the generated motion feature M t ′; The denoising module consists of 12 stacked denoising blocks, each of which includes a feedforward layer, a convolutional layer, a multi-head self-attention module, and a layer normalization operation; Based on generating motion features M t ′ and motion feature M to construct mean square error loss L m :

7. The method for generating a personalized digital teacher according to claim 1, characterized in that: In step S6, the reconstruction loss L c and variance loss L m Training the generative model includes: Reconstruction loss L c and variance loss L m The total loss function L is constructed, and the generative model is trained based on the total loss function L: L=L c +L m .

Images