Virtual face construction method, device, computer equipment and computer readable medium
By converting the target text into phoneme text and inputting it into the recurrent neural network model, the problem of inefficiency in traditional virtual anchor technology is solved, and efficient and accurate virtual anchor generation is achieved.
Patent Information
- Application Number
- CN202011285914.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-17
AI Technical Summary
The construction efficiency of traditional virtual anchor technology is low, and it requires recording a large amount of audio and image data for generalization learning, resulting in low efficiency and low accuracy of virtual anchor generation.
By obtaining the target text, generating audio using TTS, and performing phoneme segmentation, inputting phoneme text into the recurrent neural network model to obtain the face characteristics of the live anchor, and building virtual faces based on these characteristics, so as to directly construct virtual anchors based on the input text.
The virtual anchor generation program is simplified, the generation efficiency and accuracy are improved, the demand for real anchors to record data, and the time cost of generating virtual anchors is reduced.
Smart Images

Figure CN112420014B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and computer-readable storage medium for constructing a virtual face based on phoneme text. Background Art
[0002] Virtual anchors are anchors or customer service representatives who use virtual images to interact with customers in videos based on voice, NLP, and vision technologies. Virtual anchors can solve problems such as high costs of traditional customer service seats (such as the need for salary and benefits, high management and training costs, and long cycles), unstable work quality (easily affected by emotions, fatigue, etc.), etc., reduce the company's customer service costs, improve customer satisfaction, and ensure the stability of work quality.
[0003] At present, traditional virtual anchor technologies are mostly based on audio or audio combined with images to construct virtual video images, which requires recording corresponding audio or audio and images as input. Different virtual anchors need to learn different audio features during the customization process, or collect a large amount of video / audio data recorded by different real anchors for generalized learning. Therefore, there is a technical problem of low efficiency in constructing virtual anchors in traditional technologies. Summary of the invention
[0004] The present application provides a method, apparatus, computer device and computer-readable storage medium for constructing a virtual face based on phoneme text, which can solve the problem of low efficiency in constructing a virtual anchor in traditional technologies.
[0005] In a first aspect, the present application provides a method for constructing a virtual face based on phonemic text, the method comprising: obtaining a target text, and generating audio from the target text through a preset TTS method; performing phoneme segmentation on the audio to obtain a phonemic text corresponding to the audio; inputting the phonemic text into a preset recurrent neural network model to obtain facial features of a preset real-life host corresponding to the phonemic text; and constructing a virtual face of the preset real-life host corresponding to the phonemic text based on the facial features of the preset real-life host.
[0006] In the second aspect, the present application also provides a virtual face construction device based on phoneme text, including: a first acquisition unit, used to acquire a target text, and generate audio from the target text through a preset TTS method; a first segmentation unit, used to perform phoneme segmentation on the audio to obtain the phoneme text corresponding to the audio; a first input unit, used to input the phoneme text into a preset recurrent neural network model to obtain the facial features of a preset real-life host corresponding to the phoneme text; a construction unit, used to construct a virtual face of the preset real-life host corresponding to the phoneme text according to the facial features of the preset real-life host.
[0007] In a third aspect, the present application also provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for constructing a virtual face based on phoneme text are implemented.
[0008] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the method for constructing a virtual face based on phoneme text.
[0009] The present application provides a method, device, computer equipment and computer-readable storage medium for constructing a virtual face based on phoneme text. The present application obtains a segment of input target text, generates audio from the target text through a preset TTS method, performs phoneme segmentation on the audio to obtain the phoneme text corresponding to the audio, inputs the phoneme text into a preset recurrent neural network model to obtain the facial features of a preset real-person host corresponding to the phoneme text, and constructs a virtual face of the preset real-person host corresponding to the phoneme text according to the facial features of the preset real-person host. The virtual face corresponding to the preset real-person host can be directly constructed according to the input target text, which simplifies the generation procedure of converting a real-person host into a corresponding virtual host, and improves the generation efficiency and accuracy of the virtual host. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0011] Figure 1 A flowchart of a method for constructing a virtual face based on phoneme text provided in an embodiment of the present application;
[0012] Figure 2 A schematic diagram of a first sub-process of a method for constructing a virtual face based on phoneme text provided in an embodiment of the present application;
[0013] Figure 3 A schematic diagram of a second sub-process of the method for constructing a virtual face based on phoneme text provided in an embodiment of the present application;
[0014] Figure 4 A schematic diagram of a third sub-process of the method for constructing a virtual face based on phoneme text provided in an embodiment of the present application;
[0015] Figure 5 A schematic diagram of a fourth sub-process of the method for constructing a virtual face based on phoneme text provided in an embodiment of the present application;
[0016] Figure 6 A fifth sub-process diagram of the method for constructing a virtual face based on phoneme text provided in an embodiment of the present application;
[0017] Figure 7 A schematic block diagram of a device for constructing a virtual face based on phoneme text provided in an embodiment of the present application; and
[0018] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0020] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0021] See also Figure 1 , Figure 1 A flowchart of a method for constructing a virtual face based on phoneme text provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps S11-S14:
[0022] S11, obtaining a target text, and generating an audio from the target text through a preset TTS method.
[0023] Among them, TTS, which stands for Text To Speech in English, is the process of intelligently converting text into speech.
[0024] Specifically, in an embodiment of the present application, when constructing a virtual face based on a phoneme text, it is only necessary to input a target text, and the construction of a virtual face can be realized according to the input target text. After obtaining the target text, the target text is used to generate the audio corresponding to the target text through a preset TTS method. Among them, TTS includes a "splicing method" and a "parameter method". The splicing method is to prepare a large number of voices, which are spliced together by basic units (basic units such as syllables, phonemes, etc.), and then extract them from the prepared sounds to synthesize the target sound. The parameter method is to generate the voice parameters (including fundamental frequency, resonance peak frequency, etc.) at every moment according to the statistical model, and then convert these parameters into waveforms.
[0025] S12: segment the audio into phonemes to obtain phoneme text corresponding to the audio.
[0026] Among them, phoneme (Phone in English) is the smallest speech unit divided according to the natural properties of speech. It is analyzed based on the pronunciation actions in the syllable, and one action constitutes a phoneme.
[0027] Specifically, after generating audio based on the acquired target text, the audio is segmented and aligned to obtain the factor text corresponding to the text. Subsequently, a virtual face is constructed based on the factor text, and when the virtual face and the audio are played as a virtual video, the factor text corresponds to each frame in the virtual video.
[0028] S13. Input the phoneme text into a preset recurrent neural network model to obtain facial features of a preset live host corresponding to the phoneme text.
[0029] S14. Constructing a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor.
[0030] Among them, the recurrent neural network is called Recurrent Neural Network, abbreviated as RNN.
[0031] Specifically, first, a preset recurrent neural network model is trained. The preset recurrent neural network model is trained by recording a training video sample of a real anchor, segmenting the training audio sample contained in the training video sample to obtain a training phoneme text, and inputting the training phoneme text and the image of the real anchor corresponding to the training phoneme text contained in the training video sample into the preset recurrent neural network model, so that the recurrent neural network model automatically learns the correlation between the training phoneme text and the facial features of the real anchor of the image of the real anchor corresponding to the training phoneme text, thereby matching the training phoneme text with the facial features of the real anchor contained in the image of the real anchor.
[0032] After the phoneme text corresponding to the acquired target text is input into the preset recurrent neural network model, the preset recurrent neural network model can obtain the facial features of the preset real-life anchor contained in each video frame corresponding to the phoneme text based on the phoneme text and the association between the trained phoneme text and the facial features of the real-life anchor contained in the image of the real-life anchor during training. According to the facial features of the preset real-life anchor and the relationship between facial features and faces, a virtual face corresponding to the facial features of the preset real-life anchor can be constructed to obtain the virtual face of the preset real-life anchor corresponding to the phoneme text, render the virtual face, and then correspond the virtual face to the audio to obtain the virtual anchor video.
[0033] In an embodiment of the present application, a target text is obtained, and an audio is generated from the target text through a preset TTS method, the audio is segmented into phonemes to obtain a phoneme text corresponding to the audio, and the phoneme text is input into a preset recurrent neural network model to obtain facial features of a preset real-life host corresponding to the phoneme text. According to the facial features of the preset real-life host, a virtual face of the preset real-life host corresponding to the phoneme text is constructed. In this way, a virtual face corresponding to the preset real-life host can be directly constructed according to the input target text, thereby simplifying the generation procedure of converting a real-life host into a corresponding virtual host, and improving the generation efficiency and accuracy of the virtual host.
[0034] See also Figure 2 , Figure 2 This is a schematic diagram of the first sub-process of the method for constructing a virtual face based on phoneme text provided in the embodiment of the present application. Figure 2 As shown, in this embodiment, before the step of inputting the phoneme text into a preset recurrent neural network model to obtain the facial features of a preset live anchor corresponding to the phoneme text, it also includes:
[0035] S21, obtaining a training video sample recorded by a preset real-person anchor, wherein the training video sample includes a training audio sample and an image of the preset real-person anchor;
[0036] S22, performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting facial features of the training live host corresponding to the training phoneme text from the image of the preset live host;
[0037] S23, inputting the training phoneme text and the facial features of the training real-person anchor into a preset recurrent neural network model to train the preset recurrent neural network model.
[0038] Specifically, before the step of inputting the phoneme text into a preset recurrent neural network model to obtain facial features of a preset real-life anchor corresponding to the phoneme text, the preset recurrent neural network model is trained using training video samples corresponding to the preset real-life anchor. When using the training video samples corresponding to the preset real-life anchor to train the preset recurrent neural network model, the training video samples corresponding to the preset real-life anchor are recorded. The training video samples corresponding to the preset real-life anchor within a preset time period can be recorded. The preset real-life anchor is required to record the front face and expose the whole face. During recording, fill light and microphone recording can be performed. For example, training video samples with a length of 2 to 3 hours can meet the training requirements of the preset recurrent neural network model in the embodiment of the present application. The training video samples include training audio samples and the image of the preset real-life anchor. The training audio samples are phoneme segmented to obtain training phoneme text, and the facial features of the training real-life anchor corresponding to the training phoneme text are extracted from the training video samples. For example, the 3DMM features of the face (i.e., 3dmax's Morph, 3D deformable model) can be extracted, wherein each phoneme corresponds to each video frame of the video.
[0039] The training phoneme text and the facial features of the training live anchor are input into the preset recurrent neural network model to train the preset recurrent neural network model. For example, the phoneme text obtained above is embedded to obtain the input vector of the preset recurrent neural network model, and then the pre-extracted 3DMM 150-dimensional features are used as the label of the preset recurrent neural network model. Training the preset recurrent neural network model includes the following two processes:
[0040] 1) During forward propagation: the Embedding feature vector passes through two layers of bidirectional RNN to obtain a 150-dimensional feature Predict. 2) During backward propagation: MSE / MAE / L1 / L2 and other losses are calculated based on Predict and Label, and gradient updates are performed in reverse. When using the preset recurrent neural network model to build a virtual face, you only need to input the phoneme text (assuming the length is N). According to the forward propagation process mentioned above, you can get the N*150-dimensional 3DMM features of Ppredict, which correspond to each frame of the N frames of the video, and are used for face reconstruction of each frame in the subsequent virtual anchor, thereby obtaining a complete and continuous virtual anchor video.
[0041] Repeat steps 1) and 2) until convergence, so that the preset recurrent neural network model adjusts the parameters of the preset recurrent neural network model through automatic learning, and trains the preset recurrent neural network model using the training video samples corresponding to the preset real-life anchor, wherein different preset RNN models, different losses, and different optimization methods, as well as initialization methods of model parameters can be tried to adapt to different types of data. Since the embodiments of the present application can adopt training video samples of shorter duration, for example, recording for 2 to 3 hours, there is no need for long-term training audio and video sample data, which can meet the training requirements of the preset recurrent neural network model in the embodiments of the present application, that is, it is possible to train a recurrent neural network model with better effect, and realize the construction of an accurate virtual face according to the input target text. Compared with the traditional technology that requires a large number of training video samples or a long training video sample, the embodiments of the present application greatly simplifies the training process of the preset recurrent neural network model, avoids the problem of requiring a large amount of audio and video of the same host when training the preset recurrent neural network model, greatly reduces the amount of data used in the training process of the recurrent neural network model, improves the training efficiency and training accuracy of the preset recurrent neural network model, greatly reduces the amount of training video sample data required to customize a virtual host, and greatly reduces the time cost of generating a recurrent neural network model.
[0042] See also Figure 3 , Figure 3 This is a schematic diagram of the second sub-process of the method for constructing a virtual face based on phoneme text provided in the embodiment of the present application. Figure 3 As shown, in this embodiment, the steps of performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting the facial features of the training live host corresponding to the training phoneme text from the image of the preset live host include:
[0043] S31, obtaining a training audio sample corresponding to the training video sample;
[0044] S32, according to the chronological order, performing phoneme segmentation on the training audio sample according to preset phonemes to obtain a training phoneme text sequence arranged according to the chronological order, wherein the training phoneme text sequence includes a plurality of training phoneme text elements;
[0045] S33, acquiring, in the chronological order, a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame includes the image of the preset real-person anchor;
[0046] S34, extracting the appearance features, expression features and posture features of the preset real-person anchor contained in the video frame corresponding to the training phoneme text element, so as to obtain the facial features of the training real-person anchor corresponding to the training phoneme text.
[0047] Specifically, since the audio contained in the training video sample is a set of phonemes, the video contained in the training video sample is a set of video frames, and the phoneme elements in the phoneme set and the video frame elements in the video frame set correspond to each other in chronological order. Therefore, after obtaining the training video sample, obtain the training audio sample corresponding to the training video sample, and the training audio sample is a set of training phoneme text elements in chronological order. According to the chronological order, the training audio sample is segmented into phonemes according to preset phonemes to obtain a training phoneme text sequence containing several training phoneme text elements arranged in the chronological order. For example, in one example, the phoneme segmentation result is as follows:
[0048] Intervals[1]:xmin=0.000 xmax=0.700 text="sil";
[0049] Intervals[2]:xmin=0.700 xmax=0.780 text="HH";
[0050] Intervals[3]:xmin=0.780 xmax=0.920 text="IY1";
[0051] Intervals[4]:xmin=0.920 xmax=1.070 text="HH";
[0052] Among them, intervals[i] represents the i-th phoneme, xmin, xmax represent the start time and end time respectively, and so on.
[0053] According to the segmentation results, the factor text of the corresponding video frame is generated. In one example, the generated text is as follows:
[0054] ["sil","sil","sil","HH","HH""HH","IY1","IY1","IY1","HH","HH","HH","HH"].
[0055] Among them, each factor corresponds to each frame of the video.
[0056] According to the chronological order, the video frames corresponding to the training phoneme text elements are obtained from the training video samples, wherein the video frames contain the image of the preset real-life anchor. Then, according to the phoneme text elements, the facial features of the training real-life anchor contained in the video frames corresponding to each of the training phoneme text elements are extracted. The facial features of the training real-life anchor may include facial appearance features, expression features and posture features. Thus, the facial features of the training real-life anchor corresponding to the training phoneme texts arranged in chronological order can be obtained, thereby obtaining the training phoneme text sequence contained in the training audio sample and the facial features of the training real-life anchor in each video frame corresponding to the training phoneme text sequence.
[0057] When extracting facial features from training video samples, 3DMM facial features (3DMM, 3D Morphable Models in English, i.e., 3D deformation statistical models of faces) can be extracted from the training video samples. 3DMM features are extracted from each video frame of the training video samples, and [0-80]-dimensional Shape features (i.e., appearance features), [80-144]-dimensional expression features, and [224-227], [254-257] Pose features (i.e., posture features) can be selected to obtain facial features of the training live anchor, thereby obtaining rich live anchor facial features from the perspectives of appearance features, expression features, and posture features, which can improve the richness, vividness, and authenticity of the constructed virtual face. Among them, appearance features refer to the description of the appearance of the face by extracting key points of the face appearance, expression feature extraction refers to the positioning and extraction of organ features, texture areas, and predefined feature points of the face, and posture features refer to the posture characteristics of the face, such as the posture features of the face when looking up, looking down, or looking sideways.
[0058] See also Figure 4 , Figure 4 The third sub-process diagram of the method for constructing a virtual face based on phoneme text provided in the embodiment of the present application. Figure 4 As shown, in this embodiment, after the step of constructing the virtual face of the preset real anchor corresponding to the phoneme text according to the facial features of the preset real anchor, it also includes:
[0059] S41, obtaining all the phoneme texts contained in the audio and the virtual faces corresponding to the phoneme texts;
[0060] S42, combining all the virtual faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
[0061] S43: Combine the audio with the video frame sequence to obtain a virtual video.
[0062] Specifically, after constructing the virtual face corresponding to the phoneme text, all the phoneme texts contained in the audio are sorted according to the chronological order of the phoneme texts in the audio, so as to obtain a phoneme text sorting sequence consistent with the audio, and the virtual faces corresponding to the phoneme texts are sorted according to the order of the phoneme text sorting sequence, and all the virtual faces are combined into a video frame sequence, and then the audio is combined with the video frame sequence to obtain a virtual video, so that when constructing a virtual anchor video, there is no need to record audio data, and the phoneme text can be directly used to generate the corresponding video, thereby reducing the customer's usage cost and improving usage efficiency.
[0063] See also Figure 5 , Figure 5 This is a schematic diagram of the fourth sub-process of the method for constructing a virtual face based on phoneme text provided in an embodiment of the present application. Figure 5 As shown, in this embodiment, after the step of constructing the virtual face of the preset real anchor corresponding to the phoneme text according to the facial features of the preset real anchor, it also includes:
[0064] S51, generating other audios from the target text through other preset TTS modes, wherein the other audios and the audio are of different audio styles;
[0065] S52: Combine the other audio with the video frame sequence to obtain other virtual videos.
[0066] Specifically, for the same target text, different TTS (Text-To-Speech) tools can be used to generate audio of different styles. Since it is the same target text, the generated audio of different audio styles have the same factor text and the same time sequence corresponding to the factor text. By combining the audio of different styles with the same video frame sequence, videos with different audio style types and thus different video style types can be obtained. According to different scene requirements, it is possible to replace different sounds in the same video, and different sound support can be provided for the videos of the same host without re-recording the video, processing data and training, and adjusting the model to regenerate the model. This improves the efficiency of generating videos of various styles and the wide applicability of the generated videos.
[0067] See also Figure 6 , Figure 6 This is a schematic diagram of the fifth sub-process of the method for constructing a virtual face based on phoneme text provided in the embodiment of the present application. Figure 6 As shown, in this embodiment, before the step of generating audio from the target text in a preset TTS mode, it also includes:
[0068] S61, counting the text length corresponding to the target text;
[0069] S62, determining whether the text length is greater than or equal to a preset text length threshold;
[0070] S63: If the text length is greater than or equal to the preset text length threshold, executing the step of generating audio from the target text in a preset TTS mode;
[0071] S64: If the text length is less than the preset text length threshold, issue an alarm that the target text does not meet preset requirements.
[0072] Specifically, the target text belongs to an extreme case and does not meet the conditions for conversion to audio, for example, the target text is an interjection or a phrase, etc. In this case, the target text cannot be converted into speech, and then the virtual anchor video is constructed according to the target text. Therefore, before generating audio from the target text in a preset TTS mode, the text length corresponding to the target text can be counted first, for example, the number of characters corresponding to the characters contained in the target text can be counted, and the text length corresponding to the target text can be obtained, and then the text length can be used to determine whether the text meets the requirements for conversion to speech. First, determine whether the text length is greater than or equal to a preset text length threshold, for example, the preset text length threshold can be 2 characters or 5 characters, etc. If the target text is greater than or equal to the preset text length threshold, the step of generating audio from the target text in a preset TTS mode is performed. If the target text is less than the preset text length threshold, that is, the target text is too short, an alarm is issued that the target text does not meet the preset requirements, thereby avoiding the subsequent inability to construct a virtual anchor according to the target text that does not meet the requirements.
[0073] It should be noted that the method for constructing a virtual face based on phoneme text described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but all of them are within the scope of protection required by this application.
[0074] See also Figure 7 , Figure 7 A schematic block diagram of a virtual face construction device based on phoneme text provided in an embodiment of the present application. Corresponding to the above-mentioned virtual face construction method based on phoneme text, an embodiment of the present application also provides a virtual face construction device based on phoneme text. Figure 7As shown, the virtual face construction device based on phoneme text includes a unit for executing the above-mentioned virtual face construction method based on phoneme text, and the virtual face construction device based on phoneme text can be configured in a computer device. Specifically, please refer to Figure 7 The virtual face construction device 70 based on phoneme text includes a first acquisition unit 71, a first segmentation unit 72, a first input unit 73 and a construction unit 74.
[0075] The first acquisition unit 71 is used to acquire the target text and generate audio from the target text in a preset TTS mode;
[0076] A first segmentation unit 72 is used to segment the audio into phonemes to obtain a phoneme text corresponding to the audio;
[0077] A first input unit 73, used to input the phoneme text into a preset recurrent neural network model to obtain facial features of a preset live anchor corresponding to the phoneme text;
[0078] The construction unit 74 is used to construct a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor.
[0079] In one embodiment, the phoneme text-based virtual face construction device 70 further includes:
[0080] A second acquisition unit is used to acquire a training video sample recorded by a preset real-person anchor, wherein the training video sample includes a training audio sample and an image of the preset real-person anchor;
[0081] A second segmentation unit is used to perform phoneme segmentation on the training audio sample to obtain a training phoneme text, and extract facial features of the training live host corresponding to the training phoneme text from the image of the preset live host;
[0082] The training unit is used to input the training phoneme text and the facial features of the training real anchor into a preset recurrent neural network model to train the preset recurrent neural network model.
[0083] In one embodiment, the second segmentation unit includes:
[0084] A first acquisition subunit, used to acquire a training audio sample corresponding to the training video sample;
[0085] The first segmentation unit is used to segment the training audio sample into phonemes according to preset phonemes according to the time sequence to obtain a training phoneme text sequence arranged according to the time sequence, wherein the training phoneme text sequence includes a plurality of training phoneme text elements;
[0086] A second acquisition subunit is used to acquire, in accordance with the chronological order, a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame includes the image of the preset real-person anchor;
[0087] The extraction subunit is used to extract the appearance features, expression features and posture features of the preset real-life anchor contained in the video frame corresponding to the training phoneme text element, so as to obtain the facial features of the training real-life anchor corresponding to the training phoneme text.
[0088] In one embodiment, the phoneme text-based virtual face construction device 70 further includes:
[0089] A third acquisition unit, used to acquire all the phoneme texts contained in the audio and the virtual faces corresponding to the phoneme texts;
[0090] A first combining unit, used to combine all the virtual faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
[0091] The second combining unit is used to combine the audio with the video frame sequence to obtain a virtual video.
[0092] In one embodiment, the phoneme text-based virtual face construction device 70 further includes:
[0093] A generating unit, configured to generate other audios from the target text by using other preset TTS methods, wherein the other audios and the audio are of different audio styles;
[0094] The third combining unit is used to combine the other audio with the video frame sequence to obtain other virtual videos.
[0095] In one embodiment, the phoneme text-based virtual face construction device 70 further includes:
[0096] A statistical unit, used for counting the text length corresponding to the target text;
[0097] A judging unit, used to judge whether the text length is greater than or equal to a preset text length threshold;
[0098] An execution unit is used to execute the step of generating audio from the target text through a preset TTS method if the text length is greater than or equal to the preset text length threshold.
[0099] In one embodiment, the phoneme text-based virtual face construction device 70 further includes:
[0100] The issuing unit is used to issue a warning that the target text does not meet the preset requirements if the text length is less than the preset text length threshold.
[0101] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned virtual face construction device based on phoneme text and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.
[0102] At the same time, the division and connection methods of each unit in the above-mentioned virtual face construction device based on phoneme text are only used for illustration. In other embodiments, the virtual face construction device based on phoneme text can be divided into different units as needed, and the units in the virtual face construction device based on phoneme text can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned virtual face construction device based on phoneme text.
[0103] The above-mentioned virtual face construction device based on phoneme text can be implemented in the form of a computer program, which can be used in Figure 8 Runs on the computer device shown.
[0104] See also Figure 8 , Figure 8 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.
[0105] See also Figure 8 The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile computer-readable storage medium.
[0106] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned method for constructing a virtual face based on phoneme text.
[0107] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0108] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned method for constructing a virtual face based on phoneme text.
[0109] The network interface 505 is used to communicate with other devices over the network. Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 8 The embodiments shown are consistent and will not be described again here.
[0110] Among them, the processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps: obtain the target text, and generate audio from the target text through a preset TTS method; segment the audio into phonemes to obtain the phoneme text corresponding to the audio; input the phoneme text into a preset recurrent neural network model to obtain the facial features of a preset real-life host corresponding to the phoneme text; and construct a virtual face of the preset real-life host corresponding to the phoneme text based on the facial features of the preset real-life host.
[0111] In one embodiment, before implementing the step of inputting the phoneme text into a preset recurrent neural network model to obtain the facial features of a preset live anchor corresponding to the phoneme text, the processor 502 further implements the following steps:
[0112] Obtaining a training video sample recorded by a preset real-person anchor, wherein the training video sample includes a training audio sample and an image of the preset real-person anchor;
[0113] The training audio sample is segmented into phonemes to obtain a training phoneme text, and the facial features of the training live host corresponding to the training phoneme text are extracted from the image of the preset live host;
[0114] The training phoneme text and the facial features of the training real-person anchor are input into a preset recurrent neural network model to train the preset recurrent neural network model.
[0115] In one embodiment, when the processor 502 implements the step of segmenting the training audio sample into phonemes to obtain training phoneme text, and extracting the facial features of the training live anchor corresponding to the training phoneme text from the image of the preset live anchor, the processor 502 specifically implements the following steps:
[0116] Obtaining a training audio sample corresponding to the training video sample;
[0117] According to the chronological order, the training audio sample is segmented into phonemes according to preset phonemes to obtain a training phoneme text sequence arranged according to the chronological order, wherein the training phoneme text sequence includes a plurality of training phoneme text elements;
[0118] Obtaining, in the chronological order, a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame includes an image of the preset real-person anchor;
[0119] The appearance features, expression features and posture features of the preset real-person anchor contained in the video frame corresponding to the training phoneme text element are extracted to obtain the facial features of the training real-person anchor corresponding to the training phoneme text.
[0120] In one embodiment, after implementing the step of constructing a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor, the processor 502 further implements the following steps:
[0121] Obtaining all the phoneme texts contained in the audio and the virtual faces corresponding to the phoneme texts;
[0122] Combining all the virtual faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
[0123] The audio is combined with the sequence of video frames to obtain a virtual video.
[0124] In one embodiment, after implementing the step of constructing a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor, the processor 502 further implements the following steps:
[0125] Generate other audios from the target text through other preset TTS modes, wherein the other audios and the audio are of different audio styles;
[0126] The other audio is combined with the sequence of video frames to obtain other virtual video.
[0127] In one embodiment, before implementing the step of generating audio from the target text in a preset TTS manner, the processor 502 further implements the following steps:
[0128] Counting the text length corresponding to the target text;
[0129] Determining whether the text length is greater than or equal to a preset text length threshold;
[0130] If the text length is greater than or equal to the preset text length threshold, the step of generating audio from the target text through a preset TTS method is performed.
[0131] In one embodiment, after implementing the step of generating audio from the target text in a preset TTS manner if the text length is greater than or equal to the preset text length threshold, the processor 502 further implements the following steps:
[0132] If the text length is less than the preset text length threshold, an alarm is issued that the target text does not meet the preset requirements.
[0133] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0134] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.
[0135] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the steps of the method for constructing a virtual face based on phoneme text described in the above embodiments.
[0136] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.
[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0138] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.
[0139] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0141] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.
[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for constructing a virtual face based on phoneme text, characterized in that: include: Obtaining a target text, and generating an audio from the target text through a preset TTS method; Segmenting the audio into phonemes to obtain phoneme text corresponding to the audio; Inputting the phoneme text into a preset recurrent neural network model to obtain facial features of a preset live host corresponding to the phoneme text; According to the facial features of the preset real-person anchor, construct a virtual face of the preset real-person anchor corresponding to the phoneme text; After the step of constructing a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor, the method further includes: Obtaining all the phoneme texts contained in the audio and the virtual faces corresponding to the phoneme texts; Combining all the virtual faces into a video frame sequence according to the time sequence of the phoneme texts in the audio; combining the audio with the sequence of video frames to obtain a virtual video; The phoneme is the smallest unit of speech divided according to the natural properties of speech, and one pronunciation action constitutes one phoneme; Before the step of inputting the phoneme text into a preset recurrent neural network model to obtain the facial features of a preset live anchor corresponding to the phoneme text, the method further includes: Obtaining a training video sample recorded by a preset real-person anchor, wherein the training video sample includes a training audio sample and an image of the preset real-person anchor; The training audio sample is segmented into phonemes to obtain a training phoneme text, and the facial features of the training live host corresponding to the training phoneme text are extracted from the image of the preset live host; Inputting the training phoneme text and the facial features of the training live anchor into a preset recurrent neural network model to train the preset recurrent neural network model; The duration of the training video sample is 2 to 3 hours; The step of performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting facial features of the training live host corresponding to the training phoneme text from the image of the preset live host comprises: Obtaining a training audio sample corresponding to the training video sample; According to the chronological order, the training audio sample is segmented into phonemes according to preset phonemes to obtain a training phoneme text sequence arranged according to the chronological order, wherein the training phoneme text sequence includes a plurality of training phoneme text elements; Obtaining, in the chronological order, a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame includes an image of the preset real-person anchor; Extracting the appearance features, expression features and posture features of the preset real-person anchor contained in the video frame corresponding to the training phoneme text element to obtain the facial features of the training real-person anchor corresponding to the training phoneme text; Among them, the appearance feature refers to the description of the appearance of the face by extracting the key points of the face appearance, the expression feature refers to the organ features, texture areas and predefined features of the face, and the posture feature refers to the posture characteristics of the face.
2. The method for constructing a virtual face based on phoneme text according to claim 1, characterized in that: After the step of constructing a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor, the method further includes: Generate other audios from the target text through other preset TTS modes, wherein the other audios and the audio are of different audio styles; The other audio is combined with the sequence of video frames to obtain other virtual video.
3. The method for constructing a virtual face based on phoneme text according to claim 1, characterized in that: Before the step of generating audio from the target text in a preset TTS mode, the method further includes: Counting the text length corresponding to the target text; Determining whether the text length is greater than or equal to a preset text length threshold; If the text length is greater than or equal to the preset text length threshold, the step of generating audio from the target text through a preset TTS method is performed.
4. The method for constructing a virtual face based on phoneme text according to claim 3, characterized in that: If the text length is greater than or equal to the preset text length threshold, after executing the step of generating audio from the target text in a preset TTS mode, the method further includes: If the text length is less than the preset text length threshold, an alarm is issued that the target text does not meet the preset requirements.
5. A virtual face construction device based on phoneme text, characterized in that: include: A first acquisition unit is used to acquire a target text and generate an audio from the target text in a preset TTS mode; A first segmentation unit is used to segment the audio into phonemes to obtain a phoneme text corresponding to the audio; A first input unit, used to input the phoneme text into a preset recurrent neural network model to obtain facial features of a preset live anchor corresponding to the phoneme text; A construction unit, configured to construct a virtual face of the preset real-person anchor corresponding to the phoneme text according to the facial features of the preset real-person anchor; The virtual face construction device based on phoneme text also includes: Obtaining all the phoneme texts contained in the audio and the virtual faces corresponding to the phoneme texts; Combining all the virtual faces into a video frame sequence according to the time sequence of the phoneme texts in the audio; combining the audio with the sequence of video frames to obtain a virtual video; The phoneme is the smallest unit of speech divided according to the natural properties of speech, and one pronunciation action constitutes one phoneme; The virtual face construction device based on phoneme text also includes: A second acquisition unit is used to acquire a training video sample recorded by a preset real-person anchor, wherein the training video sample includes a training audio sample and an image of the preset real-person anchor; A second segmentation unit is used to perform phoneme segmentation on the training audio sample to obtain a training phoneme text, and extract facial features of the training live host corresponding to the training phoneme text from the image of the preset live host; A training unit, used for inputting the training phoneme text and the facial features of the training live anchor into a preset recurrent neural network model to train the preset recurrent neural network model; The duration of the training video sample is 2 to 3 hours; The second segmentation unit comprises: A first acquisition subunit, used to acquire a training audio sample corresponding to the training video sample; The first segmentation unit is used to segment the training audio sample into phonemes according to preset phonemes according to the time sequence to obtain a training phoneme text sequence arranged according to the time sequence, wherein the training phoneme text sequence includes a plurality of training phoneme text elements; A second acquisition subunit is used to acquire, in accordance with the chronological order, a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame includes the image of the preset real-person anchor; An extraction subunit, used to extract the appearance features, expression features and posture features of the preset real anchor contained in the video frame corresponding to the training phoneme text element, so as to obtain the facial features of the training real anchor corresponding to the training phoneme text; Among them, the appearance feature refers to the description of the appearance of the face by extracting the key points of the face appearance, the expression feature refers to the organ features, texture areas and predefined features of the face, and the posture feature refers to the posture characteristics of the face.
6. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 can be implemented.
