Virtual face generation method, device, computer equipment and readable storage medium

通过预设流式语音识别模型和自注意机制模型,生成流式人脸特征,解决了传统技术中语音驱动虚拟人脸生成不支持流式语音的问题,实现了在低时延场景下的流 Real-time virtual face generation and high-quality face effects.

CN115482832BActive Publication Date: 2024-12-03SHENZHEN YUANXIANG INFORMATION TECH CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210945884.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-08
Publication Date
2024-12-03
Estimated Expiration
2042-08-08

AI Technical Summary

Technical Problem

The virtual face generation algorithm driven by traditional voice cannot support streaming voice, resulting in the real-time generation of streaming virtual faces in scenarios with low latency requirements, such as real-time calls of the metaverse virtual humans and online games.

Method used

By preset streaming speech recognition model and self-attention mechanism model, streaming speech features are obtained and streaming face features are generated based on the correspondence between preset semantic features and face features, thereby realizing speech-driven streaming virtual faces generate.

Benefits of technology

Real-time generation of streaming virtual faces in low-latency scenarios is realized, fully meeting the virtual face generation needs of voice-driven, and improving the face generation effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115482832B_ABST
    Figure CN115482832B_ABST
Patent Text Reader

Abstract

The present application belongs to the field of multimedia technology, and provides a method, apparatus, computer device and computer-readable storage medium for generating a virtual face. In order to solve the problem that virtual face generation based on voice drive does not support streaming voice, a preset correspondence between preset semantic features of voice and preset facial features is pre-set, and source audio is obtained, and the source audio is streamed to obtain streaming voice features, and then based on a self-attention mechanism, streaming semantic features corresponding to the streaming voice features are obtained, and according to the preset correspondence, streaming facial features corresponding to the streaming semantic features are obtained, and then the streaming facial features are used to generate a virtual face, thereby supporting the streaming processing of voice to drive the generation of virtual faces, and realizing the real-time generation of voice-driven streaming virtual faces, which can not only fully meet the real-time virtual face generation with low latency under voice drive, but also has a better face generation effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to a method, device, computer equipment and computer-readable storage medium for generating a virtual face. Background Art

[0002] The generation of virtual faces often requires motion capture equipment to capture the facial contour before rendering. In virtual face scenarios such as real-time conversations between virtual humans in the metaverse, not only is the motion capture equipment expensive, but the direct transmission of the collected facial video information will bring heavy network transmission pressure.

[0003] The speech-driven virtual face generation algorithm only requires the user to input voice information. Compared with collecting and transmitting face video information, the collection and transmission of audio is not only convenient but also can effectively alleviate the pressure of network transmission, which can greatly reduce the cost of generating virtual faces and bring high-quality face and lip generation results. However, in traditional technologies, speech-driven virtual face generation algorithms usually rely on the whole sentence of speech to generate virtual faces, that is, the virtual face can only be generated after processing the complete sentence audio and using the timing information and contextual semantics contained in the speech. In the process of calculating the speech features of streaming speech, the shorter speech segments lack timing information and necessary semantic information, which can easily cause facial jitter of the virtual image in the speech-driven process, thus failing to support streaming speech-driven virtual face generation, that is, failing to meet the streaming virtual face generation requirements in low-latency scenarios, such as real-time calls of virtual people in the metaverse or real-time generation of virtual people in online games. Summary of the invention

[0004] The present application provides a virtual face generation method, apparatus, computer device and computer-readable storage medium, which can solve the technical problem that traditional voice-driven virtual face generation does not support streaming voice.

[0005] In a first aspect, the present application provides a method for generating a virtual face, comprising: obtaining source audio, and based on a hidden layer of a preset streaming speech recognition model, streaming processing the source audio to obtain streaming speech features, wherein the streaming speech features are the output of the hidden layer; based on a preset self-attention mechanism model, encoding and decoding the streaming speech features to obtain streaming semantic features; according to a preset correspondence between preset semantic features and preset facial features, if the streaming semantic features are consistent with the preset semantic features, using the preset facial features as the streaming facial features corresponding to the streaming semantic features, and generating a virtual face using the streaming facial features.

[0006] In the second aspect, the present application provides a virtual face generation device, comprising: a speech processing unit, used to obtain source audio, and based on the hidden layer of a preset streaming speech recognition model, stream-process the source audio to obtain streaming speech features, wherein the streaming speech features are the output of the hidden layer; a coding and decoding unit, used to encode and decode the streaming speech features based on a preset self-attention mechanism model to obtain streaming semantic features; a face generation unit, used to generate a virtual face based on a preset correspondence between a preset semantic feature and a preset face feature, if the streaming semantic feature is consistent with the preset semantic feature, use the preset face feature as the streaming face feature corresponding to the streaming semantic feature, and use the streaming face feature to generate a virtual face.

[0007] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the virtual face generation method when executing the computer program.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the virtual face generation method.

[0009] The present application provides a method, apparatus, computer device and computer-readable storage medium for generating a virtual face. The processing method pre-sets a preset correspondence between preset semantic features of speech and preset facial features, obtains source audio, and streams the source audio to obtain streaming speech features. Then, based on a self-attention mechanism, the streaming semantic features corresponding to the streaming speech features are obtained, and according to the preset correspondence, the streaming facial features corresponding to the streaming semantic features are obtained, and the streaming facial features are used to generate a virtual face, thereby supporting the streaming processing of speech to drive the generation of virtual faces, and realizing the real-time generation of streaming virtual faces driven by speech. This method can not only fully meet the real-time virtual face generation with low latency under speech drive, but also has a good face generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a method for generating a virtual face provided in an embodiment of the present application;

[0012] Figure 2A schematic block diagram of obtaining streaming voice features in a virtual face generation method provided in an embodiment of the present application;

[0013] Figure 3 A schematic diagram of speech-driven streaming facial feature generation in a virtual face generation method provided in an embodiment of the present application;

[0014] Figure 4 A schematic flowchart of obtaining facial features in a virtual face generation method provided in an embodiment of the present application;

[0015] Figure 5 Another schematic flowchart of obtaining facial features in the virtual face generation method provided in an embodiment of the present application;

[0016] Figure 6 A schematic diagram of the process framework structure of the self-attention mechanism decoder in the virtual face generation method provided in an embodiment of the present application;

[0017] Figure 7 A schematic block diagram of a virtual face generation device provided in an embodiment of the present application;

[0018] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0021] The embodiment of the present application provides a method for generating a virtual human face. The processing method can be applied to electronic devices such as smart phones, tablet computers, laptops or desktop computers, AR / VR, and can be applied to multimedia application scenarios that require the generation of virtual human images, such as real-time calls, online games, and online live broadcasts in the metaverse.

[0022] In the face of the technical problem that traditional technology of voice-driven virtual face generation does not support real-time generation of virtual faces with streaming voice, the inventor proposes a virtual face generation method of the embodiment of the present application. The core idea of ​​the embodiment of the present application is: for voice-driven virtual face generation, a preset correspondence between preset semantic features of voice and preset facial features is pre-set, and a preset streaming voice recognition model is used to perform streaming processing on the input voice to obtain the streaming voice features output by the hidden layer of the preset streaming voice recognition model, and based on the self-attention mechanism, the streaming semantic features corresponding to the streaming voice features are obtained, and according to the preset correspondence, the streaming facial features corresponding to the streaming semantic features are obtained, and then the streaming facial features are generated into a virtual face, thereby realizing voice-driven streaming virtual face generation, which can not only fully meet the low-latency streaming virtual face generation driven by voice, but also has a better face generation effect.

[0023] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0024] See also Figures 1 to 3 , Figure 1 A flowchart of a method for generating a virtual face provided in an embodiment of the present application is provided. Figure 2 A schematic block diagram of obtaining streaming voice features in a virtual face generation method provided in an embodiment of the present application, Figure 3 Schematic diagram of speech-driven streaming facial feature generation in the virtual face generation method provided in the embodiment of the present application. Figure 1 As shown, the method includes the following steps S11-S13:

[0025] S11. Acquire source audio, and perform streaming processing on the source audio based on a hidden layer of a preset streaming speech recognition model to obtain streaming speech features, wherein the streaming speech features are outputs of the hidden layer.

[0026] Specifically, a streaming speech recognition model is pre-set, that is, a preset streaming speech recognition model. The preset streaming speech recognition model includes a hidden layer, which is an encoder module. The preset streaming speech recognition model supports streaming processing of audio to obtain speech frames corresponding to segmented speech, which can be referred to as speech frames or frames for short, thereby supporting streaming processing of speech and obtaining streaming speech features, wherein a speech frame describes each segment of speech in speech, and a frame is a Frame in English. A speech frame is a waveform segment in which speech is cut into preset time segments according to an audio waveform. The preset time segment can be a time length of 10ms or 20ms. In one example, the hidden layer of the preset streaming speech recognition model can be a block-based mask streaming processing structure, that is, a chunk-mask-based streaming processing structure. The hidden layer of the preset streaming speech recognition model can be a component of other neural network models, that is, other neural network models have added a block-based mask streaming processing structure. For example, the hidden layer of the preset streaming speech recognition model is a component of a chunk-mask-based Transformer model, or the hidden layer of the preset streaming speech recognition model is a component of a chunk-mask-based Conformer model.

[0027] Get the source audio, and input the source audio to the hidden layer of the preset streaming speech recognition model for streaming processing, so as to divide the source audio into frames, obtain the speech of each frame, i.e., speech frame, and obtain the streaming speech features corresponding to each frame of speech based on the speech frame. The streaming speech features are described by streaming speech feature vectors, where the streaming speech features are the output of the hidden layer of the preset streaming speech recognition model, and the hidden layer is the layers other than the input layer and the output layer in the preset streaming speech recognition model. Please refer to Figure 2 After the source audio is processed by the encoder module of the preset streaming speech recognition model, the phoneme sequence is not obtained through the softmax output, but the output of the encoder module, that is, the streaming speech feature vector, is directly obtained, and the output of the hidden layer of the preset streaming speech recognition model is used as the streaming speech feature.

[0028] S12. Based on a preset self-attention mechanism model, the streaming speech features are encoded and decoded to obtain streaming semantic features.

[0029] Specifically, a neural network model based on the self-attention mechanism is pre-set, that is, a preset self-attention mechanism model, and the preset self-attention mechanism model can be a Transformer model or a Conformer model. In one example, the preset self-attention mechanism model can be a chunk-mask-based Transformer model or a chunk-mask-based Conformer model, that is, a self-attention mechanism model such as a Transformer model or a Conformer model with a block structure that supports streaming speech processing. The self-attention mechanism, also known as the self-attention mechanism, reduces the dependence on external information and is better at capturing the internal correlation of data or features.

[0030] Based on the preset self-attention mechanism model, the streaming speech features are encoded based on the self-attention mechanism, and the required content features are extracted from the streaming speech features, so as to better utilize the information contained in the streaming speech features, and obtain the encoded semantic features corresponding to the encoding, and then the encoded semantic features are decoded to obtain streaming semantic features. The streaming semantic features are described by streaming semantic feature vectors, and the streaming semantic features describe the semantic content corresponding to the speech frames extracted from the source audio.

[0031] S13. According to a preset correspondence between a preset semantic feature and a preset facial feature, if the streaming semantic feature is consistent with the preset semantic feature, use the preset facial feature as the streaming facial feature corresponding to the streaming semantic feature, and generate a virtual face using the streaming facial feature.

[0032] Specifically, a preset correspondence between preset semantic features of speech and preset facial features is preset, and the preset correspondence describes what kind of streaming semantic features correspond to what kind of facial features. The preset correspondence can be derived from the correspondence between semantic features and facial features of real people used as model training samples, that is, the correspondence between semantic features and facial features of real people in actual business scenarios. The preset correspondence can be obtained by marking the correspondence between the semantics and facial features of real people in actual business scenarios. For example, the preset correspondence can be obtained by matching the facial features corresponding to semantic contents such as "today", "sunny" or "weather" when the real person says such contents.

[0033] According to the preset correspondence between the preset semantic features and the preset facial features, the streaming semantic features are compared with the preset semantic features. If it is determined that the streaming semantic features are consistent with the preset semantic features, the preset facial features matched by the preset semantic features are used as the facial features corresponding to the streaming semantic features, and the streaming facial features corresponding to the streaming semantic features are obtained. The facial features include features of key points of the human face such as mouth shape features, cheek features, eye features, and jaw features, especially the mouth shape features in the facial features. The mouth shape features describe the shape of the lips when the lips pronounce. The facial features are generally described as a sequence of facial feature parameters. The facial features are generally multi-dimensional vectors to describe the features and shapes of various parts of the human face such as the mouth shape, eyes, and cheeks, and the multi-dimensional vectors describing the facial features are rendered, thereby converting the streaming facial features into a virtual facial image to generate the corresponding virtual face.

[0034] like Figure 3 As shown, in this example, for the input source audio, 80ms of audio is used as a speech block, that is, a speech frame, as the processing unit corresponding to the streaming speech. Based on this, the source audio is divided into an input audio sequence, and for each speech block, a corresponding streaming facial feature is generated, and the corresponding virtual face is output. Among them, the 80ms audio is used as the time unit of the speech block. It comes from cutting the sound into frames first, and the length of each frame is based on 40ms. The time unit of the speech block can be a multiple of 40ms. Taking 80ms as the time unit takes into account the balance between the effect of facial feature generation and the data volume processing of streaming speech, which not only realizes the streaming face generation driven by speech, but also can achieve a better virtual face generation effect. At the same time, Figure 3 In the example shown, the streaming semantic features of the historical frames are also used, and the causal convolution structure is adopted to generate streaming facial features. The embodiment of the present application can achieve an RFT of 0.3, where RFT, in English, stands for Real_time_factor, real-time rate, and online recognition index. Each time 80ms of audio information is input, the total delay for virtual face generation is 80ms+80ms*0.3=104ms, which supports voice-driven streaming face generation and shortens the real-time generation time of virtual faces. Compared with the traditional technology where the delay of virtual faces exceeds 1 second, the generation time of virtual faces is greatly shortened.

[0035] The embodiment of the present application pre-sets a preset correspondence between preset semantic features of speech and preset facial features, obtains source audio, and streams the source audio to obtain streaming speech features, and then based on the self-attention mechanism, obtains streaming semantic features corresponding to the streaming speech features, and according to the preset correspondence, obtains streaming facial features corresponding to the streaming semantic features, and generates a virtual face from the streaming facial features, thereby not only supporting streaming processing of speech, but also realizing real-time generation of speech-driven streaming virtual faces, which can fully meet the real-time generation of streaming virtual faces with low latency under speech, and has better face generation effect.

[0036] In one embodiment, see Figure 4 , Figure 4 This is a schematic flow chart of obtaining facial features in the virtual face generation method provided in the embodiment of the present application. Figure 4 As shown, in this embodiment, the streaming speech features are encoded and decoded to obtain streaming semantic features, including:

[0037] Encoding the streaming speech features to obtain initial encoded semantic features;

[0038] Acquire a prosodic feature, and use the prosodic feature to correct the initial coding semantic feature to obtain a coding semantic feature;

[0039] The encoded semantic features are decoded to obtain streaming semantic features.

[0040] Specifically, a prosody predictor is pre-set, i.e., a preset prosody predictor. The preset prosody predictor can be a neural network such as DNN or Transformer. The preset prosody predictor is used to describe the prosody features corresponding to the streaming speech features. The prosody features are also called super-sound quality features or super-segmental features. The prosody features mainly include the content of dimensions such as fundamental frequency and energy. The preset prosody predictor can be trained in a supervised manner, i.e., the first speech training sample is labeled and then the preset prosody predictor is trained. The prosody predictor can be expressed as follows:

[0041] loss = loss(face) + λloss(baseband energy); Formula (1)

[0042] Among them, loss(face) describes the loss function of face features, loss(fundamental frequency energy) describes the loss function of fundamental frequency and energy in rhythmic features, and λ is the weight, which can be 0.05. Figure 4 , Figure 4The data in the middle are marked with a dotted rectangle, and the neural network structure is marked with a solid rounded rectangle. When training the rhythm predictor, the rhythm features in the semantic features are used to train the rhythm predictor. The rhythm features are trained as fundamental frequency features and energy features. The subsequent prediction loss of the rhythm predictor is back-transmitted to the self-attention mechanism content encoder to use the rhythm features to correct the initial encoded semantic features to obtain the encoded semantic features.

[0043] Based on the preset speech recognition model, the source audio is segmented to obtain the speech frames corresponding to each speech segment. The speech frames are processed by the hidden layer of the speech recognition model to obtain streaming speech features, and then based on the encoder module of the preset self-attention mechanism model, that is, Figure 4 The self-attention mechanism content encoder in the embodiment encodes the streaming speech features, extracts the required content features from them, obtains the initial encoded semantic features, obtains the prosodic features, and uses the prosodic features to correct the initial encoded semantic features to obtain the encoded semantic features, such as obtaining the fundamental frequency features and energy features of the speech, and then uses the prosodic features to correct the initial encoded semantic features, thereby combining the prosodic features with the initial encoded semantic features to correct the initial encoded semantic features and obtain the encoded semantic features, so that the encoded semantic features have higher accuracy than the real speech frames, and then based on the decoder module of the preset self-attention mechanism model, that is, Figure 4 The self-attention mechanism content decoder in decodes the encoded semantic features to obtain streaming semantic features.

[0044] Furthermore, the rhythmic features include at least one of fundamental frequency features and energy features. Among them, fundamental frequency, also known as basic frequency, is called Fundamental frequency in English. Fundamental frequency describes the height of sound, that is, pitch. The higher the basic frequency of sound, the higher the pitch, and the lower the basic frequency of sound, the lower the pitch. Energy, also known as intensity, or volume, describes the size of sound. It can be compared to the amplitude of the sound signal. The larger the amplitude, the larger the volume of the sound waveform.

[0045] The embodiment of the present application obtains the prosodic features corresponding to the speech frames, especially obtains at least one of the fundamental frequency features and energy features included in the prosodic features, and uses the prosodic features of the streaming speech to constrain the initial coding semantic features, thereby correcting the initial coding semantic features and obtaining the coding semantic features, and realizes the use of the prosodic features of the streaming speech to assist in predicting the facial features, especially the lip shape features in the facial features, so that the facial features, especially the lip shape features in the facial features, have higher accuracy, improve the accuracy of the lip shape prediction, and achieve high-accuracy lip shape prediction, thereby further improving the generation effect of the virtual face.

[0046] In one embodiment, see Figure 4 and Figure 5 , Figure 5 Another schematic flowchart of obtaining facial features in the virtual face generation method provided in an embodiment of the present application. In this embodiment, decoding the encoded semantic features to obtain streaming semantic features includes:

[0047] Acquire emotion label features according to the speech frames corresponding to the streaming speech features;

[0048] The emotion tag feature is combined with the encoded semantic feature and decoded to obtain a streaming semantic feature.

[0049] Specifically, an emotion label detection model is pre-set, that is, a preset emotion label detection model. The preset emotion label detection model can be a model based on a deep neural network. The preset emotion label detection model is used to detect the emotion label corresponding to the speech frame, and the emotion label feature is obtained according to the detected emotion label. The emotion label feature is described by an emotion label feature vector, wherein the emotion label describes the emotion contained in the speech frame. The emotion label includes features such as pleasure, smile, pain, frustration, etc. that describe the speaker's emotion. The emotion label feature vector describes the vector corresponding to the emotion label feature of the speech frame. The preset emotion label detection model can be trained in a supervised manner, that is, the second speech training sample is labeled and then the preset emotion label detection model is trained.

[0050] like Figure 5 As shown, Figure 5 The data in the figure are marked with a dotted rectangle, and the neural network structure is marked with a solid rounded rectangle. According to the speech frame corresponding to the streaming speech feature, the emotion label of the speech is extracted from the speech frame, such as the emotion label of the joy, frustration or excitement of the speech, and the emotion label feature corresponding to the speech frame is obtained according to the emotion label. The emotion label feature describes the characteristics of the emotion label of the speech frame, and then based on the face parameter generation module, the emotion label feature is combined and decoded with the encoded semantic feature to obtain the streaming semantic feature, and the streaming face feature corresponding to the streaming semantic feature is obtained according to the streaming speech feature. The streaming face feature is described by the facial feature parameter sequence, and the streaming face feature is used to generate a virtual face. For example, if the emotion label feature is 40-dimensional and the encoded semantic feature is 100-dimensional, the vector obtained by combining the emotion label feature with the encoded semantic feature is 140-dimensional, and the 140-dimensional vector is decoded to obtain the streaming semantic feature.

[0051] Furthermore, the emotion tag feature is an emotion tag described based on one-hot encoding.

[0052] Among them, One-Hot encoding, that is, One-Hot encoding, is the representation of categorical variables as binary vectors, also known as one-bit effective encoding, which uses an N-bit state register to encode N states. Each state has its own independent register bit, and at any time, only one bit is valid.

[0053] Specifically, for the emotion label features describing the emotion labels, one-hot encoding is used to provide the expression control information in the facial features, so as to obtain the emotion label features, which is based on one-hot encoding to obtain the emotion label features. Since different emotion labels are categorical variables and non-continuous numerical features, one-hot encoding is used to describe the emotion label features, which can better describe the emotion label features and realize the representation and operation of emotion labels by computer equipment.

[0054] In the embodiment of the present application, the emotion label feature corresponding to the emotion label of the speech frame is applied in the self-attention mechanism decoder stage, that is, the face parameter generation module. The emotion label can be an expression feature that describes the speaker's emotion, such as "smiling face" or "crying face", and the expression control information of the facial features is provided to obtain streaming semantic features. Since the expression corresponding to the emotion is closely related to the facial features, the facial features, especially the lip shape features in the facial features, can have a higher accuracy, thereby further improving the generation effect of the virtual face.

[0055] In one embodiment, see Figure 6 and Figure 3 , Figure 6 This is a schematic diagram of the process framework structure of the self-attention mechanism decoder in the virtual face generation method provided in the embodiment of the present application. Figure 6 As shown, in this embodiment, decoding the encoded semantic features to obtain streaming semantic features includes:

[0056] Based on other model layers included in a preset self-attention mechanism decoder, decoding the encoded semantic features to obtain initial decoded semantic features, wherein the preset self-attention mechanism decoder is a decoder included in the preset self-attention mechanism model;

[0057] Performing causal convolution processing on the encoded semantic features to obtain a causal convolution output result;

[0058] The causal convolution output result is used to adjust the initial decoded semantic features to obtain streaming semantic features.

[0059] Specifically, in the self-attention mechanism decoder included in the preset self-attention mechanism model, a causal convolution structure is constructed, that is, Figure 6The causal convolution layer shown in the figure, the self-attention mechanism decoder includes the causal convolution layer and other model layers except the causal convolution layer, so that the self-attention mechanism decoder becomes a residual neural network, where the causal convolution is called CausalConvolutions in English. The causal convolution description ensures that the output at time t is only convolved with the elements of the previous layer at time t and earlier.

[0060] Based on other model layers in the self-attention mechanism decoder included in the preset self-attention mechanism model, the encoded semantic features are decoded to obtain the initial decoded semantic features corresponding to the decoding, and the encoded semantic features are input into the causal convolution layer for causal convolution processing, so as to obtain the causal convolution output result with the help of the streaming semantic features of the historical frame. The causal convolution output result is a relatively small value, which can be regarded as the adjustment amount of the initial decoded semantic features, and the causal convolution output result is used to adjust the initial decoded semantic features. The causal convolution output result can be added to the initial decoded semantic features, so that the causal convolution result is added to the initial decoded semantic features in an additive form to obtain the streaming semantic features. Figure 3 As shown in the figure, when generating the facial features of the current frame, the streaming semantic features based on the previous 4 frames are used to process the facial features of the current frame. Since future frame information cannot be seen, the system is causal, which reduces the dependence on future frame information, so that its latency can meet the needs of actual business scenarios.

[0061] In the embodiment of the present application, a causal convolution structure is constructed in the self-attention mechanism decoder included in the preset self-attention mechanism model to realize residual learning using a causal convolution layer, which can smooth the jitter problem of facial parameters of facial features. The causal convolution output result is a supplement to the initial decoded semantic features, and the initial decoded semantic features are smoothed and modified, which can further improve the generation effect of virtual faces.

[0062] It should be noted that the virtual face generation method described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but all of them are within the scope of protection required by this application.

[0063] See also Figure 7 , Figure 7 A schematic block diagram of a virtual face generation device provided in an embodiment of the present application. Corresponding to the above-mentioned virtual face generation method, an embodiment of the present application also provides a virtual face generation device. Figure 7 As shown, the virtual face generation device includes a unit for executing the above-mentioned virtual face generation method, and the virtual face generation device can be configured in a computer device. Figure 7 The virtual face generating device 70 includes a speech processing unit 71, a coding and decoding unit 72 and a face generating unit 73.

[0064] The speech processing unit 71 is used to obtain source audio, and based on the hidden layer of the preset streaming speech recognition model, stream process the source audio to obtain streaming speech features, wherein the streaming speech features are the output of the hidden layer;

[0065] The encoding and decoding unit 72 is used to encode and decode the streaming speech features based on a preset self-attention mechanism model to obtain streaming semantic features;

[0066] The face generation unit 73 is used to generate a virtual face from the streaming face feature according to the preset correspondence between the preset semantic feature and the preset face feature. If the streaming semantic feature is consistent with the preset semantic feature, the preset face feature is used as the streaming face feature corresponding to the streaming semantic feature.

[0067] In one embodiment, the encoding and decoding unit 72 includes:

[0068] An encoding subunit, used for encoding the streaming speech feature to obtain an initial encoded semantic feature;

[0069] A first acquisition subunit is used to acquire a prosodic feature, and use the prosodic feature to correct the initial coding semantic feature to obtain a coding semantic feature;

[0070] The first decoding subunit is used to decode the encoded semantic features to obtain streaming semantic features.

[0071] In one embodiment, the rhythmic feature includes at least one of a fundamental frequency feature and an energy feature.

[0072] In one embodiment, the first decoding subunit includes:

[0073] A second acquisition subunit is used to acquire an emotion label feature according to the speech frame corresponding to the streaming speech feature;

[0074] The second decoding subunit is used to combine and decode the emotion tag feature with the encoded semantic feature to obtain a streaming semantic feature.

[0075] In one embodiment, the emotion tag feature is an emotion tag described based on one-hot encoding.

[0076] In one embodiment, the first decoding subunit includes:

[0077] A third decoding subunit is used to decode the encoded semantic features based on other model layers included in a preset self-attention mechanism decoder to obtain initial decoded semantic features, wherein the preset self-attention mechanism decoder is a decoder included in the preset self-attention mechanism model;

[0078] A causal convolution subunit, used for performing causal convolution processing on the encoded semantic features to obtain a causal convolution output result;

[0079] The adjustment subunit is used to adjust the initial decoded semantic features using the causal convolution output result to obtain a streaming semantic feature.

[0080] In one embodiment, the preset self-attention mechanism model is a Chunk-mask based Transformer model or a Chunk-mask based Conformer model.

[0081] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned virtual face generation device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0082] At the same time, the division and connection method of each unit in the above-mentioned virtual face generating device is only for illustrative purposes. In other embodiments, the virtual face generating device can be divided into different units as needed, and the units in the virtual face generating device can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned virtual face generating device.

[0083] The above-mentioned virtual face generation device can be implemented in the form of a computer program. The computer program can be used in Figure 8 Runs on the computer device shown.

[0084] See also Figure 8 , Figure 8 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0085] See also Figure 8 The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile storage medium.

[0086] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned virtual face generation method.

[0087] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0088] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned virtual face generation method.

[0089] The network interface 505 is used to communicate with other devices over the network. Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 8 The embodiments shown are consistent and will not be described again here.

[0090] The processor 502 is used to run the computer program 5032 stored in the memory to implement the virtual face generation method described above.

[0091] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0092] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0093] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

[0094] A computer program product, when running on a computer, enables the computer to execute the steps of the virtual face generation method described in the above embodiments.

[0095] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0098] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0100] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a virtual face, characterized in that: include: Acquire source audio, and perform streaming processing on the source audio based on a hidden layer of a preset streaming speech recognition model to obtain a streaming speech feature, wherein the streaming speech feature is an output of the hidden layer; Based on a preset self-attention mechanism model, the streaming speech features are encoded and decoded to obtain streaming semantic features; According to the preset correspondence between the preset semantic feature and the preset facial feature, if the streaming semantic feature is consistent with the preset semantic feature, the preset facial feature is used as the streaming facial feature corresponding to the streaming semantic feature, and the streaming facial feature is used to generate a virtual face.

2. The method for generating a virtual face according to claim 1, characterized in that: The encoding and decoding of the streaming speech features to obtain streaming semantic features includes: Encoding the streaming speech features to obtain initial encoded semantic features; Acquire a prosodic feature, and use the prosodic feature to correct the initial coding semantic feature to obtain a coding semantic feature; The encoded semantic features are decoded to obtain streaming semantic features.

3. The method for generating a virtual face according to claim 2, characterized in that: The rhythmic feature includes at least one of a fundamental frequency feature and an energy feature.

4. The method for generating a virtual face according to claim 2 or 3, characterized in that: The decoding of the encoded semantic features to obtain streaming semantic features includes: Acquire emotion label features according to the speech frames corresponding to the streaming speech features; The emotion tag feature is combined with the encoded semantic feature and decoded to obtain a streaming semantic feature.

5. The method for generating a virtual face according to claim 4, characterized in that: The emotion tag feature is an emotion tag described based on one-hot encoding.

6. The method for generating a virtual face according to claim 2 or 3, characterized in that: The decoding of the encoded semantic features to obtain streaming semantic features includes: Based on other model layers included in a preset self-attention mechanism decoder, decoding the encoded semantic features to obtain initial decoded semantic features, wherein the preset self-attention mechanism decoder is a decoder included in the preset self-attention mechanism model; Performing causal convolution processing on the encoded semantic features to obtain a causal convolution output result; The causal convolution output result is used to adjust the initial decoded semantic features to obtain streaming semantic features.

7. The method for generating a virtual face according to claim 1, characterized in that: The preset self-attention mechanism model is a Chunk-mask-based Transformer model or a Chunk-mask-based Conformer model.

8. A virtual face generation device, characterized in that: include: A speech processing unit, used for acquiring source audio, and performing streaming processing on the source audio based on a hidden layer of a preset streaming speech recognition model to obtain a streaming speech feature, wherein the streaming speech feature is an output of the hidden layer; A coding and decoding unit, used for encoding and decoding the streaming speech features based on a preset self-attention mechanism model to obtain streaming semantic features; A face generation unit is used to generate a virtual face from the streaming face feature according to a preset correspondence between a preset semantic feature and a preset face feature. If the streaming semantic feature is consistent with the preset semantic feature, the preset face feature is used as the streaming face feature corresponding to the streaming semantic feature, and the streaming face feature is used to generate a virtual face.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Citation Information

Patent Citations

  • Face image restoration method based on semantic analysis generation guidance

    CN113888417A

  • Face image restoration method based on multi-scale local self-attention generative adversarial network

    CN113962893A