Virtual face generation method and device based on artificial intelligence, equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a virtual face generation method and device based on artificial intelligence, equipment and a medium, and the method comprises the steps: extracting text semantic feature information and text emotion feature information of second text data based on a text coding model, extracting face feature information and lip feature information of the second face video data based on a face detection model; obtaining a pre-trained multi-mode voice model, and inputting the extracted semantic feature information and text emotion feature information of the second text data and the extracted face feature information and lip feature information of the second face video data into the pre-trained multi-mode voice model for training; and obtaining third text data of a to-be-generated virtual face, inputting the third text data into the trained multi-mode voice model, predicting voice representation information and facial expression information of the virtual face, and generating virtual face video data based on an animation synthesis algorithm. According to the invention, the sense of reality of the generated virtual face is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method, device, equipment and medium for generating a virtual face based on artificial intelligence. Background Art

[0002] At present, most of the AI-based virtual face generation content on the market is based on video content on video websites, usually based on real-person voice and real-person face. After deep learning model training, it can generate a virtual face image with lip changes for a single static face by using pulse code modulation of voice. However, during the training stage of the deep learning model, the source of voice data is usually downloaded directly from the Internet or obtained through real-person recording, and the massive amount of text data cannot be directly used as a corpus source. It can only be converted into voice data with the help of a speech synthesis model. This method will cause the virtual face to lose the ability to express emotions when outputting voice data, and ultimately make the generated virtual face look less realistic and lack of realism.

[0003] In view of this, how to effectively improve the realism of generated virtual faces is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for generating a virtual face based on artificial intelligence, so as to solve the technical problem that the virtual face generated by the existing virtual face generating method lacks a sense of reality.

[0005] In a first aspect, the present invention provides a method for generating a virtual face based on artificial intelligence, comprising:

[0006] Acquire first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data;

[0007] Extracting text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and extracting face feature information and lip feature information of the second face video data based on the face detection model;

[0008] Obtaining a pre-trained multimodal speech model, and inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training;

[0009] Constructing a target loss function for training a pre-trained multimodal speech model, and updating the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model;

[0010] Acquire third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face;

[0011] According to the predicted speech representation information and facial expression information of the virtual face, virtual face video data is generated based on an animation synthesis algorithm.

[0012] In a second aspect, the present invention provides an artificial intelligence-based virtual face generation device, the device is used to implement the artificial intelligence-based virtual face generation method as described in the first aspect, comprising:

[0013] a processing module, configured to obtain first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data;

[0014] A feature extraction module, used to extract text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and to extract face feature information and lip feature information of the second face video data based on the face detection model;

[0015] A training module, used for obtaining a pre-trained multimodal speech model, and inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training;

[0016] A construction module is used to construct a target loss function for training a pre-trained multimodal speech model, and to update the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model;

[0017] A prediction module, used to obtain third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face;

[0018] The generation module is used to generate virtual face video data based on the predicted speech representation information and facial expression information of the virtual face and based on the animation synthesis algorithm.

[0019] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned artificial intelligence-based virtual face generation method when executing the computer program.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned artificial intelligence-based virtual face generation method are implemented.

[0021] In the scheme implemented by the above-mentioned artificial intelligence-based virtual face generation method, device, equipment and medium, the first face video data and the first text data corresponding to the first face video data can be obtained through the client, and the first face video data and the first text data can be preprocessed to obtain the second face video data and the second text data; the text semantic feature information and the text emotional feature information of the second text data can be extracted based on the text encoding model, and the face feature information and the lip feature information of the second face video data can be extracted based on the face detection model; a pre-trained multimodal speech model is obtained, and the extracted semantic feature information and text emotional feature information of the second text data, as well as the face feature information and the lip feature information of the second face video data are input into the pre-trained multimodal speech model for training; and a target loss function is constructed during the training of the pre-trained multimodal speech model. A loss function is obtained, and the pre-trained multimodal speech model is updated based on the target loss function to obtain a trained multimodal speech model; a third text data of a virtual face to be generated is obtained, and the third text data is input into the trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face; according to the predicted speech representation information and facial expression information of the virtual face, virtual face video data is generated based on an animation synthesis algorithm. In the present invention, text semantic feature information and text emotional feature information of the second text data can be extracted through a text encoding model, and facial feature information and lip feature information of the second face video data can be extracted through a face detection model, and the third text data is input into the trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face, thereby effectively improving the realism of the generated virtual face. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 It is a schematic diagram of an application environment of a method for generating a virtual face based on artificial intelligence in one embodiment of the present invention;

[0024] Figure 2 It is a flow chart of a method for generating a virtual face based on artificial intelligence in one embodiment of the present invention;

[0025] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0026] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S40;

[0027] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S50;

[0028] Figure 6 is a structural schematic diagram of a virtual face generation device based on artificial intelligence in one embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0030] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The method for generating a virtual face based on artificial intelligence provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, Figure 1It is a schematic diagram of an application environment of a method for generating a virtual face based on artificial intelligence in one embodiment of the present invention; wherein, the client communicates with the server via a network. The server can obtain first facial video data and first text data corresponding to the first facial video data through the client, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data; extract text semantic feature information and text emotional feature information of the second text data based on a text encoding model, and extract facial feature information and lip feature information of the second facial video data based on a face detection model; obtain a pre-trained multimodal speech model, and input the extracted semantic feature information and text emotional feature information of the second text data, as well as the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training; construct a target loss function for the pre-trained multimodal speech model during training, and perform pre-processing on the pre-trained multimodal speech model based on the target loss function. The multimodal speech model trained by the training is updated to obtain a trained multimodal speech model; the third text data of the virtual face to be generated is obtained, and the third text data is input into the trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face; according to the predicted speech representation information and facial expression information of the virtual face, the virtual face video data is generated based on the animation synthesis algorithm. In the present invention, the text semantic feature information and text emotional feature information of the second text data can be extracted through the text encoding model, and the facial feature information and lip feature information of the second face video data can be extracted through the face detection model, and the third text data is input into the trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face, which effectively improves the realism of the generated virtual face. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 As shown, Figure 2 A flowchart of a method for generating a virtual face based on artificial intelligence provided by an embodiment of the present invention is provided. The method for generating a virtual face based on artificial intelligence specifically comprises the following steps:

[0034] S10: Acquire first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data.

[0035] When the present invention generates a virtual face, first, it is necessary to collect video information containing face images, for example, downloading videos containing specific characters speaking from an online video platform, self-recorded character speech videos, etc., in which the faces show various expressions, actions and corresponding voice information. Secondly, it is also necessary to collect the specific text content of the character's speech in the video containing the face image, for example, if the character in the video is reading an article, the text content of the article is the corresponding text data, so as to obtain the first face video data and the first text data corresponding to the first face video data. Then, it is necessary to perform data preprocessing on the first face video data and the first text data to obtain the second face video data and the second text data.

[0036] Specifically, the first face video data is preprocessed to obtain the second face video data, including: performing noise reduction processing on the first face video data to obtain the face video data after noise reduction; performing coding format conversion processing on the face video data after noise reduction to obtain the second face video data after coding format conversion. First, during the acquisition process, the first face video data is often affected by various factors and mixed with noise. These noises may appear as some irregular spots and stripes on the screen, or noise in the audio, etc., which will interfere with the subsequent accurate extraction and analysis of face-related features in the video. Therefore, the first face video data can be subjected to noise reduction processing by existing technologies such as adaptive filtering. Secondly, since face video data from different sources may have various coding formats, but the subsequent processing links have unified requirements for the video coding format, it is also necessary to convert the coding format of the first face video data after noise reduction to obtain the second face video data that meets the corresponding coding format specification.

[0037] In one embodiment of the present invention, the first text data is preprocessed to obtain the second text data, including: performing data cleaning on the first text data to obtain text data after data cleaning; performing word segmentation on the text data after data cleaning to obtain the second text data after word segmentation. When obtaining the first text data, there are often various problems that do not meet the subsequent processing specifications, such as including extra spaces, irregular use of punctuation marks, and typos, which will affect the subsequent accurate understanding, analysis and further use of the text data. Therefore, it is necessary to perform data cleaning on the first text data to improve the quality and standardization of the text. For example, by writing a string processing function to remove extra spaces, multiple consecutive spaces are replaced with a single space, so that the format of the first text data is more regular. Secondly, it is also necessary to segment the continuous first text data into an independent word according to semantic and grammatical rules, so as to facilitate subsequent part-of-speech tagging and semantic understanding. When the present invention performs word segmentation on the text data after data cleaning, the text data can be first converted into a text vector representation through word vector technology, and then the word boundary is automatically identified by the neural network model for word segmentation to obtain the second text data after word segmentation.

[0038] S20: Extract text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and extract facial feature information and lip feature information of the second facial video data based on the face detection model. The text semantic feature information of the second text data reflects the actual meaning conveyed by the text and the inherent semantic connection between different words and sentences. Specifically, the present invention can use text encoding models such as Word2Vec and BERT to extract text semantic feature information of the second text data. The emotional feature information of the text reflects the subjective emotional tendency conveyed by the text, such as positive or negative emotions. After the second text data is input, the model can analyze the words, intonation, and overall expression style of the text based on the emotional pattern it has learned, and then determine the emotional feature information of the text.

[0039] In addition, the present invention extracts facial feature information and lip feature information from the second facial video data based on a face detection model. Specifically, after the second facial video data is input into the face detection model, the face detection model will first locate the area where the face is located in each frame of the second facial video data, and then further analyze the facial area, and extract different levels of feature representations through a convolutional neural network, such as the overall morphology of the face, the structural characteristics of the facial features, and other facial feature information. After the face detection model detects the facial area, it will focus on a more detailed analysis of the local area where the lips are located, and extract the shape features, texture features, and dynamic change features of the lips at different times of the lip area through a convolutional neural network. These lip feature information can provide a data basis for the subsequent generation of a virtual face with realistic lip movements.

[0040] S30: Obtain a pre-trained multimodal speech model, and input the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training. Specifically, Figure 3 As stated, Figure 3 yes Figure 2 A specific implementation flow diagram of step S30 in the embodiment of the present invention specifically includes the following steps S31-S32:

[0041] S31: Input the semantic feature information and text emotion feature information of the second text data, as well as the facial feature information and lip feature information of the second facial video data, into the cross-modal fusion layer of the pre-trained multimodal speech model for feature fusion to obtain target feature information after cross-modal fusion. Specifically, the present invention can splice the semantic feature information and text emotion feature information of the second text data, as well as the facial feature information and lip feature information of the second facial video data in dimensional order to obtain target feature information after cross-modal fusion.

[0042] S32: Input the target feature information after cross-modal fusion into the pre-trained multimodal speech model to predict the generated virtual face video data. Specifically, the present invention inputs the target feature information after cross-modal fusion into the pre-trained multimodal speech model. The model will perform complex calculations and reasoning based on the patterns and rules learned by itself through the internal neural network structure to predict the facial features, expressions, lip opening and closing, facial muscle movements, and corresponding audio information in each frame of the image, and finally completely construct the virtual face video data.

[0043] S40: constructing a target loss function for training the pre-trained multimodal speech model, and updating the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model. Specifically, Figure 4 As stated, Figure 4 yes Figure 2 A specific implementation flow diagram of step S40 in the embodiment of the present invention specifically includes the following steps S41-S42:

[0044] S41: constructing a loss function for face image generation during training of a pre-trained multimodal speech model, and constructing a loss function for lip movement generation during training of a pre-trained multimodal speech model. The purpose of constructing a loss function for face image generation in the present invention is mainly to measure the difference between the image generated by the pre-trained multimodal speech model and the expected real face image when generating face images. For example, whether the feature information of the generated face image in terms of facial features layout, facial contour, facial texture, etc. is consistent with the actual situation or meets the expected set standards, the difference between the generated face image and the target face image can be directly compared by constructing a mean square error loss function. In addition, the purpose of constructing a loss function for lip movement generation is mainly to measure the gap between the model in generating virtual face lip movements and the expected lip movements, such as the difference between the degree of opening and closing of the lips, the dynamic changes of the lip shape and the actual degree of opening and closing of the lips, the dynamic changes of the lip shape. Specifically, the difference between the generated lip movements and the target lip movements can be directly compared by constructing a key point mean square error loss function.

[0045] S42: Perform weighted processing on the face image generation loss function and the lip movement generation loss function to obtain the target loss function for the pre-trained multimodal speech model training. Specifically, the present invention can set corresponding weight coefficients for the face image generation loss function and the lip movement generation loss function, and assign different weights to the face image generation loss function and the lip movement generation loss function, respectively, to obtain the face image generation loss function and the lip movement generation loss function.

[0046] S50: Obtain third text data of the virtual face to be generated, and input the third text data into the trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face. Figure 5 As stated, Figure 5 yes Figure 2 A specific implementation flow diagram of step S50 in the embodiment of the present invention specifically includes the following steps S51-S52:

[0047] S51: Input the third text data into the trained multimodal speech model, and extract the text semantic feature information and text emotional feature information of the third text data. Specifically, the present invention inputs the third text data into the trained multimodal speech model, and can extract the text semantic feature information by inputting the third text data into the trained multimodal speech model to show the core content, key concepts and semantic associations of the text. It can also extract the text emotional feature information through the trained multimodal speech model to judge the overall emotional tendency of the text and the specific emotional category.

[0048] S52: Predict the speech content information, speech emotion information and speech style information of the virtual face, as well as the facial features information and expression category information of the virtual face, based on the extracted semantic feature information and text emotion feature information of the third text data. Specifically, the present invention can predict the speech-related information of the virtual face, the facial features information of the virtual face and the expression category information of the facial features information of the virtual face, based on the extracted semantic feature information and text emotion feature information of the third text data, to prepare for the subsequent generation of the virtual face. Among them, the speech-related information of the virtual face includes speech content information, speech emotion information that matches the text emotion, and speech style information, to prepare for the subsequent generation of the virtual face.

[0049] S60: Generate virtual face video data based on the predicted speech representation information and facial expression information of the virtual face based on an animation synthesis algorithm. Specifically, the present invention can be based on a key frame animation synthesis algorithm. Based on the predicted speech representation information and facial expression information of the virtual face, complete virtual face video data is constructed in a chronological order and in a reasonable logical relationship.

[0050] In one embodiment of the present invention, the method further includes: presetting a matching benchmark of the voice representation information and facial expression information with the text emotion feature information, and calculating the matching degree of the voice representation information and facial expression information with the text emotion feature information; scoring the generated virtual face video data according to the preset matching benchmark and the calculated matching degree of the voice representation information and facial expression information with the text emotion feature information. Specifically, in order to measure whether the generated virtual face video data is accurate and reasonable in terms of emotional expression, it is necessary to preset a matching benchmark of the voice representation information and facial expression information with the text emotion feature information, which is used to determine whether the emotion of the subsequent virtual face video data actually generated is consistent with the emotion of the preset generated virtual face video data. In addition, the generated virtual face video data is scored according to the preset matching benchmark and the calculated matching value of the actual voice representation information and facial expression information with the text emotion feature information. This score intuitively reflects the quality of the virtual face video data in terms of the accuracy of emotional expression, so as to further evaluate whether the generated virtual face video data meets the requirements, and also provides a reference for further optimizing the relevant models of virtual face generation.

[0051] It can be seen that in the above scheme, when generating a virtual face, the text semantic feature information and text emotional feature information of the second text data can be extracted through the text encoding model, and the face feature information and lip feature information of the second face video data can be extracted through the face detection model, and the third text data can be input into the trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face, thereby effectively improving the realism of the generated virtual face.

[0052] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0053] In one embodiment, a virtual face generation device based on artificial intelligence is provided, and the virtual face generation device based on artificial intelligence corresponds to the virtual face generation method based on artificial intelligence in the above embodiment. Figure 6 As shown, Figure 6 6 is a schematic diagram of a structure of a virtual face generation device based on artificial intelligence in one embodiment of the present invention, the virtual face generation device based on artificial intelligence includes a processing module 61, a feature extraction module 62, a training module 63, a construction module 64, a prediction module 65, and a generation module 66. The functional modules are described in detail as follows:

[0054] A processing module 61 is used to obtain first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data;

[0055] A feature extraction module 62, configured to extract text semantic feature information and text emotional feature information of the second text data based on a text encoding model, and to extract face feature information and lip feature information of the second face video data based on a face detection model;

[0056] A training module 63 is used to obtain a pre-trained multimodal speech model, and input the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training;

[0057] A construction module 64 is used to construct a target loss function for training the pre-trained multimodal speech model, and update the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model;

[0058] A prediction module 65, used to obtain third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face;

[0059] The generation module 66 is used to generate virtual face video data based on the predicted speech representation information and facial expression information of the virtual face and based on an animation synthesis algorithm.

[0060] In one embodiment, the processing module 61 is specifically configured to:

[0061] Performing noise reduction processing on the first face video data to obtain noise-reduced face video data;

[0062] The face video data after noise reduction is subjected to coding format conversion processing to obtain second face video data after coding format conversion.

[0063] In one embodiment, the processing module 61 is specifically configured to:

[0064] Performing data cleaning on the first text data to obtain text data after data cleaning;

[0065] The text data after data cleaning is subjected to word segmentation processing to obtain second text data after word segmentation processing.

[0066] In one embodiment, the training module 63 is specifically configured to:

[0067] Inputting the semantic feature information and text emotion feature information of the second text data, and the face feature information and lip feature information of the second face video data into the cross-modal fusion layer of the pre-trained multimodal speech model for feature fusion, to obtain target feature information after cross-modal fusion;

[0068] The target feature information after the cross-modal fusion is input into the pre-trained multimodal speech model to predict and generate virtual face video data.

[0069] In one embodiment, the construction module 64 is specifically configured to:

[0070] Constructing a face image generation loss function when training a pre-trained multimodal speech model, and constructing a lip movement generation loss function when training a pre-trained multimodal speech model;

[0071] The face image generation loss function and the lip movement generation loss function are weighted to obtain a target loss function for training a pre-trained multimodal speech model.

[0072] In one embodiment, the prediction module 65 is specifically configured to:

[0073] Inputting the third text data into the trained multimodal speech model to extract text semantic feature information and text emotional feature information of the third text data;

[0074] Based on the extracted semantic feature information and text emotion feature information of the third text data, the voice content information, voice emotion information and voice style information of the virtual face, as well as the facial features morphology information and expression category information of the virtual face are predicted.

[0075] In one embodiment, the artificial intelligence-based virtual face generation device is further used to:

[0076] Presetting a matching degree benchmark between the speech representation information and the facial expression information and the text emotion feature information, and calculating the matching degree between the speech representation information and the facial expression information and the text emotion feature information;

[0077] The generated virtual face video data is scored according to the preset matching benchmark and the matching degree of the calculated speech representation information and facial expression information with the text emotion feature information.

[0078] The present invention provides a virtual face generation device based on artificial intelligence. When generating a virtual face, text semantic feature information and text emotional feature information of second text data can be extracted through a text encoding model, and face feature information and lip feature information of second face video data can be extracted through a face detection model, and the third text data is input into a trained multimodal speech model to predict the speech representation information and facial expression information of the virtual face, thereby effectively improving the realism of the generated virtual face.

[0079] For the specific definition of the virtual face generation device based on artificial intelligence, please refer to the definition of the virtual face generation method based on artificial intelligence above, which will not be repeated here. Each module in the above-mentioned virtual face generation device based on artificial intelligence can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, Figure 7 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a virtual face generation method based on artificial intelligence.

[0081] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 81 is another structural diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a virtual face generation method based on artificial intelligence are implemented.

[0082] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0083] Acquire first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data;

[0084] Extracting text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and extracting face feature information and lip feature information of the second face video data based on the face detection model;

[0085] Obtaining a pre-trained multimodal speech model, and inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training;

[0086] Constructing a target loss function for training a pre-trained multimodal speech model, and updating the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model;

[0087] Acquire third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face;

[0088] According to the predicted speech representation information and facial expression information of the virtual face, virtual face video data is generated based on an animation synthesis algorithm.

[0089] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0090] Acquire first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data;

[0091] Extracting text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and extracting face feature information and lip feature information of the second face video data based on the face detection model;

[0092] Obtaining a pre-trained multimodal speech model, and inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training;

[0093] Constructing a target loss function for training a pre-trained multimodal speech model, and updating the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model;

[0094] Acquire third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face;

[0095] According to the predicted speech representation information and facial expression information of the virtual face, virtual face video data is generated based on an animation synthesis algorithm.

[0096] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0098] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0099] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for generating a virtual face based on artificial intelligence, characterized in that: include: Acquire first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data; Extracting text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and extracting face feature information and lip feature information of the second face video data based on the face detection model; Obtaining a pre-trained multimodal speech model, and inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training; Constructing a target loss function for training a pre-trained multimodal speech model, and updating the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model; Acquire third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face; According to the predicted speech representation information and facial expression information of the virtual face, virtual face video data is generated based on an animation synthesis algorithm.

2. The method for generating a virtual face based on artificial intelligence according to claim 1, characterized in that: The performing data preprocessing on the first face video data to obtain the second face video data includes: Performing noise reduction processing on the first face video data to obtain noise-reduced face video data; The face video data after noise reduction is subjected to coding format conversion processing to obtain second face video data after coding format conversion.

3. The method for generating a virtual face based on artificial intelligence according to claim 1, characterized in that: The performing data preprocessing on the first text data to obtain second text data includes: Performing data cleaning on the first text data to obtain text data after data cleaning; The text data after data cleaning is subjected to word segmentation processing to obtain second text data after word segmentation processing.

4. The method for generating a virtual face based on artificial intelligence according to claim 1, characterized in that: The step of inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into a pre-trained multimodal speech model for training includes: Inputting the semantic feature information and text emotion feature information of the second text data, and the face feature information and lip feature information of the second face video data into the cross-modal fusion layer of the pre-trained multimodal speech model for feature fusion, to obtain target feature information after cross-modal fusion; The target feature information after the cross-modal fusion is input into the pre-trained multimodal speech model to predict and generate virtual face video data.

5. The method for generating a virtual face based on artificial intelligence according to claim 1, characterized in that: The target loss function for constructing the pre-trained multimodal speech model training includes: Constructing a face image generation loss function when training a pre-trained multimodal speech model, and constructing a lip movement generation loss function when training a pre-trained multimodal speech model; The face image generation loss function and the lip movement generation loss function are weighted to obtain a target loss function for training a pre-trained multimodal speech model.

6. The method for generating a virtual face based on artificial intelligence according to claim 1, characterized in that: Acquire third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face; Inputting the third text data into the trained multimodal speech model to extract text semantic feature information and text emotional feature information of the third text data; Based on the extracted semantic feature information and text emotion feature information of the third text data, the voice content information, voice emotion information and voice style information of the virtual face, as well as the facial features morphology information and expression category information of the virtual face are predicted.

7. The method for generating a virtual face based on artificial intelligence according to claim 1, characterized in that: The method further comprises: Presetting a matching degree benchmark between the speech representation information and the facial expression information and the text emotion feature information, and calculating the matching degree between the speech representation information and the facial expression information and the text emotion feature information; The generated virtual face video data is scored according to the preset matching benchmark and the matching degree of the calculated speech representation information and facial expression information with the text emotion feature information.

8. A virtual face generation device based on artificial intelligence, characterized in that: The device is used to implement the artificial intelligence-based virtual face generation method according to any one of claims 1 to 7, comprising: a processing module, configured to obtain first facial video data and first text data corresponding to the first facial video data, and perform data preprocessing on the first facial video data and the first text data to obtain second facial video data and second text data; A feature extraction module, used to extract text semantic feature information and text emotional feature information of the second text data based on the text encoding model, and to extract face feature information and lip feature information of the second face video data based on the face detection model; A training module, used for obtaining a pre-trained multimodal speech model, and inputting the extracted semantic feature information and text emotion feature information of the second text data, and the facial feature information and lip feature information of the second facial video data into the pre-trained multimodal speech model for training; A construction module is used to construct a target loss function for training a pre-trained multimodal speech model, and to update the pre-trained multimodal speech model based on the target loss function to obtain a trained multimodal speech model; A prediction module, used to obtain third text data of a virtual face to be generated, and input the third text data into a trained multimodal speech model to predict speech representation information and facial expression information of the virtual face; The generation module is used to generate virtual face video data based on the predicted speech representation information and facial expression information of the virtual face and based on the animation synthesis algorithm.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a virtual face based on artificial intelligence as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a virtual face based on artificial intelligence as described in any one of claims 1 to 7 are implemented.

Images