Digital human video generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a digital human video generation method, device and equipment and a medium, and the method comprises the steps: obtaining target audio data, a target digital human image and target emotion prompt information, carrying out the feature conversion of the target audio data, determining the pixel features corresponding to the target audio data, and according to the target emotion prompt information, carrying out the feature conversion of the target digital human image; and matching a corresponding emotion feature extraction technology, performing feature extraction on the target emotion prompt information through the emotion feature extraction technology to obtain an emotion feature corresponding to the target emotion prompt information, inputting the vision feature and the emotion feature into a model which has been subjected to facial action prediction training, outputting a facial action coefficient, and performing facial action prediction. And generating a target digital human video according to the target audio data, the target digital human image and the facial action coefficient. And the authenticity and accuracy of the generated digital human in emotion expression are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating a digital human video. Background Art

[0002] Digital human refers to a simulated character model generated by computer vision, computer graphics, etc. In a broad sense, all virtual humans with human forms simulated by computers are called digital humans. With the development of virtual reality technology and digital human technology, virtual digital humans, as an important branch of the field of artificial intelligence, are gradually becoming an emerging way to provide customer service. However, existing technologies have obvious deficiencies in generating digital humans with natural emotional expressions. Specifically, when processing voice-driven digital humans, existing systems can often only generate simple lip synchronization, but cannot accurately express complex emotions such as joy, anger, sorrow, and happiness. The existence of these problems limits the use of digital humans in application scenarios that require rich emotional interactions. For example, in virtual customer service scenarios, if digital humans cannot accurately express emotions of sympathy or understanding, they may not be able to effectively establish a trusting relationship with users, thereby greatly reducing the user experience.

[0003] Therefore, how to improve the authenticity and accuracy of the generated digital human's emotional expression has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a method, device, computer equipment and medium for generating a digital human video, so as to solve the technical problem of how to improve the authenticity and accuracy of the generated digital human in terms of emotional expression.

[0005] In a first aspect, a method for generating a digital human video is provided, comprising: Obtain target audio data, target digital human image and target emotional prompt information; Performing feature conversion on the target audio data to determine a visual element feature corresponding to the target audio data; According to the target emotion prompt information, a corresponding emotion feature extraction technology is matched, and the target emotion prompt information is subjected to feature extraction by the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information; Inputting the viseme feature and the emotion feature into a model that has been trained for facial action prediction, and outputting a facial action coefficient; A target digital human video is generated according to the target audio data, the target digital human image and the facial action coefficient.

[0006] In a second aspect, a digital human video generation device is provided, comprising: The first acquisition module is used to acquire target audio data, target digital human image and target emotional prompt information; A feature conversion module, used to perform feature conversion on the target audio data to determine the viseme features corresponding to the target audio data; A first feature extraction module is used to match the corresponding emotion feature extraction technology according to the target emotion prompt information, and extract the characteristics of the target emotion prompt information by using the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information; A first prediction module, used for inputting the viseme feature and the emotion feature into a model that has been trained for facial action prediction, and outputting a facial action coefficient; The first generating module is used to generate a target digital human video according to the target audio data, the target digital human image and the facial action coefficient.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the digital human video generation method of the first aspect are implemented.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the digital human video generation method in the first aspect are implemented.

[0009] In the scheme implemented by the above-mentioned digital human video generation method, device, computer equipment and storage medium, the corresponding visual element features are obtained by performing feature conversion on the target audio data, and the target emotional prompt information is extracted by matching the corresponding emotional feature extraction technology to obtain the corresponding emotional features. The visual element features and emotional features are input into the model that has been trained for facial action prediction, and the facial action coefficients are output. According to the target audio data, the target digital human image and the facial action coefficients, the target digital human video is generated.

[0010] Among them, by performing feature conversion on the target audio data, the corresponding visual element features are obtained, the emotional information in the target audio data is removed, and the purpose of decoupling speech and emotion is achieved. Multimodal target emotional prompt information is introduced, so that the generated digital human learns emotional expression from the multimodal target emotional prompt information to integrate the emotional expression in the multimodal target emotional prompt information, and by performing feature extraction and feature fusion on the multimodal target emotional prompt information, the difference between the emotional representations of different modalities is minimized, so that the generated digital human can consistently express the specified emotions, thereby improving the authenticity and accuracy of the generated digital human video in emotional expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0012] Figure 1 A schematic diagram of an application environment of a digital human video generation method provided in Embodiment 1 of the present invention; Figure 2 A schematic diagram of a flow chart of a method for generating a digital human video provided in the second embodiment of the present invention; Figure 3 A schematic diagram of the overall process of a digital human video generation method provided in Embodiment 2 of the present invention; Figure 4 A schematic diagram of a flow chart of a method for generating a digital human video provided in Embodiment 3 of the present invention; Figure 5 A schematic diagram of a flow chart of a method for generating a digital human video provided in Embodiment 4 of the present invention; Figure 6 A schematic diagram of a flow chart of a method for generating a digital human video provided in Embodiment 5 of the present invention; Figure 7 A schematic diagram of a flow chart of a method for generating a digital human video according to a sixth embodiment of the present invention; Figure 8 A schematic diagram of a flow chart of a method for generating a digital human video provided in Embodiment 7 of the present invention; Fig. 9 A schematic diagram of a flow chart of a method for generating a digital human video provided in Embodiment 8 of the present invention; Fig.10 A schematic diagram of the structure of a digital human video generating device provided in Embodiment 9 of the present invention; Fig.11 A schematic diagram of the structure of a computer device provided in Embodiment 10 of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The digital human video generation method provided in the first embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment, the server communicates with the client, the server provides digital human video generation service, and the client triggers the digital human video generation task to the server. The client includes but is not limited to PDA, desktop computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cloud computer equipment, personal digital assistant (PDA) and other devices. The computer equipment corresponding to the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0015] like Figure 2 FIG. 1 is a flow chart of a method for generating a digital human video provided by Embodiment 2 of the present invention, comprising the following steps: Step S201: Acquire target audio data, target digital human image and target emotion prompt information.

[0016] In this embodiment, the target audio data may refer to the audio data customized for the constructed digital human video, the target digital human image may refer to the digital human image customized for the constructed digital human video, and the target emotional prompt information may refer to the emotional expression information customized for the constructed digital human video, wherein the target emotional prompt information may include data information of data types such as audio, text, and pictures.

[0017] For example, the target emotional prompt information of the text type can be: When generating digital facial animation, ensure that the expression transitions smoothly from smiling to surprised. In the first half of the video, the character smiles, the eyes are slightly narrowed, and the corners of the mouth are slightly raised. In the second half of the video, the character is surprised and encounters an unexpected situation. The character's eyebrows rise quickly, the eyes are widened, and the mouth is slightly opened to form an O shape.

[0018] Specifically, a user-defined digital human image can be obtained as a target digital human image, audio data in a video or voice can be obtained as target audio data, and emotional prompt information in data types such as audio, text or pictures can be obtained as target emotional prompt information.

[0019] Step S202: Perform feature conversion on the target audio data to determine the visual element features corresponding to the target audio data.

[0020] In this embodiment, the visual element feature may refer to a mouth shape representation associated with a specific pronunciation action in the target audio data.

[0021] Specifically, first, the phoneme time sequence corresponding to the target audio data can be determined by calling a tool with speech phoneme detection, and then, the phoneme time sequence corresponding to the target audio data can be converted into a viseme time sequence through a phoneme-to-viseme algorithm. The viseme time sequence is the viseme feature corresponding to the target audio data.

[0022] Step S203: According to the target emotion prompt information, the corresponding emotion feature extraction technology is matched, and the target emotion prompt information is subjected to feature extraction through the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information.

[0023] In this embodiment, the emotion feature extraction technology may refer to a technology that has the function of extracting emotion features from target emotion prompt information. The emotion features may refer to various characteristics manifested when expressing emotions. For example, the emotion features may include psychological characteristics such as joy, sadness, and anger, as well as action characteristics such as squinting, pouting, and frowning.

[0024] Specifically, according to the data type of the target emotional prompt information, the corresponding emotional feature extraction technology is matched, and the emotional prompt data of the corresponding data type is subjected to feature extraction through the emotional feature extraction technology to obtain the emotional features corresponding to the corresponding emotional prompt data, and the emotional features corresponding to the emotional prompt data of all data types in the target emotional prompt information are subjected to feature fusion to obtain the emotional features corresponding to the target emotional prompt information.

[0025] Step S204: input the viseme features and emotion features into the model that has been trained for facial action prediction, and output the facial action coefficients.

[0026] In this embodiment, the model that has been trained for facial action prediction may refer to a generative model that can use input visual element features and emotional features to predict output facial action coefficients. Through training, the model has learned the mapping relationship between input and output. The facial action coefficient may refer to a specific parameter or value that describes the specific movement mode and degree of each part of the digital human's face.

[0027] Specifically, the visual element features and emotional features are input into a model that has been trained for facial action prediction to obtain numerical values ​​representing the specific movement mode and degree of each part of the digital human's face, and the numerical values ​​of each part of the face are used to form facial action coefficients.

[0028] Step S205: Generate a target digital human video according to the target audio data, the target digital human image and the facial action coefficients.

[0029] In this embodiment, the target digital human video may refer to digital human video data generated according to target audio data, target digital human image and target emotion prompt information.

[0030] Specifically, the target audio data, the target digital human image and the facial action coefficients can be input into a renderer with video rendering generation, and the target digital human video can be obtained after being processed by the renderer.

[0031] For example, Figure 3 This is a schematic diagram of the overall process of a digital human video generation method provided in Embodiment 2 of the present invention. The overall process of the digital human video generation method may include two stages: voice to action and action to video. The overall process of the digital human video generation method may be: Speech-to-action stage: 1) Perform phoneme recognition on the target audio data to determine the phoneme features corresponding to the target audio data; 2) Perform feature conversion on the phoneme features corresponding to the target audio data to obtain the viseme features corresponding to the target audio data; 3) Perform feature extraction on the target emotional prompt information through emotional feature extraction technology to obtain the emotional features corresponding to the target emotional prompt information; 4) Input the viseme features corresponding to the target audio data and the emotional features corresponding to the target emotional prompt information into the model that has been trained for facial action prediction, and output the facial action coefficients.

[0032] Action to video stage: 5) Input the target audio data, target digital human image and facial action coefficients into the video renderer to obtain the target digital human video.

[0033] In this embodiment, by performing feature conversion on the target audio data to obtain the corresponding visual element features, the emotional information in the target audio data is removed to achieve the purpose of decoupling speech and emotion, and multimodal target emotional prompt information is introduced so that the generated digital human learns emotional expression from the multimodal target emotional prompt information to integrate the emotional expression in the multimodal target emotional prompt information, and by performing feature extraction and feature fusion on the multimodal target emotional prompt information, the difference between the emotional representations of different modalities is minimized, so that the generated digital human can consistently express the specified emotions, thereby improving the authenticity and accuracy of the generated digital human video in emotional expression.

[0034] like Figure 4 As shown in FIG. 1 , a flow chart of a method for generating a digital human video provided by Embodiment 3 of the present invention is shown. In the above step S202, feature conversion of the target audio data is performed to determine the visual element features corresponding to the target audio data, which may include the following steps: Step S401: Perform phoneme recognition on the target audio data to determine the phoneme features corresponding to the target audio data.

[0035] Step S402: Perform feature conversion on the phoneme features through phoneme-viseme conversion technology to obtain viseme features.

[0036] In this embodiment, the phoneme feature may refer to a phoneme time series corresponding to the target audio data, and the phoneme-viseme conversion technology may refer to a technology that has the function of converting phonemes into visemes.

[0037] Specifically, first, the target audio data can be preprocessed such as noise reduction and standardization, and then the phoneme time series corresponding to the target audio data can be determined by calling the speech phoneme detection tool. Finally, the phoneme time series corresponding to the target audio data can be converted into a viewme time series through the phoneme-viseme conversion technology. The viewme time series is the viewme feature corresponding to the target audio data.

[0038] In this embodiment, the phoneme features corresponding to the target audio data are determined by performing phoneme recognition on the target audio data, and then the phoneme features are converted to viseme features by using the phoneme-viseme conversion technology. The emotional information in the target audio data is removed to achieve the purpose of decoupling speech and emotion, so that the generated digital human learns emotional expression from the multimodal target emotional prompt information, thereby improving the authenticity and accuracy of the generated digital human video in emotional expression.

[0039] like Figure 5 As shown, it is a flow chart of a method for generating a digital human video provided by the fourth embodiment of the present invention. In the above step S203, according to the target emotion prompt information, the corresponding emotion feature extraction technology is matched, and the target emotion prompt information is extracted by the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information, which may include the following steps: Step S501: determine the data types included in the target emotion prompt information, and determine the emotion prompt data corresponding to each data type.

[0040] Step S502: For any emotion prompt data in the target emotion prompt information, match the corresponding emotion feature extraction technology according to the data type corresponding to the emotion prompt data, extract features from the emotion prompt data through the emotion feature extraction technology, and obtain the emotion features corresponding to the emotion prompt data.

[0041] Step S503: performing feature fusion on the emotion features corresponding to all emotion prompt data in the target emotion prompt information to obtain the emotion features corresponding to the target emotion prompt information.

[0042] In this embodiment, the emotion prompt data may refer to data of a specific data type included in the target emotion prompt information.

[0043] Specifically, for the audio type emotion prompt data contained in the target emotion prompt information, feature extraction can be performed on the audio type emotion prompt data through a first emotion extraction model that has been trained for audio emotion feature extraction, so as to obtain emotion features corresponding to the audio type emotion prompt data; for the text type or picture type emotion prompt data contained in the target emotion prompt information, feature extraction can be performed on the text type or picture type emotion prompt data through a second emotion extraction model that has been trained for text emotion feature and picture emotion feature extraction, so as to obtain emotion features corresponding to the text type or picture type emotion prompt data; the emotion features corresponding to all the emotion prompt data in the target emotion prompt information are feature fused to obtain the emotion features corresponding to the target emotion prompt information.

[0044] In the process of training the first emotion extraction model and the second emotion extraction model, the emotion features corresponding to the emotion prompt data of the audio type are calculated through the untrained first emotion extraction model, and the emotion features corresponding to the emotion prompt data of the text type or the picture type are calculated through the untrained second emotion extraction model. Then, the contrast loss between the emotion features corresponding to the emotion prompt data of different data types extracted by the two models is calculated through a predefined emotion alignment contrast loss function. When the calculated emotion alignment contrast loss meets the training termination condition, the first emotion extraction model that has been trained for audio emotion feature extraction and the second emotion extraction model that has been trained for text emotion feature extraction and picture emotion feature extraction are obtained.

[0045] Among them, the predefined sentiment alignment comparison loss function in the above training process can be: , , They represent the styles of emotional features corresponding to emotional prompt data of different data types. is the corresponding style vector, is the cosine similarity function.

[0046] In this embodiment, by matching the corresponding emotion feature extraction technology according to the data type of the emotion prompt data contained in the target emotion prompt information, the emotion feature extraction technology is used to extract the emotion prompt data of the corresponding data type, thereby obtaining the emotion feature corresponding to the target emotion prompt information. By selecting the most appropriate emotion extraction model for each data type of emotion prompt data to extract the corresponding emotion feature, and the corresponding emotion extraction model has minimized the difference between the emotion features corresponding to the emotion prompt data of different data types extracted through the predefined emotion alignment loss function during the training process, so that the generated digital human can consistently express the specified emotion, thereby improving the accuracy of the emotion features corresponding to the extracted target emotion prompt information.

[0047] like Figure 6 As shown, it is a flow chart of a method for generating a digital human video provided by Embodiment 5 of the present invention. In the above step S502, according to the data type corresponding to the emotional prompt data, the corresponding emotional feature extraction technology is matched, and the emotional prompt data is feature extracted by the emotional feature extraction technology to obtain the emotional features corresponding to the emotional prompt data, which may include the following steps: Step S601: If the emotion prompt data is of audio type, a first emotion extraction model that has been trained for audio emotion feature extraction is obtained.

[0048] Step S602: Using the first emotion extraction model, feature extraction is performed on the audio type emotion prompt data to obtain emotion features corresponding to the audio type emotion prompt data.

[0049] In this embodiment, the first emotion extraction model may refer to a model for extracting emotion features from audio-type emotion prompt data.

[0050] Specifically, the audio type emotion prompt data is subjected to preprocessing steps such as denoising, standardization and segmentation, and the audio type emotion prompt data is input into a first emotion extraction model. The first emotion extraction model performs emotion feature extraction on the input audio type emotion prompt data, and outputs the emotion features corresponding to the audio type emotion prompt data.

[0051] In this embodiment, the emotion feature extraction of the audio type emotion prompt data is performed through the first emotion extraction model that has been trained for audio emotion feature extraction, and the emotion feature corresponding to the audio type emotion prompt data is obtained, thereby improving the accuracy of the emotion feature corresponding to the extracted audio type emotion prompt data.

[0052] like Figure 7As shown, it is a flow chart of a method for generating a digital human video provided by Embodiment 6 of the present invention. In the above step S502, according to the data type corresponding to the emotion prompt data, the corresponding emotion feature extraction technology is matched, and the emotion prompt data is subjected to feature extraction by the emotion feature extraction technology to obtain the emotion feature corresponding to the emotion prompt data, which may include the following steps: Step S701: If the emotion prompt data is of text type or picture type, a second emotion extraction model that has been trained for extracting text emotion features and picture emotion features is obtained.

[0053] Step S702: Using a second emotion extraction model, feature extraction is performed on the emotion prompt data of the text type or the picture type to obtain emotion features corresponding to the emotion prompt data of the text type or the picture type.

[0054] In this embodiment, the second emotion extraction model may refer to a model for extracting emotion features from text-type or image-type emotion prompt data.

[0055] Specifically, denoising, standardization, segmentation and other preprocessing steps are performed on the text type or picture type emotion prompt data and the audio type emotion prompt data, and the text type or picture type emotion prompt data is input into the second emotion extraction model. The second emotion extraction model performs emotion feature extraction on the input text type or picture type emotion prompt data, and outputs the emotion feature corresponding to the text type or picture type emotion prompt data.

[0056] In this embodiment, the emotion feature extraction is performed on the emotion prompt data of the text type or the picture type by using the second emotion extraction model that has been trained for text emotion feature extraction and audio emotion feature extraction, so as to obtain the emotion feature corresponding to the emotion prompt data of the text type or the picture type. This improves the accuracy of the emotion feature corresponding to the extracted emotion prompt data of the text type or the picture type.

[0057] like Figure 8 As shown in FIG. 1 , a flow chart of a method for generating a digital human video provided by Embodiment 7 of the present invention is shown. In the above step S203, the viseme features and the emotion features are input into a model that has been trained for facial action prediction, and the facial action coefficient is output, which may include the following steps: Step S801: Input the viseme features and emotion features into the model that has been trained for facial action prediction to obtain the part action coefficients of each facial part.

[0058] Step S802: All the part action coefficients are converted into facial action coefficients.

[0059] In this embodiment, the part movement coefficient may refer to a specific parameter or value representing the specific movement mode and degree of each part of the face. For example, the part movements corresponding to the facial movement coefficient may include blinking, looking downward, squinting, pouting with the chin to the right, lips pressed down, eyebrows to the upper left, nose frowned to the left, cheeks raised and rotated, etc.

[0060] Specifically, the visual element features and emotional features are input into a model that has been trained for facial movement prediction. After being processed by the model, the parameters and values ​​representing the degree and direction of movement of various parts such as eyebrows, eyes and mouth are output. The part movement coefficients of various facial parts output by the model are integrated to obtain the facial action coefficients.

[0061] In this embodiment, by inputting viseme features and emotion features into a model that has been trained for facial action prediction, the part action coefficients of each part of the face are obtained, and all the part action coefficients are combined to form facial action coefficients. Combining viseme features and emotion features, the facial action coefficients predicted by the model integrate the lip shape features in the target audio data and the emotional expression in the multimodal target emotional prompt information, so that the generated digital human can consistently express the specified emotions, thereby improving the authenticity and accuracy of the generated digital human video in terms of emotional expression.

[0062] like Fig. 9 FIG. 2 is a flow chart of a method for generating a digital human video according to an eighth embodiment of the present invention. In the above step S204, generating a target digital human video according to target audio data, target digital human image and facial action coefficients may include the following steps: Step S901: adjusting the target digital human image according to the facial action coefficient to obtain an adjusted target digital human image.

[0063] Step S902: Process the adjusted target digital human image and target audio data through video generation technology to obtain the target digital human video.

[0064] Specifically, by calling a specific facial movement adjustment algorithm, the corresponding parts of the target digital human image are adjusted according to the parameters and values ​​representing the movement degree and direction of each part in the facial movement coefficient, and the corresponding movement movements are simulated to obtain the adjusted target digital human image. Through video generation technology, the target audio data and the adjusted target digital human image are synchronized on the time axis, and the synchronized audio and image are combined to generate the target digital human video.

[0065] In this embodiment, the target digital human image is adjusted according to the facial action coefficient to obtain the adjusted target digital human image, and the adjusted target digital human image and target audio data are processed by video generation technology to obtain the target digital human video. The facial action coefficient can describe the movement of various parts of the face. By adjusting the target digital human image by the facial action coefficient, the facial action can be matched with the emotional expression corresponding to the target emotional prompt information, so that the generated digital human video integrates the emotional expression in the target emotional prompt information, and improves the authenticity and accuracy of the generated digital human video in terms of emotional expression.

[0066] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0067] like Fig.10 As shown, a digital human video generation device provided by Embodiment 9 of the present invention is provided. The digital human video generation device corresponds one-to-one to the digital human video generation method in the above-mentioned embodiment. Fig.10 As shown, the digital human video generation device includes a first acquisition module 1001, a feature conversion module 1002, a first feature extraction module 1003, a first prediction module 1004 and a first generation module 1005. The functional modules are described in detail as follows: The first acquisition module 1001 is used to acquire target audio data, target digital human image and target emotion prompt information; A feature conversion module 1002 is used to perform feature conversion on the target audio data to determine the viseme features corresponding to the target audio data; The first feature extraction module 1003 is used to match the corresponding emotion feature extraction technology according to the target emotion prompt information, and extract the characteristics of the target emotion prompt information by using the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information; A first prediction module 1004, for inputting the viseme feature and the emotion feature into a model that has been trained for facial action prediction, and outputting a facial action coefficient; The first generating module 1005 is used to generate a target digital human video according to the target audio data, the target digital human image and the facial action coefficient.

[0068] Optionally, the feature conversion module 1002 includes: A phoneme recognition unit, used to perform phoneme recognition on the target audio data to determine phoneme features corresponding to the target audio data; The viseme conversion unit is used to perform feature conversion on the phoneme feature through the phoneme-viseme conversion technology to obtain the viseme feature.

[0069] Optionally, the first feature extraction module 1003 includes: A determination unit, used to determine the data type contained in the target emotion prompt information, and determine the emotion prompt data corresponding to each data type; A matching unit, for matching a corresponding emotion feature extraction technology to any emotion prompt data in the target emotion prompt information according to the data type corresponding to the emotion prompt data, and extracting features from the emotion prompt data by using the emotion feature extraction technology to obtain the emotion features corresponding to the emotion prompt data; The first feature fusion unit is used to fuse the emotional features corresponding to all the emotional prompt data in the target emotional prompt information to obtain the emotional features corresponding to the target emotional prompt information.

[0070] Optionally, the matching unit includes: A second acquisition subunit is used for acquiring a first emotion extraction model that has been trained for audio emotion feature extraction if the emotion prompt data is of audio type; The second feature extraction subunit is used to perform feature extraction on the emotion prompt data of the audio type through the first emotion extraction model to obtain the emotion features corresponding to the emotion prompt data of the audio type.

[0071] Optionally, the matching unit includes: A third acquisition subunit is used to acquire a second emotion extraction model that has been trained to extract text emotion features and image emotion features if the emotion prompt data is of text type or image type; The third feature extraction subunit is used to perform feature extraction on the emotional prompt data of the text type or picture type through the second emotion extraction model to obtain emotional features corresponding to the emotional prompt data of the text type or picture type.

[0072] Optionally, the first prediction module 1004 includes: A second prediction unit is used to input the viseme feature and the emotion feature into the model that has been trained for facial action prediction to obtain the part action coefficient of each part of the face; A forming unit is used to form the facial action coefficient from all the part action coefficients.

[0073] Optionally, the first generating module 1005 includes: An adjustment unit, configured to adjust the target digital human image according to the facial action coefficient to obtain an adjusted target digital human image; The second generating unit is used to process the adjusted target digital human image and the target audio data through a video generating technology to obtain the target digital human video.

[0074] The specific definition of the digital human video generation device can be found in the definition of the digital human video generation method above, which will not be repeated here. Each module in the above-mentioned digital human video generation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0075] like Fig.11 FIG. 1 is a schematic diagram of a computer device provided in Embodiment 10 of the present invention. The computer device of this embodiment includes: at least one processor ( Fig.11 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned digital human video generation method embodiments are implemented.

[0076] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.11 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0077] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0079] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0080] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application. The non-company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.

Claims

1. A method for generating a digital human video, characterized in that: include: Obtain target audio data, target digital human image and target emotional prompt information; Performing feature conversion on the target audio data to determine a visual element feature corresponding to the target audio data; According to the target emotion prompt information, a corresponding emotion feature extraction technology is matched, and the target emotion prompt information is subjected to feature extraction by the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information; Inputting the viseme feature and the emotion feature into a model that has been trained for facial action prediction, and outputting a facial action coefficient; A target digital human video is generated according to the target audio data, the target digital human image and the facial action coefficient.

2. The method for generating a digital human video according to claim 1, wherein: The performing feature conversion on the target audio data to determine the viseme features corresponding to the target audio data includes: Performing phoneme recognition on the target audio data to determine phoneme features corresponding to the target audio data; The phoneme feature is converted into the viseme feature by using the phoneme-viseme conversion technology.

3. The method for generating a digital human video according to claim 1, wherein: The step of matching a corresponding emotion feature extraction technology according to the target emotion prompt information, extracting features of the target emotion prompt information by using the emotion feature extraction technology, and obtaining emotion features corresponding to the target emotion prompt information includes: Determine the data types included in the target emotion prompt information, and determine the emotion prompt data corresponding to each data type; For any emotion prompt data in the target emotion prompt information, according to the data type corresponding to the emotion prompt data, a corresponding emotion feature extraction technology is matched, and the emotion prompt data is subjected to feature extraction by the emotion feature extraction technology to obtain the emotion feature corresponding to the emotion prompt data; The emotional features corresponding to all the emotional prompt data in the target emotional prompt information are feature fused to obtain the emotional features corresponding to the target emotional prompt information.

4. The method for generating a digital human video according to claim 3, wherein: The method of matching a corresponding emotion feature extraction technology according to a data type corresponding to the emotion prompt data, extracting features of the emotion prompt data by using the emotion feature extraction technology, and obtaining the emotion features corresponding to the emotion prompt data includes: If the emotion prompt data is of audio type, obtaining a first emotion extraction model that has been trained for audio emotion feature extraction; The first emotion extraction model is used to perform feature extraction on the emotion prompt data of the audio type to obtain emotion features corresponding to the emotion prompt data of the audio type.

5. The method for generating a digital human video according to claim 3, wherein: The method of matching a corresponding emotion feature extraction technology according to a data type corresponding to the emotion prompt data, extracting features of the emotion prompt data by using the emotion feature extraction technology, and obtaining the emotion features corresponding to the emotion prompt data includes: If the emotion prompt data is of text type or picture type, obtaining a second emotion extraction model that has been trained to extract text emotion features and picture emotion features; The second emotion extraction model is used to perform feature extraction on the emotion prompt data of the text type or the picture type to obtain emotion features corresponding to the emotion prompt data of the text type or the picture type.

6. The method for generating a digital human video according to claim 1, wherein: The step of inputting the viseme feature and the emotion feature into a model that has been trained for facial action prediction and outputting a facial action coefficient comprises: Inputting the viseme feature and the emotion feature into the model that has been trained for facial action prediction to obtain the part action coefficients of each part of the face; All the part motion coefficients are formed into the facial motion coefficient.

7. The method for generating a digital human video according to claim 1, wherein: The step of generating a target digital human video according to the target audio data, the target digital human image and the facial action coefficient comprises: According to the facial action coefficient, the target digital human image is adjusted to obtain an adjusted target digital human image; The adjusted target digital human image and the target audio data are processed by video generation technology to obtain the target digital human video.

8. A digital human video generation device, characterized in that: include: The first acquisition module is used to acquire target audio data, target digital human image and target emotional prompt information; A feature conversion module, used to perform feature conversion on the target audio data to determine the viseme features corresponding to the target audio data; A first feature extraction module is used to match the corresponding emotion feature extraction technology according to the target emotion prompt information, and extract the characteristics of the target emotion prompt information by using the emotion feature extraction technology to obtain the emotion feature corresponding to the target emotion prompt information; A first prediction module, used for inputting the viseme feature and the emotion feature into a model that has been trained for facial action prediction, and outputting a facial action coefficient; The first generating module is used to generate a target digital human video according to the target audio data, the target digital human image and the facial action coefficient.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a digital human video according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a digital human video according to any one of claims 1 to 7 are implemented.

Images