Video generation methods, devices and electronic equipment

Abstract

This application provides a video generation method, apparatus, and electronic device, relating to the field of artificial intelligence technology. The method includes: acquiring target audio; inputting the target audio into a trained facial expression parameter recognition model for processing to obtain first facial expression data, the first facial expression data being related to the facial expression style of a training object; inputting the first facial expression data into a pre-configured style transfer matrix for adjustment to obtain adjusted second facial expression data, the second facial expression data being related to the facial expression style of a target object; and generating a target video based on the second facial expression data, in which the target object expresses the target audio. In the above design, obtaining the second facial expression data through a style transfer matrix enables accurate style transfer between the target object and the training object, greatly saving time and cost in the production of virtual digital human facial animation. Simultaneously, it can also improve the natural realism of the virtual digital human and the user experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a video generation method, apparatus, and electronic device. Background Technology

[0002] With the development of artificial intelligence (AI) technology, virtual digital humans have been widely used in live streaming, news broadcasting, voice prompts, and other fields. Virtual digital humans are character models generated using computer vision or computer graphics techniques, creating lifelike or cartoonish figures. Currently, virtual digital humans can be controlled via audio input to display rich facial expressions and lip movements, achieving synchronization between lip animation and input audio. However, in driving virtual digital humans, it is often impossible to accurately reflect the speaking styles of different individuals, especially in terms of personalized characteristics such as emotional expression and lip-syncing habits. This weakens the realism of the virtual digital human and the naturalness of the interactive experience. Summary of the Invention

[0003] In order to at least overcome the above-mentioned shortcomings in the prior art, the purpose of this application is to provide a video generation method, apparatus and electronic device.

[0004] In a first aspect, embodiments of this application provide a video generation method, the video generation method comprising:

[0005] Obtain the target audio;

[0006] The target audio is input into a trained facial expression parameter recognition model for processing to obtain first facial expression data, which is related to the facial expression style of the training object.

[0007] The first facial expression data is input into a pre-configured style transfer matrix for adjustment to obtain the adjusted second facial expression data, which is related to the facial expression style of the target object.

[0008] A target video representing the target audio is generated based on the second facial expression data.

[0009] In one possible implementation, the method further includes:

[0010] The target audio is input into a pre-trained speech emotion recognition model to obtain the emotion recognition result corresponding to the target audio.

[0011] The corresponding style transfer matrix is ​​determined based on the emotion recognition results.

[0012] In one possible implementation, the method further includes:

[0013] Obtain the reference audio and reference video corresponding to the target object;

[0014] The reference audio is input into the trained facial expression parameter recognition model for processing to obtain third facial expression data, which is related to the facial expression style of the training object.

[0015] A fourth facial expression data is obtained based on the reference video, and the fourth facial expression data is related to the facial expression style of the target object;

[0016] The third facial expression data and the fourth facial expression data are compared and analyzed to obtain the style transfer matrix, which is used to convert the facial expression style of the training object into the facial expression style of the target object.

[0017] In one possible implementation, the step of comparing and analyzing the third facial expression data and the fourth facial expression data to obtain the style transfer matrix includes:

[0018] The style transfer matrix is ​​obtained by comparing and analyzing the third and fourth facial expression data using the least squares method.

[0019] In one possible implementation, the style transfer matrix X is calculated as follows:

[0020]

[0021] Among them, A mesh B represents the third facial expression data. mesh This represents the fourth facial expression data, where n represents the number of input images in the trained facial expression parameter recognition model.

[0022] In one possible implementation, the method further includes:

[0023] Obtain the training dataset corresponding to the training object, wherein the training dataset includes multiple sample audios and multiple sample videos corresponding to the sample audios;

[0024] Facial data corresponding to the training object is obtained from the sample video.

[0025] The sample audio and facial data are input into a trained facial expression parameter recognition model for processing to obtain the fifth facial expression data.

[0026] In one possible implementation, the step of obtaining facial data corresponding to the training object based on the sample video includes:

[0027] The sample video is processed using 3D facial tracking technology to obtain facial expression basis coefficients and facial mesh data corresponding to the training object;

[0028] The step of inputting the sample audio and the facial data into a trained facial expression parameter recognition model for processing to obtain the fifth facial expression data includes:

[0029] The sample audio, the facial expression basis coefficients, and the facial mesh data are input into a trained expression parameter recognition model for processing to obtain the fifth facial expression data.

[0030] Secondly, embodiments of this application also provide a video generation apparatus, comprising:

[0031] The receiving module is used to acquire the target audio;

[0032] The processing module is used to input the target audio into a trained facial expression parameter recognition model for processing to obtain first facial expression data, wherein the first facial expression data is related to the facial expression style of the training object;

[0033] A style adjustment model is used to input the first facial expression data into a pre-configured style transfer matrix for adjustment, thereby obtaining the adjusted second facial expression data, which is related to the facial expression style of the target object.

[0034] The generation module is used to generate a target video that describes the target audio based on the second facial expression data.

[0035] Thirdly, embodiments of this application also provide an electronic device, including:

[0036] Memory, used to store one or more programs;

[0037] A processor, when the one or more programs are executed by the processor, implements the video generation method provided in the first aspect above.

[0038] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the video generation method provided in the first aspect above.

[0039] Based on any of the above aspects, the video generation method, apparatus, and electronic device provided in this application, by processing the target audio input into a trained facial expression parameter recognition model to obtain first facial expression data related to the facial expression style of the training object, and by adjusting the first facial expression data into a style transfer matrix to obtain second facial expression data related to the facial expression style of the target object, can achieve style transfer between the target object and the training object, greatly saving the time and cost of virtual digital human facial animation production. Furthermore, through accurate style transfer, the natural realism of the virtual digital human and the user experience can also be improved. In addition, style transfer of the target object can be achieved without collecting a large amount of data, effectively reducing training costs. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is an illustrative application scenario diagram of the video generation method provided in this embodiment;

[0042] Figure 2 This is one of the flowcharts for the video generation method provided in this embodiment;

[0043] Figure 3 This is the second flowchart of the video generation method provided in this embodiment;

[0044] Figure 4 This is the third flowchart of the video generation method provided in this embodiment;

[0045] Figure 5 This is the fourth flowchart of the video generation method provided in this embodiment;

[0046] Figure 6 This is a schematic structural block diagram of the electronic device provided in this embodiment;

[0047] Figure 7 This is a schematic diagram of the functional modules of the video generation device provided in this embodiment.

[0048] Icons: 100 - Server; 200 - User terminal; 700 - Electronic device; 710 - Processor; 720 - Computer-readable storage medium; 730 - Video generation device; 731 - Receiving module; 732 - Processing module; 733 - Style adjustment module; 734 - Generation module. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0052] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] It should be noted that, where there is no conflict, different features in the embodiments of this application can be combined with each other.

[0054] The inventors discovered that when training a model using training data to extract the speaking styles of different individuals, it is necessary to train each individual separately. However, for some individuals, it may not be possible to obtain enough available training data to train a personalized model individually.

[0055] This embodiment provides a solution to the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.

[0056] Please refer to Figure 1 , Figure 1This embodiment provides a video generation system, which includes a server 100 and a user terminal 200. The server 100 and the user terminal 200 can communicate with each other via a wired network or a wireless network. The server 100 can be an independent electronic device or a cluster of multiple electronic devices. The user terminal 200 can be a mobile terminal device, such as a smartphone, personal computer (PC), tablet computer, etc.

[0057] Please refer to Figure 2 , Figure 2 This embodiment provides a flowchart illustrating the video generation method, which can be applied to, for example... Figure 1 The video generation method can be implemented through an application installed on the server 100 shown. When driving a virtual digital human using target audio, the video generation method may include the following steps.

[0058] Step S110: Obtain the target audio.

[0059] In this embodiment, the target audio can be a segment of voice data that needs to be broadcast or broadcast live, and the target audio is the audio that the virtual digital human needs to output. The target audio can be the audio of the real voice output by the target object corresponding to the virtual digital human, or it can be the audio of synthesized voice. The virtual digital human is a digital human used to broadcast the target voice, and the virtual digital human can be a two-dimensional virtual digital human generated based on the target object.

[0060] It should be noted that the target object is not limited to people; it can also be animals, or anthropomorphic animals, objects, etc.

[0061] Step S120: Input the target audio into the trained facial expression parameter recognition model for processing to obtain first facial expression data, which is related to the facial expression style of the training object.

[0062] Because each person has individual characteristics in terms of mouth size, mouth movement speed, and facial expression changes when speaking, using a single, standardized mouth animation to replicate and use between different virtual digital humans will result in the virtual digital human animation lacking personality and realism.

[0063] Therefore, in this embodiment, the target audio obtained in step S110 can be input into the trained facial expression parameter recognition model for processing, and the trained facial expression parameter recognition model can output the first facial expression data related to the facial expression style of the training object. The first facial expression data may include the mouth shape changes and facial expression details unique to the training object, such as the mouth opening range and expression habits.

[0064] Specifically, the facial expression parameter recognition model can map audio signals into facial expression data related to the facial expression style of the training subject. The facial expression parameter recognition model can be a FaceFormer model.

[0065] Step S130: The first facial expression data is input into a pre-configured style transfer matrix for adjustment to obtain the adjusted second facial expression data, which is related to the facial expression style of the target object.

[0066] In this embodiment, the first facial expression data related to the facial expression style of the training object obtained in step S120 can be input into the pre-configured style conversion matrix for adjustment, thereby obtaining the adjusted second facial expression data. The second facial expression data is related to the facial expression style of the target object. The second facial expression data may include the mouth shape changes and facial expression details unique to the target object, such as the mouth opening range and expression habits.

[0067] Specifically, the style transfer matrix can convert the facial expression style of the training object into the facial expression style of the target object.

[0068] Step S140: Generate a target video of the target object describing the target audio based on the second facial expression data.

[0069] In this embodiment, a target video in which the target object expresses the target audio can be generated based on the second facial expression data related to the facial expression style of the target object obtained in step S130. The target video includes the target object, which can be a two-dimensional virtual digital human. The two-dimensional virtual digital human can display lip movements that match the facial expression style of the target object based on the target audio. The target audio is the audio output by the target object in the target-driven video, and the lip movements and facial expressions of the target object match the target audio.

[0070] As can be seen, based on the above design, the video generation method provided in this application, by processing the target audio into a trained facial expression parameter recognition model to obtain first facial expression data related to the facial expression style of the training object, and by adjusting the first facial expression data into the style transfer matrix to obtain second facial expression data related to the facial expression style of the target object, can achieve style transfer between the target object and the training object, greatly saving the time and cost of virtual digital human facial animation production. Furthermore, through accurate style transfer, the natural realism of the virtual digital human and the user experience can be improved. In addition, style transfer of the target object can be achieved without collecting a large amount of data, effectively reducing training costs.

[0071] In one possible implementation, please refer to Figure 3 The video generation method may further include the following steps.

[0072] Step S210: Input the target audio into a pre-trained speech emotion recognition model to obtain the emotion recognition result corresponding to the target audio.

[0073] In this embodiment, when the target audio is input into the pre-trained speech emotion recognition model, the text information corresponding to the target audio can be obtained through speech-to-text and text recognition. The text information is then input into the pre-trained speech emotion recognition model for processing to obtain the emotion recognition result corresponding to the target audio.

[0074] It should be noted that when the text information is input into the pre-trained speech emotion recognition model for processing to obtain the emotion recognition result corresponding to the target audio, the emotion corresponding to the text information can be further determined based on the context of the text information.

[0075] Step S220: Determine the corresponding style transfer matrix based on the emotion recognition result.

[0076] In this embodiment, since different emotions can correspond to different style conversion matrices, the corresponding style conversion matrix can be determined based on the emotion recognition result obtained in step S210.

[0077] In the above design, by processing the target audio to obtain the emotion recognition result, and determining the corresponding style conversion matrix based on the emotion recognition result, the realism of the generated target video can be effectively improved, thereby enhancing the user experience.

[0078] In one possible implementation, due to insufficient training data for the target object, facial expression coefficients related to the target object's facial expression style cannot be directly generated using the trained facial expression parameter recognition model. Therefore, the style transfer matrix can be used to convert the facial expression style of the training object into the facial expression style of the target object. Please refer to... Figure 4 The video generation method may further include the following steps.

[0079] Step S310: Obtain the reference audio and reference video corresponding to the target object.

[0080] In this embodiment, the reference audio and the reference video corresponding to the reference audio can be obtained. The reference audio can be the audio of the real speech output by the target object. The reference video can be the video containing the target object corresponding to the reference audio, and the mouth shape of the target object in the reference video matches the reference audio. The target object can be the object contained in the target video generated in step S140.

[0081] Specifically, when obtaining the reference audio and the reference video corresponding to the target object, only one reference audio and one reference video corresponding to the reference audio can be obtained.

[0082] Step S320: The reference audio is input into the trained facial expression parameter recognition model for processing to obtain third facial expression data, which is related to the facial expression style of the training object.

[0083] In this embodiment, the reference audio obtained in step S310 can be input into the trained facial expression parameter recognition model for processing, and the trained facial expression parameter recognition model can output the third facial expression data. The third facial expression data is related to the facial expression style of the training object, and the third facial expression data may include the mouth shape changes and facial expression details unique to the training object.

[0084] Step S330: Obtain fourth facial expression data based on the reference video, wherein the fourth facial expression data is related to the facial expression style of the target object.

[0085] In this embodiment, the reference video corresponding to the reference audio obtained in step S310 can be processed using three-dimensional facial tracking technology to extract the facial mesh data of the target object in the reference video, thereby obtaining the fourth facial expression data. The fourth facial expression data is the original facial expression data of the target object. The fourth facial expression data is related to the facial expression style of the target object. The fourth facial expression data may include the unique mouth shape changes and facial expression details of the target object.

[0086] Step S340: Compare and analyze the third facial expression data and the fourth facial expression data to obtain the style transfer matrix. The style transfer matrix is ​​used to convert the facial expression style of the training object into the facial expression style of the target object.

[0087] In this embodiment, the style transfer matrix can be obtained by comparing and analyzing the third facial expression data obtained in step S320 and the fourth facial expression data obtained in step S330. The style transfer matrix can be used to convert the facial expression style of the training object into the facial expression style of the target object, thereby obtaining facial expression data related to the facial expression style of the target object. Specifically, the style transfer matrix can store the differences in mouth shape style between the training object and the target object.

[0088] In the above design, even if the number of reference audio and reference video corresponding to the target object is insufficient, facial expression data related to the facial expression style of the target object can still be obtained through the style conversion matrix, thus improving the production efficiency of virtual digital humans.

[0089] In one possible implementation, when comparing and analyzing the third facial expression data and the fourth facial expression data to obtain the style transfer matrix, the style transfer matrix can be obtained by comparing and analyzing the third facial expression data and the fourth facial expression data using the least squares method.

[0090] In this embodiment, the style transformation matrix can be obtained by comparing and analyzing the third facial expression data related to the facial expression style of the training object and the fourth facial expression data related to the facial expression style of the target object using the least squares method. This style transformation matrix can be used to convert the facial expression style of the training object into the facial expression style of the target object. Specifically, the style transformation matrix can store the differences in mouth shape styles between the training object and the target object.

[0091] It should be noted that the method for solving the style transfer matrix is ​​not limited to the least squares method; other optimization techniques can also be used to obtain the style transfer matrix.

[0092] When solving for the style transfer matrix X using the least squares method, the style transfer matrix X can be calculated in the following way:

[0093]

[0094] Among them, A mesh B represents the third facial expression data. mesh This represents the fourth facial expression data, where n represents the number of input images in the trained facial expression parameter recognition model.

[0095] It should be noted that after generating the target video, the synchronization and naturalness of the target video can be checked, and the style transfer matrix can be fine-tuned based on the check results to ensure that the quality and style expression of the target video reach the optimal state.

[0096] In the above design, by applying the style transfer matrix, the facial expression style of the training object can be converted into the facial expression style of the target object, which not only significantly improves the efficiency and scalability of animation production, but also better meets the personalized needs of virtual digital humans.

[0097] In one possible implementation, to correlate the output of the trained facial expression parameter recognition model with the facial expression style of the training subject, it is also necessary to train the output process of the trained facial expression parameter recognition model to extract the facial expression style of the training subject. Please refer to... Figure 5 The video generation method may further include the following steps.

[0098] Step S410: Obtain the training dataset corresponding to the training object. The training dataset includes multiple sample audios and multiple sample videos corresponding to the sample audios.

[0099] In this embodiment, the training object can be any object different from the target object. The training dataset can be obtained by collecting videos, recordings, and films related to the training object. The training dataset can include multiple sample audios and multiple sample videos corresponding to the sample audios. The sample audios can be real audio output by the training object, and the sample videos can be videos containing the training object corresponding to the sample audios, and the lip movements of the training object contained in the sample audios match the sample audios.

[0100] Since there is a large amount of training data corresponding to the training object, by obtaining the training dataset corresponding to the training object, it is convenient to train the output process of the trained facial expression parameter recognition model, thereby extracting the facial expression style of the training object.

[0101] The inventors discovered that when training a pre-trained facial expression parameter recognition model using the training dataset, the sample audio in the training dataset is delayed relative to the sample video, causing them to be out of sync. This can lead to a mismatch between audio and video data during training, affecting the model's accuracy. Therefore, after obtaining the training dataset corresponding to the training object, it is necessary to synchronize the sample audio and sample video in the training dataset. Specifically, the sample audio can be input into an audio-video alignment model to obtain a first facial expression parameter sequence, and the sample video corresponding to the sample audio can be detected to obtain a second facial expression parameter sequence. By comparing the first and second facial expression parameter sequences, the number of delayed frames can be obtained. This allows for adjustment of the time track of the sample audio based on the number of delayed frames, achieving synchronization between the sample audio and sample video in the training data and ensuring facial animation accuracy.

[0102] Step S420: Obtain facial data corresponding to the training object based on the sample video.

[0103] In this embodiment, the sample video corresponding to the sample audio obtained in step S410 can be processed to obtain facial data corresponding to the training object.

[0104] Step S430: Input the sample audio and the facial data into the trained facial expression parameter recognition model for processing to obtain the fifth facial expression data.

[0105] In this embodiment, the sample audio obtained in step S410 and the facial data obtained in step S420 can be input into a trained expression parameter recognition model for processing to obtain the fifth facial expression data, which is related to the facial expression style of the training object.

[0106] In the above design, the output process of the trained facial expression parameter recognition model is repeatedly trained using the training dataset. This allows audio input to be mapped to the correct lip-sync animation output, extracts the facial expression style of the training object, correlates the facial expression parameters output by the trained facial expression parameter recognition model with the facial expression style of the training object, and applies the facial expression style of the training object to other virtual digital humans.

[0107] In one possible implementation, in step S420, when obtaining facial data corresponding to the training object based on the sample video, the sample video can be processed by three-dimensional facial tracking technology to obtain facial expression basis coefficients and facial mesh data corresponding to the training object.

[0108] In this embodiment, the sample video corresponding to the sample audio can be processed using three-dimensional facial tracking technology to extract the facial expression basis coefficients and facial mesh data corresponding to the training object in the sample video.

[0109] In step S430, when the sample audio and the facial data are input into the trained expression parameter recognition model for processing to obtain the fifth facial expression data, the sample audio, the facial expression basis coefficients, and the facial grid data can be input into the trained expression parameter recognition model for processing to obtain the fifth facial expression data.

[0110] In this embodiment, the sample audio, the facial expression base coefficients, and the facial mesh data can be input into a trained expression parameter recognition model for processing to obtain the fifth facial expression data, which is related to the facial expression style of the training object.

[0111] Based on the same inventive concept, this embodiment also provides an electronic device 700, which can be as follows: Figure 1 Please refer to server 100 shown. Figure 6 , Figure 6 This is a block diagram illustrating the electronic device 700. The electronic device 700 includes a video generation device 730, a computer-readable storage medium 720, and a processor 710.

[0112] The computer-readable storage medium 720 and the processor 710 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The video generation device 730 includes multiple software function modules that can be stored in the computer-readable storage medium 720 in the form of software or firmware or embedded in the operating system (OS) of the video generation device 730. The processor 710 is used to execute the executable modules stored in the computer-readable storage medium 720, such as the software function modules and computer programs included in the video generation device 730.

[0113] The computer-readable storage medium 720 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The computer-readable storage medium 720 is used to store a program, which the processor 710 executes upon receiving an execution instruction.

[0114] The processor 710 may be an integrated circuit chip with signal processing capabilities. The processor 710 can be a general-purpose processor 710, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor 710 can be a microprocessor 710, or any conventional processor 710, etc.

[0115] Please refer to Figure 7 This application also provides a video generation apparatus 730. The video generation apparatus 730 includes multiple functional modules that can be stored in software form in a computer-readable storage medium 720. Functionally, the video generation apparatus 730 may include a receiving module 731, a processing module 732, a style adjustment module 733, and a generation module 734. Wherein:

[0116] The receiving module 731 can be used to acquire the target audio.

[0117] In this embodiment, the receiving module 731 can be used to perform... Figure 2 For a detailed description of the receiving module 731, please refer to the description of step S110 shown.

[0118] The processing module 732 can be used to input the target audio into a trained facial expression parameter recognition model for processing to obtain first facial expression data, wherein the first facial expression data is related to the facial expression style of the training object.

[0119] In this embodiment, the processing module 732 can be used to execute... Figure 2 For a detailed description of the processing module 732, please refer to the description of step S120 shown.

[0120] The style adjustment module 733 can be used to input the first facial expression data into a pre-configured style conversion matrix for adjustment, and obtain the adjusted second facial expression data, which is related to the facial expression style of the target object.

[0121] In this embodiment, the style adjustment module 733 can be used to perform... Figure 2 For a detailed description of the style adjustment module 733, please refer to the description of step S130 shown.

[0122] The generation module 734 can be used to generate a target video that describes the target audio based on the second facial expression data.

[0123] In this embodiment, the generation module 734 can be used to execute... Figure 2 For a detailed description of the generation module 734, please refer to the description of step S140 shown.

[0124] In summary, the video generation method, apparatus, and electronic device provided in this application, by processing target audio input into a trained facial expression parameter recognition model, obtains first facial expression data related to the facial expression style of the training object. Then, by inputting the first facial expression data into a style transfer matrix for adjustment, second facial expression data related to the facial expression style of the target object is obtained. This enables style transfer between the target object and the training object, significantly saving time and cost in the production of virtual digital human facial animation. Furthermore, through accurate style transfer, the natural realism of the virtual digital human and the user experience can be improved. In addition, style transfer of the target object can be achieved without collecting large amounts of data, effectively reducing training costs.

[0125] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0126] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for generating digital human videos, characterized in that, The method includes: Obtain the target audio; The target audio is input into a trained facial expression parameter recognition model for processing to obtain first facial expression data, which is related to the facial expression style of the training object. The first facial expression data is input into a pre-configured style transfer matrix for adjustment to obtain the adjusted second facial expression data, which is related to the facial expression style of the target object. Generate a target video that describes the target audio based on the second facial expression data; The method further includes: Obtain reference audio and reference video corresponding to the target object; the lip shape of the target object contained in the reference video is matched with the reference audio; The reference audio is input into the trained facial expression parameter recognition model for processing to obtain third facial expression data. The third facial expression data is related to the facial expression style of the training object. The third facial expression data includes mouth shape changes and facial expression details unique to the training object. The facial mesh data of the target object in the reference video is extracted to obtain fourth facial expression data, which is related to the facial expression style of the target object; the fourth facial expression data includes the unique mouth shape changes and facial expression details of the target object. The third facial expression data and the fourth facial expression data are compared and analyzed to obtain the style transfer matrix, which is used to convert the facial expression style of the training object into the facial expression style of the target object.

2. The digital human video generation method according to claim 1, characterized in that, Different style transfer matrices correspond to different emotions, and the method further includes: The target audio is input into a pre-trained speech emotion recognition model to obtain the emotion recognition result corresponding to the target audio. The corresponding style transfer matrix is ​​determined based on the emotion recognition results.

3. The digital human video generation method according to claim 1, characterized in that, The step of comparing and analyzing the third facial expression data and the fourth facial expression data to obtain the style transfer matrix includes: The style transfer matrix is ​​obtained by comparing and analyzing the third and fourth facial expression data using the least squares method.

4. The digital human video generation method according to claim 3, characterized in that, The style transfer matrix X is calculated in the following way: Among them, A mesh B represents the third facial expression data. mesh This represents the fourth facial expression data, where n represents the number of input images in the trained facial expression parameter recognition model.

5. The digital human video generation method according to claim 1, characterized in that, The method further includes: Obtain the training dataset corresponding to the training object, wherein the training dataset includes multiple sample audios and multiple sample videos corresponding to the sample audios; Facial data corresponding to the training object is obtained from the sample video. The sample audio and facial data are input into a trained facial expression parameter recognition model for processing to obtain the fifth facial expression data.

6. The digital human video generation method according to claim 5, characterized in that, The step of obtaining facial data corresponding to the training object based on the sample video includes: The sample video is processed using 3D facial tracking technology to obtain facial expression basis coefficients and facial mesh data corresponding to the training object; The step of inputting the sample audio and the facial data into a trained facial expression parameter recognition model for processing to obtain the fifth facial expression data includes: The sample audio, the facial expression basis coefficients, and the facial mesh data are input into a trained expression parameter recognition model for processing to obtain the fifth facial expression data.

7. A video generation apparatus, characterized in that, include: The receiving module is used to acquire the target audio; The processing module is used to input the target audio into a trained facial expression parameter recognition model for processing to obtain first facial expression data, wherein the first facial expression data is related to the facial expression style of the training object; The style adjustment module is used to input the first facial expression data into a pre-configured style conversion matrix for adjustment, so as to obtain the adjusted second facial expression data, which is related to the facial expression style of the target object. The generation module is used to generate a target video of the target object expressing the target audio based on the second facial expression data; The style adjustment module is also used to obtain reference audio and reference video corresponding to the target object; The mouth shape of the target object in the reference video is matched with the reference audio; the reference audio is input into a trained facial expression parameter recognition model for processing to obtain third facial expression data, which is related to the facial expression style of the training object; The third facial expression data includes mouth shape changes and facial expression details unique to the training subject; the fourth facial expression data is obtained by extracting the facial mesh data of the target subject from the reference video, and the fourth facial expression data is related to the facial expression style of the target subject. The fourth facial expression data includes mouth shape changes and facial expression details unique to the target object; the third facial expression data and the fourth facial expression data are compared and analyzed to obtain the style conversion matrix, which is used to convert the facial expression style of the training object into the facial expression style of the target object.

8. An electronic device, characterized in that, include: Memory, used to store one or more programs; A processor, when the one or more programs are executed by the processor, implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, characterized in that, when the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images