Video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring audio data for generating a character video, target character image data and reference video data; separating target audio with character acoustic features from the audio data; motion parameters are generated based on the target audio and the target person image data, and initial face motion parameters of the target person are obtained; performing correction processing on the initial face motion parameter based on the reference video data to obtain a target face motion parameter; on the basis of the target audio, key postures are generated through a preset figure posture generation model, and figure posture motion parameters of the target figure are obtained; and performing video generation based on the target face motion parameter and the figure posture motion parameter to obtain a figure video of the target figure. The generation accuracy of the figure video can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, applicable to the field of financial technology, and in particular to a video generation method and device, an electronic device and a storage medium. Background Art

[0002] With the rapid development of computer graphics and artificial intelligence technology, speech-driven character speaking video generation technology has been widely used in the field of video processing. The core goal of speech-driven character speaking video generation technology is to generate more realistic character videos by combining audio data with the character's facial and body animation, so that virtual characters or real people can express voice information in a natural form.

[0003] However, in the related art, when generating speaking videos of people based on voice, it mainly relies on 2D motion fields or facial landmarks to drive the generation of head speaking animations. This method is often difficult to accurately capture human body postures and facial expressions, resulting in the generated video content appearing unrealistic, and there is a problem of low accuracy in the generation of speaking videos of people. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a video generation method and device, an electronic device and a storage medium, aiming to improve the generation accuracy of character videos.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a video generation method, the method comprising:

[0006] Acquire audio data, target person image data, and reference video data for generating a person video;

[0007] Separating target audio with acoustic features of a person from the audio data;

[0008] Generating motion parameters based on the target audio and the target person image data to obtain initial facial motion parameters of the target person;

[0009] Correcting the initial face motion parameters based on the reference video data to obtain target face motion parameters;

[0010] Based on the target audio, key posture generation is performed through a preset character posture generation model to obtain character posture motion parameters of the target character;

[0011] Video generation is performed based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person.

[0012] In some embodiments, separating the target audio with the acoustic features of the person from the audio data includes:

[0013] Segmenting the audio data to obtain multiple candidate audio segments;

[0014] For each of the candidate audio segments, extract the audio features of the candidate audio segment, and perform recognition processing on the audio features based on a preset audio recognition model to obtain an audio category of the candidate audio segment, where the audio category is used to indicate that the candidate audio segment contains a person acoustic feature, or is used to indicate that the candidate audio segment does not contain a person acoustic feature;

[0015] Determine the candidate audio segment whose audio category is that the candidate audio segment contains the acoustic features of the person as the target audio segment;

[0016] Multiple target audio clips are spliced ​​in time sequence to obtain the target audio.

[0017] In some embodiments, the initial facial motion parameters include character expression parameters;

[0018] The step of generating motion parameters based on the target audio and the target person image data to obtain initial face motion parameters of the target person includes:

[0019] An audio encoder based on a preset expression parameter generation model encodes the target audio to obtain audio encoding features, wherein the expression parameter generation model further includes a linear layer;

[0020] Performing expression recognition on the audio coding features based on the linear layer to obtain an expression category corresponding to the target audio;

[0021] According to the expression category, the character expression parameter is determined from a plurality of candidate expression parameters.

[0022] In some embodiments, the initial facial motion parameters include head posture parameters;

[0023] The step of generating motion parameters based on the target audio and the character image data to obtain initial face motion parameters corresponding to the target audio includes:

[0024] Extracting style features of the target audio based on a preset head posture encoder to obtain audio style features;

[0025] The head posture parameter is selected from preset posture style parameters based on the audio style feature.

[0026] In some embodiments, the correcting the initial facial motion parameters based on the reference video data to obtain target facial motion parameters includes:

[0027] Determining the audio duration of the target audio;

[0028] Based on the audio duration, the reference video data is intercepted to obtain a reference video segment, wherein the segment duration of the reference video segment is equal to the audio duration;

[0029] Extracting facial features from the reference video clip to obtain reference facial feature parameters;

[0030] The initial facial motion parameters are corrected based on the reference facial feature parameters to obtain the target facial motion parameters.

[0031] In some embodiments, the step of generating key postures based on the target audio by using a preset character posture generation model to obtain character posture motion parameters of the target character includes:

[0032] Extracting features of the target audio based on the character posture generation model to obtain audio content features;

[0033] Determining a character posture parameter corresponding to the audio content feature based on a preset key posture dictionary in the character posture generation model and the audio content feature;

[0034] The character posture parameters are corrected to obtain the character posture motion parameters.

[0035] In some embodiments, the step of generating a video based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person includes:

[0036] Mapping the target face motion parameters to a three-dimensional space to obtain the three-dimensional face motion parameters;

[0037] Generating a video based on the character posture motion parameters and the three-dimensional motion parameters of the face to obtain three-dimensional character video data;

[0038] The three-dimensional character video data is rendered to obtain the character video.

[0039] To achieve the above objective, a second aspect of an embodiment of the present application provides a video generating device, the video generating device comprising:

[0040] An acquisition unit, used to acquire audio data, target person image data, and reference video data for generating a person video;

[0041] A separation unit, used to separate target audio with acoustic features of a person from the audio data;

[0042] A first generating unit, configured to generate motion parameters based on the target audio and the target person image data, to obtain initial face motion parameters of the target person;

[0043] A correction unit, configured to perform correction processing on the initial face motion parameters based on the reference video data to obtain target face motion parameters;

[0044] A second generating unit is used to generate key postures based on the target audio by using a preset character posture generation model to obtain character posture motion parameters of the target character;

[0045] The third generating unit is used to generate a video based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video generation method described in the first aspect is implemented.

[0047] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video generation method described in the first aspect is implemented.

[0048] The video generation method and device, electronic device and storage medium proposed in the present application can effectively reduce noise interference in different scenes and improve the quality of the final generated character video by acquiring audio data, target character image data and reference video data for generating character video; separating target audio with character acoustic characteristics from audio data. Furthermore, based on the target audio and target character image data, action parameters are generated to obtain the initial face motion parameters of the target character, the initial face motion parameters are corrected based on the reference video data to obtain the target face motion parameters, and based on the target audio, key postures are generated through a preset character posture generation model to obtain the character posture motion parameters of the target character. This method can generate the expression, head movement and body posture of the target character in the character video separately, and improve the authenticity and coordination of the character full body animation in the character video. At the same time, the reference video data is introduced to correct the character motion parameters, enhance the accuracy and controllability of the generated character video, and eliminate inconsistent and unnatural character content. Finally, based on the target face motion parameters and the character posture motion parameters, the video is generated to obtain the character video of the target character, which can improve the video quality and accuracy of the generated character video. This method can make audio-driven character videos more practical in various fields such as digital entertainment and virtual meetings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a video generation method provided in an embodiment of the present application;

[0050] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0051] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0052] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0053] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0054] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0055] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.

[0056] Figure 8 is a schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0057] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] The video generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video generation method in the embodiments of the present application is described.

[0062] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0064] The video generation method provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The video generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video generation method, etc., but is not limited to the above forms.

[0065] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0067] Figure 1 is an optional flowchart of the video generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0068] Step S101, obtaining audio data, target person image data, and reference video data for generating a person video;

[0069] Step S102, separating target audio with acoustic features of a person from the audio data;

[0070] Step S103, generating motion parameters based on the target audio and the target person image data to obtain initial face motion parameters of the target person;

[0071] Step S104, correcting the initial face motion parameters based on the reference video data to obtain target face motion parameters;

[0072] Step S105, based on the target audio, key posture generation is performed through a preset character posture generation model to obtain character posture motion parameters of the target character;

[0073] Step S106, generating a video based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person.

[0074] Steps S101 to S106 shown in the embodiment of the present application, by acquiring audio data, target person image data, and reference video data for generating a person video; separating the target audio with the person acoustic characteristics from the audio data, can effectively reduce noise interference in different scenes and improve the quality of the person video finally generated. Further, based on the target audio and target person image data, action parameters are generated to obtain the initial face motion parameters of the target person, the initial face motion parameters are corrected based on the reference video data to obtain the target face motion parameters, and based on the target audio, key postures are generated through a preset person posture generation model to obtain the person posture motion parameters of the target person. This method can generate the expression, head movement and body posture of the target person in the person video separately, and improve the authenticity and coordination of the person full body animation in the person video. At the same time, the reference video data is introduced, and the person motion parameters can be corrected, the accuracy and controllability of the generated person video can be enhanced, and inconsistent and unnatural person content can be eliminated. Finally, based on the target face motion parameters and the person posture motion parameters, the video is generated to obtain the person video of the target person, which can improve the video quality and accuracy of the generated person video. This method can make audio-driven character videos more practical in various fields such as digital entertainment and virtual meetings.

[0075] In step S101 of some embodiments, audio data, target person image data, and reference video data for generating a person video are acquired.

[0076] The character video refers to a video to be generated that contains the character image of the target character. The target character in the character video can be a real person or a virtual character without restriction.

[0077] Audio data refers to the audio that describes the target person's changes in movements and expressions.

[0078] The target person image data is used to represent the image of the target person in the form of an image. For example, the overall image and facial image of the target person belong to the target person image data.

[0079] The reference video data refers to a video of a person image that is not a target person and that matches the audio data.

[0080] When this embodiment is implemented, when obtaining audio data, video information and transcription can be obtained from common online video platforms, so as to obtain audio data describing the facial changes, head movements, and body postures of the target person; or a network interface (Web API) can be used to collect audio data containing the speaker's speech content and movement changes as the audio data required by the embodiment of this application.

[0081] Similarly, when acquiring the image data of the target person, an image that can reflect the overall character features and facial features of the target person can be retrieved from the image database as the image data of the target person with authorization.

[0082] Furthermore, when obtaining reference video data, a preset code framework can be used to obtain link data corresponding to the video element in the web page, and the link data can be used as reference video data. Alternatively, a video that matches the character actions and expressions described in the audio data can be downloaded from a public video website as reference video data.

[0083] In step S102 of some embodiments, target audio with acoustic features of a person is separated from audio data.

[0084] The person acoustic features are used to indicate the acoustic characteristics of the speaking object in the audio data.

[0085] The target audio refers to the audio data that contains the acoustic characteristics of the person.

[0086] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S204:

[0087] Step S201, segmenting the audio data to obtain a plurality of candidate audio segments;

[0088] Step S202: for each candidate audio segment, extract the audio features of the candidate audio segment, and perform recognition processing on the audio features based on a preset audio recognition model to obtain the audio category of the candidate audio segment;

[0089] Step S203, determining the candidate audio segment whose audio category is the candidate audio segment containing the acoustic features of the person as the target audio segment;

[0090] Step S204: splice multiple target audio clips in time sequence to obtain target audio.

[0091] Steps S201 to S204 are described in detail below.

[0092] In step S201 of some embodiments, the audio data may be segmented according to a preset time length, and the audio data may be divided into a plurality of small segments having a duration consistent with the time length, and each small segment may be used as a candidate audio segment.

[0093] The multiple candidate audio segments do not overlap or repeat with each other, and each candidate audio segment corresponds to a different part of the audio data.

[0094] In step S202 of some embodiments, the audio feature is used to indicate feature information contained in the candidate audio segment. The audio recognition model refers to a neural network model that can identify whether the input audio contains a person's acoustic feature. The audio category is used to indicate that the candidate audio segment contains a person's acoustic feature, or to indicate that the candidate audio segment does not contain a person's acoustic feature.

[0095] When this embodiment is implemented specifically, first, for each candidate audio segment, the candidate audio segment is denoised and the volume is normalized to obtain a processed candidate audio segment to improve the accuracy and robustness of feature extraction. Then, the Mel frequency cepstral coefficients are extracted from the processed candidate audio segment to obtain the audio features of the candidate audio segment. Further, the audio features of the candidate audio segment are input into a preset audio recognition model, and the audio recognition model converts the audio features into a one-dimensional feature vector, and the one-dimensional feature vector is subjected to binary classification based on a preset function to obtain a first probability that the candidate audio segment contains the acoustic features of the person, and a second probability that the candidate audio segment does not contain the acoustic features of the person, wherein the sum of the first probability and the second probability is 1. Further, if the first probability is greater than the second probability, the audio category of the candidate audio segment is determined as the candidate audio segment contains the acoustic features of the person; if the first probability is less than the second probability, the audio category of the candidate audio segment is determined as the candidate audio segment does not contain the acoustic features of the person.

[0096] In step S203 of some embodiments, according to the audio categories of the candidate audio segments, the candidate audio segments with the audio category of candidate audio segments containing human acoustic features are determined as target audio segments to achieve human voice separation of the audio data.

[0097] In step S204 of some embodiments, first, the time point of each target audio segment in the audio data is obtained, wherein the time point of the start point of the target audio segment in the audio data can be used as the time point of the entire target audio segment in the audio data. Then, according to the early and late time points, multiple target audio segments are spliced ​​in the order of time points from early to late in time sequence to form a complete audio sequence, thereby obtaining the target audio.

[0098] The advantage of this embodiment is that it takes into account the problem that noise information in audio data may interfere with the generation of character videos. At the audio segment level, it identifies whether each candidate audio segment contains vocal features, and only splices the candidate audio segments containing character acoustic features as target audio segments to obtain target audio, which is used to generate character videos. This method can relatively achieve voice separation of audio data, eliminate the interference of noise information in audio data, improve the audio quality of target audio, and is conducive to generating high-quality character videos based on target audio.

[0099] In step S103 of some embodiments, motion parameters are generated based on the target audio and the target person image data to obtain initial facial motion parameters of the target person.

[0100] The initial face motion parameters are used to describe the facial change characteristics of the target person in the person video to be generated in parameter form.

[0101] The initial facial motion parameters include character expression parameters and head posture parameters.

[0102] The character expression parameters are used to describe the expression characteristics of the target character in the character video to be generated in the form of parameters.

[0103] The head pose parameters are used to describe the head pose characteristics of the target person in the character video to be generated in the form of parameters.

[0104] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S303:

[0105] Step S301, encoding the target audio based on the audio encoder of the preset expression parameter generation model to obtain audio encoding features;

[0106] Step S302, performing expression recognition on the audio coding features based on a linear layer to obtain an expression category corresponding to the target audio;

[0107] Step S303, determining a character expression parameter from a plurality of candidate expression parameters according to the expression category.

[0108] Steps S301 to S303 are described in detail below.

[0109] In step S301 of some embodiments, the expression parameter generation model refers to a neural network model that can generate the emotional information of the speaker in the audio data based on the speech content and speech characteristics of the speaker in the audio data. The expression parameter generation model generally takes the audio data as input and takes the expression category of the speaker in the audio data as output. Among them, the expression parameter generation model includes a linear layer and an audio encoder. The audio encoder is used to map the input audio data from the time domain space to the frequency domain space and represent the audio data in the form of a vector. The linear layer is used to predict the expression category of the speaker in the audio data based on the audio representation of the input audio data in the latent vector space.

[0110] When this embodiment is implemented specifically, first, the target audio is input into an audio encoder of a preset expression parameter generation model, and the audio encoder encodes the target audio, encodes the audio features in the target audio into audio embedding, and maps the target audio from the time domain space to the latent vector space to obtain audio coding features.

[0111] In step S302 of some embodiments, the expression category is used to indicate the expression characteristics that the target person described by the target audio should have, wherein the expression category includes but is not limited to happiness, anger, frustration, sadness, and the like.

[0112] When this embodiment is implemented, facial expression recognition is performed on audio coding features based on a linear layer, the linear layer first reduces the dimension of the audio coding features into one-dimensional audio features, and then uses the softmax function of the linear layer to perform probability calculation on the one-dimensional audio features, creating a probability distribution of the target audio on multiple candidate facial expression categories, and using the probability vectors corresponding to each candidate facial expression category in the probability distribution as the probability value that the facial expression information of the speaker in the target audio belongs to the candidate facial expression category. Further, the candidate facial expression category with the largest probability value is determined as the facial expression category corresponding to the target audio.

[0113] In step S303 of some embodiments, in order to improve the efficiency of acquiring character expression parameters, when determining character expression parameters based on expression categories, a series of candidate expression parameters corresponding to each candidate expression category are set. After determining the expression category corresponding to the target audio, a series of candidate expression parameters corresponding to the candidate expression category that is consistent with the expression category corresponding to the target audio are used as the final character expression parameters in the form of a table lookup.

[0114] The advantage of this embodiment is that, in determining the character expression parameters corresponding to the audio data, the recognition of the expression category of the target audio using the expression parameter generation model is taken into account, which can improve the efficiency and accuracy of obtaining the character expression parameters. At the same time, the character expression parameters are obtained by looking up the table based on the expression category, and the character expression parameters of different characters in the same expression category are the same, which can also improve the standardization and uniformity of the obtained character expression parameters to a certain extent.

[0115] See also Figure 4 In some embodiments, step S103 may include but is not limited to steps S401 to S402:

[0116] Step S401, extracting style features of the target audio based on a preset head posture encoder to obtain audio style features;

[0117] Step S402: Filter out head posture parameters from preset posture style parameters based on the audio style features.

[0118] Steps S401 to S402 are described in detail below.

[0119] In step S401 of some embodiments, the head posture encoder is used to extract the style information of the emotion conveyed by the audio data from the input audio data. The audio style feature is used to indicate the style characteristics of the input audio data. The audio style feature is used to make the facial expression style of the target person in the generated person video consistent with the emotion information conveyed in the audio data.

[0120] When this embodiment is implemented specifically, first, the target audio is input into a preset head posture encoder, and the head posture encoder extracts style features of the target audio, extracts the emotional style information expressed by the speaker in speaking from the target audio, and obtains audio style features.

[0121] In step S402 of some embodiments, in order to improve the efficiency of obtaining head posture parameters, when filtering head posture parameters based on audio style features, a series of posture style parameters corresponding to various audio style features are set. Therefore, a series of preset posture style parameters corresponding to audio style features consistent with the audio style features of the target audio can be used as the final head posture parameters in the form of a table lookup.

[0122] The benefit of this embodiment is that by extracting the audio style features of the target audio and using them to drive the head posture parameters, a more natural and realistic character video can be generated. The characters in the character video can not only imitate the speaker's voice rhythm and intonation, but also express corresponding emotions and styles, which can improve the naturalness and authenticity of the generated character video. At the same time, this method realizes automated audio style feature extraction and head posture parameter screening, which can effectively reduce the need for manual adjustment of character video animation and improve the efficiency of character video animation production.

[0123] In step S104 of some embodiments, the initial facial motion parameters are corrected based on the reference video data to obtain target facial motion parameters.

[0124] The target face motion parameters are used to describe the facial changes of the target person in the person video to be generated in the form of parameters.

[0125] See also Figure 5 In some embodiments, step S104 may include but is not limited to steps S501 to S504:

[0126] Step S501, determining the audio duration of the target audio;

[0127] Step S502, performing video interception on the reference video data based on the audio duration to obtain a reference video segment;

[0128] Step S503, extracting facial features from the reference video clip to obtain reference facial feature parameters;

[0129] Step S504: Correct the initial facial motion parameters based on the reference facial feature parameters to obtain target facial motion parameters.

[0130] Steps S501 to S504 are described in detail below.

[0131] In step S501 of some embodiments, the audio duration is used to indicate the duration of the target audio from the beginning to the end.

[0132] In the specific implementation of this embodiment, a timer is used to perform timing operation on the target audio, and the duration of the target audio from the beginning to the end is recorded to obtain the audio duration of the target audio.

[0133] In step S502 of some embodiments, the reference video segment refers to a portion of video data extracted from the reference video data, wherein the segment length of the reference video segment is equal to the audio length of the target audio.

[0134] In the specific implementation of this embodiment, first, according to the starting part of the target audio, the video part corresponding to the reference video data and the target audio is located. Then, starting from the video time point where the starting point of the reference video data coincides with the starting point of the target audio, a video segment equal to the audio duration is cut out from the reference video data as the reference video segment.

[0135] In step S503 of some embodiments, the reference facial feature parameters are used to indicate the facial motion of a person in a reference video clip, wherein the person in the reference video clip is different from the target person.

[0136] In the specific implementation of this embodiment, a face recognition and positioning model is used to locate the face position in each frame of the reference video clip, and then the eye and head movement information of the located face is extracted to obtain reference face feature parameters.

[0137] In step S504 of some embodiments, since the target face operation parameters include multiple motion parameter items. For example, the multiple motion parameter items include eye motion parameters, head motion parameters, lip motion parameters, eyebrow motion parameters, etc. Based on this, first, the first parameter value of each motion parameter item is obtained in the reference face feature parameters, and the second parameter value of each motion parameter item is obtained in the initial face motion parameters, wherein, since the reference face feature parameters are extracted based on the eye and head motion information of the face of the reference video clip, the first parameter values ​​of some motion parameter items are empty. Further, when the initial face motion parameters are corrected based on the reference face feature parameters, when the first parameter value of the motion parameter item is not empty, the first parameter value of the motion parameter item is used as the parameter value of the motion parameter item in the target face operation parameters; when the first parameter value of the motion parameter item is empty, the second parameter value of the motion parameter item is used as the parameter value of the motion parameter item in the target face operation parameters, thereby obtaining the complete target face operation parameters.

[0138] The benefit of this embodiment is that the reference video data is captured according to the audio duration of the target audio, and only the reference facial feature parameters contained in the reference video clip are used to correct the initial facial motion parameters. This method can freely control the influence of the reference video data on the generated facial video. At the same time, the introduction of the reference video correction mechanism can effectively enhance the accuracy and controllability of the generated character video, and can eliminate the phenomenon that the generated character video is inconsistent with expectations or unnatural, thereby improving the quality of the character video animation.

[0139] In step S105 of some embodiments, key posture generation is performed based on the target audio through a preset character posture generation model to obtain character posture motion parameters of the target character.

[0140] The character posture motion parameters are used to indicate the overall motion changes of the target character (role) in the character video to be generated.

[0141] See also Figure 6 In some embodiments, step S105 may include but is not limited to steps S601 to S603:

[0142] Step S601, extracting features of the target audio based on the character posture generation model to obtain audio content features;

[0143] Step S602, determining a character posture parameter corresponding to the audio content feature based on a preset key posture dictionary in the character posture generation model and the audio content feature;

[0144] Step S603, correcting the character posture parameters to obtain character posture motion parameters.

[0145] Steps S601 to S603 are described in detail below.

[0146] In step S601 of some embodiments, the character posture generation model may be a neural network model based on a long short-term memory network structure. The audio content feature is used to indicate the audio content information of the target video. The audio content information mainly reflects the speech content of the speaker in the target video.

[0147] In the specific implementation of this embodiment, the target audio is input into the character posture generation model, and the character posture generation model maps each target audio segment in the target audio to the latent vector space to obtain the target audio segment features. Then, attention calculation is performed on each target audio segment feature to obtain the audio attention calculation result corresponding to each target audio segment, and the audio attention calculation result is feed-forward processed to obtain the audio segment content feature corresponding to each target audio segment. Finally, the audio segment content features corresponding to multiple target audio segments are spliced ​​into audio content features.

[0148] In step S602 of some embodiments, the preset key gesture dictionary is used to indicate the gesture and action features of the speaker that match various audio contents. The character gesture parameters are used to indicate the gesture, action, etc. of the speaker in the target audio in the form of parameters.

[0149] Among them, the preset key posture dictionary includes multiple candidate posture action features.

[0150] In the specific implementation of this embodiment, first, the audio content features are matched and calculated with each candidate posture action feature of the preset key posture dictionary using a similarity algorithm, etc., to obtain the feature matching degree of each candidate posture action feature and the audio content feature. Then, the feature parameters corresponding to the candidate posture action features with a feature matching degree greater than a preset threshold are selected as the character posture parameters corresponding to the audio content features.

[0151] In step S603 of some embodiments, the character posture parameters can be corrected using an animation editing tool, and the generated character posture parameters can be edited and adjusted to correct unnatural or unexpected movements and obtain character posture motion parameters.

[0152] The benefit of this embodiment is that by extracting features from the target audio based on the character posture generation model, the audio content features can be captured more accurately, thereby generating a more natural and realistic character animation. This method can match the character movements in the generated character video animation with the audio content, thereby improving the naturalness of the character video. Furthermore, by presetting the key posture dictionary and audio content features to determine the character posture parameters, it is ensured that the character movements in the animation are synchronized with the audio rhythm and content, and a high degree of fit is achieved in terms of lip shape, expression, etc., which is conducive to improving the overall quality of the generated character video animation.

[0153] See also Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S703:

[0154] Step S701, mapping the target face motion parameters to a three-dimensional space to obtain the three-dimensional face motion parameters;

[0155] Step S702, generating a video based on the character's posture motion parameters and the three-dimensional motion parameters of the face to obtain three-dimensional character video data;

[0156] Step S703, rendering the three-dimensional character video data to obtain a character video.

[0157] Steps S701 to S703 are described in detail below.

[0158] In step S701 of some embodiments, the target character's running parameters corresponding to each frame of the character video are mapped to the three-dimensional space, and the mapping of the latent vector to the three-dimensional space motion parameters is realized to obtain the three-dimensional motion parameters of the face.

[0159] In step S702 of some embodiments, the three-dimensional character video data refers to video content generated by three-dimensional modeling and animation technology, wherein the characters and objects contained in the three-dimensional character video data all exist in three-dimensional form.

[0160] In the specific implementation of this embodiment, an optical flow generator is used to generate video based on character posture motion parameters and three-dimensional face motion parameters, and the motion information and optical flow of objects in each video frame in the character video to be generated are estimated to generate three-dimensional character video data.

[0161] In addition, when generating videos based on character posture motion parameters and three-dimensional face motion parameters, three-dimensional animation production software can be used to combine the character posture motion parameters and the three-dimensional face motion parameters to perform key frame animation on the three-dimensional character model to obtain three-dimensional character video data; or in a game engine (such as Unity, Unreal Engine), three-dimensional character video data can be generated in real time based on the input character posture motion parameters and three-dimensional face motion parameters.

[0162] In step S703 of some embodiments, first, a super-resolution renderer is used to render the three-dimensional character video data to obtain a rendered image sequence. Next, the image sequence is illuminated and material is set to enhance the realism of the character video. At the same time, the rendered image sequence is subjected to operations such as color correction and special effects addition to improve the overall visual effect of the character video to be generated. Finally, the image sequence that has undergone operations such as lighting and material setting, color correction and special effects addition is synthesized into a video format to obtain a character video. Among them, the video format of the character video includes but is not limited to MP4, AVI, etc.

[0163] The advantage of this embodiment is that the three-dimensional character video data is generated from the target face motion parameters and the character posture motion parameters, while taking into account the facial changes and body changes of the target character in the character video to be generated, and this method can improve the three-dimensional character video data. At the same time, when generating and rendering the video, an optical flow generator and a super-resolution renderer are introduced, which can improve the naturalness and coordination of the generated character video, and improve the video quality of the generated character video, so that the character video has better clarity, which can meet the generation requirements of character animation videos in different application scenarios.

[0164] See also Figure 8The embodiment of the present application further provides a video generating device, which can implement the above-mentioned video generating method, and the device includes:

[0165] An acquisition unit 801 is used to acquire audio data, target person image data, and reference video data for generating a person video;

[0166] A separation unit 802 is used to separate the target audio with the acoustic characteristics of the person from the audio data;

[0167] The first generating unit 803 is used to generate motion parameters based on the target audio and the target person image data to obtain the initial face motion parameters of the target person;

[0168] The correction unit 804 is used to correct the initial face motion parameters based on the reference video data to obtain the target face motion parameters;

[0169] The second generating unit 805 is used to generate key postures based on the target audio by using a preset character posture generation model to obtain character posture motion parameters of the target character;

[0170] The third generating unit 806 is used to generate a video based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person.

[0171] In some embodiments, the separation unit 802 is used to:

[0172] Segment the audio data to obtain multiple candidate audio segments;

[0173] For each candidate audio segment, extract the audio features of the candidate audio segment, and perform recognition processing on the audio features based on a preset audio recognition model to obtain an audio category of the candidate audio segment, where the audio category is used to indicate that the candidate audio segment contains the acoustic features of the person, or is used to indicate that the candidate audio segment does not contain the acoustic features of the person;

[0174] Determine a candidate audio segment whose audio category is a candidate audio segment containing a person acoustic feature as a target audio segment;

[0175] Multiple target audio clips are spliced ​​in time sequence to obtain target audio.

[0176] In some embodiments, the initial facial motion parameters include character expression parameters;

[0177] The first generating unit 803 is used for:

[0178] An audio encoder based on a preset expression parameter generation model encodes the target audio to obtain audio encoding features, and the expression parameter generation model also includes a linear layer;

[0179] Perform facial expression recognition on audio encoding features based on the linear layer to obtain the facial expression category corresponding to the target audio;

[0180] According to the expression category, a character expression parameter is determined from a plurality of candidate expression parameters.

[0181] In some embodiments, the initial face motion parameters include head posture parameters;

[0182] The first generating unit 803 is used for:

[0183] Extracting style features of the target audio based on a preset head posture encoder to obtain audio style features;

[0184] Based on the audio style features, head posture parameters are screened out from preset posture style parameters.

[0185] In some embodiments, the correction unit 804 is used to:

[0186] Determine the audio duration of the target audio;

[0187] Based on the audio duration, the reference video data is intercepted to obtain a reference video segment, where the segment duration of the reference video segment is equal to the audio duration;

[0188] Extracting facial features from the reference video clip to obtain reference facial feature parameters;

[0189] The initial face motion parameters are corrected based on the reference face feature parameters to obtain the target face motion parameters.

[0190] In some embodiments, the second generating unit 805 is used to:

[0191] Extract features of the target audio based on the character posture generation model to obtain audio content features;

[0192] Determine character posture parameters corresponding to the audio content features based on a preset key posture dictionary in the character posture generation model and the audio content features;

[0193] The character posture parameters are corrected to obtain the character posture motion parameters.

[0194] In some embodiments, the third generating unit 806 is used to:

[0195] Mapping the target face motion parameters to three-dimensional space to obtain the three-dimensional motion parameters of the face;

[0196] Generate video based on character posture motion parameters and face three-dimensional motion parameters to obtain three-dimensional character video data;

[0197] The three-dimensional character video data is rendered to obtain a character video.

[0198] The specific implementation of the video generating device is basically the same as the specific implementation of the above-mentioned video generating method, and will not be repeated here.

[0199] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned video generation method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0200] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0201] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0202] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video generation method of the embodiment of this application;

[0203] Input / output interface 903, used to implement information input and output;

[0204] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0205] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0206] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0207] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned video generation method.

[0208] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0209] The video generation method, video generation device, electronic device and computer-readable storage medium provided by the embodiment of the present application can effectively reduce noise interference in different scenes and improve the quality of the final generated character video by acquiring audio data, target character image data and reference video data for generating character video; separating the target audio with character acoustic characteristics from the audio data. Furthermore, based on the target audio and target character image data, action parameters are generated to obtain the initial face motion parameters of the target character, and the initial face motion parameters are corrected based on the reference video data to obtain the target face motion parameters, and based on the target audio, key postures are generated through a preset character posture generation model to obtain the character posture motion parameters of the target character. This method can generate the expression, head movement and body posture of the target character in the character video separately, and improve the authenticity and coordination of the character full-body animation in the character video. At the same time, the reference video data is introduced to correct the character motion parameters, enhance the accuracy and controllability of the generated character video, and eliminate inconsistent and unnatural character content. Finally, based on the target person's face motion parameters and the person's posture motion parameters, the target person's character video is generated, which can improve the video quality and accuracy of the generated character video. This method can make audio-driven character videos more practical in various fields such as digital entertainment and virtual meetings.

[0210] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0211] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0212] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0214] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0215] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0216] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0217] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0218] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0219] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0220] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video generation method, characterized in that: The video generation method comprises: Acquire audio data, target person image data, and reference video data for generating a person video; Separating target audio with acoustic features of a person from the audio data; Generating motion parameters based on the target audio and the target person image data to obtain initial facial motion parameters of the target person; Correcting the initial face motion parameters based on the reference video data to obtain target face motion parameters; Based on the target audio, key posture generation is performed through a preset character posture generation model to obtain character posture motion parameters of the target character; Video generation is performed based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person.

2. The method according to claim 1, characterized in that The step of separating the target audio with the acoustic features of the person from the audio data comprises: Segmenting the audio data to obtain multiple candidate audio segments; For each of the candidate audio segments, extract the audio features of the candidate audio segment, and perform recognition processing on the audio features based on a preset audio recognition model to obtain an audio category of the candidate audio segment, where the audio category is used to indicate that the candidate audio segment contains a person acoustic feature, or is used to indicate that the candidate audio segment does not contain a person acoustic feature; Determine the candidate audio segment whose audio category is that the candidate audio segment contains the acoustic features of the person as the target audio segment; Multiple target audio clips are spliced ​​in time sequence to obtain the target audio.

3. The method according to claim 1, characterized in that The initial facial motion parameters include character expression parameters; The step of generating motion parameters based on the target audio and the target person image data to obtain initial face motion parameters of the target person includes: An audio encoder based on a preset expression parameter generation model encodes the target audio to obtain audio encoding features, wherein the expression parameter generation model further includes a linear layer; Performing expression recognition on the audio coding features based on the linear layer to obtain an expression category corresponding to the target audio; According to the expression category, the character expression parameter is determined from a plurality of candidate expression parameters.

4. The method according to claim 3, characterized in that The initial face motion parameters include head posture parameters; The step of generating motion parameters based on the target audio and the character image data to obtain initial face motion parameters corresponding to the target audio includes: Extracting style features of the target audio based on a preset head posture encoder to obtain audio style features; The head posture parameter is selected from preset posture style parameters based on the audio style feature.

5. The method according to claim 1, characterized in that The correcting the initial face motion parameters based on the reference video data to obtain target face motion parameters includes: Determining the audio duration of the target audio; Based on the audio duration, the reference video data is intercepted to obtain a reference video segment, wherein the segment duration of the reference video segment is equal to the audio duration; Extracting facial features from the reference video clip to obtain reference facial feature parameters; The initial facial motion parameters are corrected based on the reference facial feature parameters to obtain the target facial motion parameters.

6. The method according to claim 1, characterized in that The step of generating key postures based on the target audio by using a preset character posture generation model to obtain character posture motion parameters of the target character includes: Extracting features of the target audio based on the character posture generation model to obtain audio content features; Determining a character posture parameter corresponding to the audio content feature based on a preset key posture dictionary in the character posture generation model and the audio content feature; The character posture parameters are corrected to obtain the character posture motion parameters.

7. The method according to any one of claims 1 to 6, characterized in that: The step of generating a video based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person includes: Mapping the target face motion parameters to a three-dimensional space to obtain the three-dimensional face motion parameters; Generating a video based on the character posture motion parameters and the three-dimensional motion parameters of the face to obtain three-dimensional character video data; The three-dimensional character video data is rendered to obtain the character video.

8. A video generating device, characterized in that: The video generating device comprises: An acquisition unit, used to acquire audio data, target person image data, and reference video data for generating a person video; A separation unit, used to separate target audio with acoustic features of a person from the audio data; A first generating unit, configured to generate motion parameters based on the target audio and the target person image data, to obtain initial face motion parameters of the target person; A correction unit, configured to perform correction processing on the initial face motion parameters based on the reference video data to obtain target face motion parameters; A second generating unit is used to generate key postures based on the target audio by using a preset character posture generation model to obtain character posture motion parameters of the target character; The third generating unit is used to generate a video based on the target person's face motion parameters and the person's posture motion parameters to obtain a person video of the target person.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.

Images