Virtual human video generation method and device, electronic equipment and medium
By processing data based on the text to be broadcast and the original video, a precisely matched virtual human video is generated, which solves the problem of mismatch between lip movements and audio, and between facial expressions and emotions in virtual human videos. This achieves natural and vivid virtual human performance and reduces generation costs and technical barriers.
Patent Information
- Application Number
- CN202511311751.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-15
- Publication Date
- 2025-12-05
AI Technical Summary
Existing technologies have poor matching between lip movements and audio, and between facial expressions and emotions when generating virtual human videos. This results in virtual humans appearing stiff and lifeless, and they are prone to falling into the uncanny valley effect. Furthermore, the generation process is costly and inefficient.
By determining the audio data, emotional data, and facial data for the broadcast based on the text to be broadcast and the target user's original video, stable video clips are selected, and a virtual human video is generated using an expression-enhanced speaking face generator, achieving precise synchronization between lip movements and audio, and between facial expressions and emotions.
It improves the realism and vividness of virtual human videos, lowers the threshold and cost of personalized generation, and enhances generation efficiency and applicability.
Smart Images

Figure CN121078290A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Embodiments of the present application relate to the technical field of artificial intelligence, in particular to a virtual human video generation method and device, electronic equipment and medium. BACKGROUND
[0002] With the rapid development of computer technology and artificial intelligence technology, the application value of virtual human is increasingly prominent. In the fields of Internet teaching, broadcasting and hosting, film industry, Internet finance, smart home and the like, virtual human is widely used to replace real people to complete various work, such as making up for the high cost of real instructors, realizing zero-failure broadcasting, completing high-difficulty dangerous actions, providing intelligent customer service and the like, due to the advantages of reducing cost, improving efficiency and breaking through real person limitations.
[0003] Currently, in related technologies, when generating a virtual human video by using a video, only the mouth shape is simulated by driving a facial expression base or a mouth part model vertex, or only facial expression driving is realized by relying on making expression base animation and playing in a random or fixed order, resulting in poor matching degree of the virtual human in the mouth shape and audio, expression and emotion, lack of naturalness, and the digital human appearing dull and rigid in interaction, which is easy to fall into the uncanny valley effect. SUMMARY
[0004] The present application provides a virtual human video generation method, device, electronic equipment and medium to solve the problems in related technologies, which quickly generates a virtual human video with personalized audio, matching emotion and stable and natural facial performance by using a user's original video of a short duration, effectively reduces the threshold and cost of personalized virtual human generation, and improves the realism and liveliness of the virtual human video.
[0005] In a first aspect, the embodiments of the present application provide a virtual human video generation method, which comprises: determining, based on a to-be-broadcast text and an original video of a target user, broadcast audio data and broadcast emotion data of a virtual human corresponding to the target user, the original video being pre-acquired according to a preset time period; determining a stable video segment in the original video, and determining, based on the stable video segment, broadcast face data of the virtual human, the video duration of the stable video segment being greater than or equal to a preset effective duration; and generating a face video of the virtual human based on the broadcast audio data, the broadcast emotion data and the broadcast face data, so as to generate a virtual human video corresponding to the target user according to the face video of the virtual human.
[0006] In a second aspect, the embodiments of the present application provide a virtual human video generation device, which comprises:
[0007] A first determination unit is configured to determine, based on a to-be-broadcast text and an original video of a target user, broadcast audio data and broadcast emotion data of a virtual human corresponding to the target user, the original video being pre-acquired according to a preset time period.
[0008] The second determining unit is configured to determine a stable video segment in the original video, and determine the anchor face data of the virtual person based on the stable video segment, wherein a video duration of the stable video segment is greater than or equal to a preset effective duration.
[0009] The generating unit is configured to generate a face video of the virtual person based on the anchor audio data, the anchor emotion data and the anchor face data, so as to generate the virtual person video corresponding to the target user according to the face video of the virtual person.
[0010] In a third aspect, an electronic device is provided, including a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to run the computer program to perform the method described in any one of the embodiments of the first aspect.
[0011] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are configured to cause a computer to perform the method described in any one of the embodiments of the first aspect.
[0012] In a fifth aspect, a computer program product is provided, including a computer program, and the computer program is configured to perform the method described in any one of the embodiments of the first aspect when executed by a processor.
[0013] The virtual person video generation method provided by the embodiments of the present application determines the anchor audio data and the anchor emotion data of the virtual person based on the to-be-announced text and the original video pre-acquired by the target user in a preset time period, simultaneously screens out a stable video segment with a duration not less than a preset effective duration from the original video and determines the anchor face data based on the stable video segment, and then generates a face video by combining the three types of data to obtain the virtual person video corresponding to the target user. By synergistically fusing the anchor audio, emotion and face data, the precise matching of the lip shape and the audio and the expression and the emotion is realized, the virtual person performs more naturally and vividly, the uncanny valley effect is effectively avoided, personalized generation can be completed only by using a short original video, the recording cost and the technical threshold are greatly reduced, and the efficiency and the applicability of the virtual person video generation are improved.
[0014] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings, and the present application can also be applied to other similar scenarios based on the provided drawings.
[0016] Figure 1 A flow chart of a virtual person video generation method provided by an embodiment of the present application;
[0017] Figure 2 A flow chart of a second virtual person video generation method provided by an embodiment of the present application;
[0018] Figure 3 A schematic diagram of a method for determining a stable video segment provided by an embodiment of the present application;
[0019] Figure 4 A flow chart of a third virtual person video generation method provided by an embodiment of the present application;
[0020] Figure 5 A flow chart of a fourth virtual person video generation method provided by an embodiment of the present application;
[0021] Figure 6 A specific schematic diagram of obtaining a target sound model provided by an embodiment of the present application;
[0022] Figure 7 A specific schematic diagram of obtaining a virtual person video provided by an embodiment of the present application;
[0023] Figure 8 A structural schematic diagram of a virtual person video generation device 800 provided by an embodiment of the present application;
[0024] Figure 9 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0025] The present application will be further described in detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. The described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0026] It should be noted that the "system", "device", "unit" and / or "module" used in the present application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.
[0027] Hereinafter, the terms "first", "second", "third" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include one or more of the features.
[0028] With the rapid development of computer technology, artificial intelligence technology is gradually mature, and the application potential and value of digital virtual human are becoming larger and larger. In order to improve user experience effect and reduce labor cost and risk cost, more and more fields tend to use digital virtual human to replace real people to complete some work. For example, in the fields of Internet teaching such as financial investment education, medical pathology teaching and education course training, virtual human teachers can well make up for the high cost problem of hiring real teachers, and virtual animation teachers are more attractive than real people; In the fields related to broadcasting and hosting, news broadcasting, digital virtual human can achieve zero failure broadcasting without being affected by external environment; In the film industry and multimedia production field, digital virtual human can complete high-difficulty dangerous actions that real people cannot complete; In the fields of Internet finance, smart home, venue and scenic spot, digital virtual human can provide intelligent and efficient real-time response personalized customer service and housekeeper service.
[0029] At present, the virtual human generation technology in the related art is mainly based on the modeling-rendering method in computer graphics. The efficiency of this kind of method is very low, and a large number of technical personnel and artists need to be invested in the early stage to carry out fine three-dimensional modeling, skinning and motion capture, and a large number of computing resources are needed to achieve realistic rendering effect. How to make virtual human be constructed and generated in a more convenient and fast way has become a rigid requirement in many application scenarios.
[0030] Visual representation plays an important role in human speech perception. For example, when watching a movie, if the mouth shape of a character does not correspond to the sound, it will bring an out-of-play feeling to the audience.
[0031] In the related art, the mainstream audio-driven three-dimensional lip movement algorithm mainly targets general models, and simulates the mouth shape when speaking by driving the expression base of the face or the model vertex related to the lip part. In this process, a lot of information including prosody and emotion is lost, and the virtual human can only mechanically show the mouth shape corresponding to the phoneme. Facial expressions are mainly driven by making expression base animation and playing the animation randomly or in a fixed order.
[0032] It can be seen that the lack of understanding of content and emotion in related technologies is an important reason for the digital person to appear dull and stiff when conversing, leading to the digital person easily falling into the uncanny valley effect.
[0033] To solve the above problems, the present application provides a virtual person video generation method, which is based on the to-be-broadcast text and the original video of the target user pre-recorded in the preset time period. First, the personalized audio processing module uses the audio in the original video to train a personalized voice model, generates broadcast audio data highly matched with the voice tone and speaking speed of the target user in combination with the to-be-broadcast text, and acquires corresponding broadcast emotion data by performing emotion classification on the dialogue text (i.e., the to-be-broadcast text) with the aid of a pre-trained emotion classification algorithm. Then, the video self-selection module selects a stable video segment with a time length not less than a preset effective time length (e.g., three minutes) from the original video based on head pose estimation and eye estimation, removes noise data such as user shaking and camera jitter, and extracts facial region key feature point coordinates from the stable video segment, which are smoothed and used as broadcast face data. Finally, in the face generation module, the broadcast audio data, the broadcast emotion data, and the broadcast face data are input into a speaker face generator based on expression enhancement, features are extracted by a face encoder and an audio encoder and spliced with emotion features, a matched face video is generated by a decoder, and the face video is fused into the stable video segment to obtain a virtual person video corresponding to the target user. Through the cooperation of the three, this process realizes accurate synchronization of lip shapes, audio, and expressions and emotions, enables the virtual person to get rid of mechanical stiffness, is more natural and lively, effectively avoids the uncanny valley effect, and significantly improves the efficiency of virtual person video generation and the applicability of applications in various fields.
[0034] Next, a virtual person video generation method provided by the present application will be described in detail with reference to the accompanying drawings.
[0035] Figure 1 A flowchart of a virtual person video generation method provided by an embodiment of the present application is shown. As shown in Figure 1 The virtual person video generation method includes steps 101-103.
[0036] Step 101, based on the to-be-broadcast text and the original video of the target user, determining the broadcast audio data and the broadcast emotion data of the virtual person corresponding to the target user, the original video being pre-acquired in a preset time period.
[0037] In the embodiments of the present application, the present application can extract the voice of the virtual person speaking and the emotion when speaking from the to-be-broadcast content input by the user and the preset original video, laying a foundation for subsequent generation of virtual persons with voice and emotion, and solving the problems of non-personalized voice and lack of emotion of virtual persons in related technologies.
[0038] Specifically, the application can first obtain the unique voice features of the user through the user's original video, and then generate exclusive sounds in combination with the to-be-broadcast text; at the same time, the emotional tendency of the to-be-broadcast text is analyzed to determine the expression emotion that the virtual person needs to match, and finally the "sound-emotion" two-dimensional data is output to ensure that the subsequent performance of the virtual person is highly consistent with the user's characteristics and the content of the text.
[0039] In the embodiments of the application, the to-be-broadcast text refers to the content that the user hopes the virtual person to finally express, that is, the lines / script that the virtual person broadcasts, and is not the audio content (for example, the text of the sales pitch, such as teaching lines, news scripts, customer service scripts, etc.) in the original video.
[0040] The original video of the target user is the user video collected in advance according to a preset time period (for example, 3 minutes), which contains image data (face, limbs) and audio data (user voice) of the user, and is the core data source for extracting the user's personalized features (voice timbre, facial contour). The preset time period is the standard for the length of the user's original video collection, and the application takes 3 minutes as an example. The length of the time is designed to balance the data quality and the cost of user recording, avoiding too long to cause high user operation threshold, and too short to cause insufficient feature extraction.
[0041] The broadcast audio data is personalized audio (also referred to as individualized TTS audio IA) generated based on the audio features of the user's original video, which is highly matched with the user's voice timbre and speech rate. Its generation logic is: separate the audio from the original video of the preset length; fine-tune the pre-trained TTS (text-to-speech) model (i.e., the personalized sound model of the application) with the audio, so that the model learns the user's voice features; input the to-be-broadcast text into the fine-tuned TTS model, and output the broadcast audio data.
[0042] The broadcast emotion data is the emotional category (also referred to as emotional IE) that the virtual person needs to express, which is determined based on the emotional tendency of the to-be-broadcast text. Its generation logic is: use a pre-trained text emotion classifier (i.e., the emotion classification algorithm of the application) to analyze the text of the sales pitch (to-be-broadcast text) sentence by sentence; output the emotional category (which can include basic emotions such as happiness, sadness, anger, surprise, fear, and disgust), which is used to drive the facial expression of the virtual person subsequently.
[0043] It should be noted that the "preset time period of 3 minutes" of the original video of the application is the optimal length verified by researchers: shorter than 3 minutes will result in insufficient fine-tuning of the TTS model, which cannot accurately match the user's voice timbre; longer than 3 minutes will increase the cost of user recording, which violates the core goal of "low cost and fast generation".
[0044] The personalization of the broadcast audio data in the present application solves the problem of "stiff sound of general audio-driven virtual human" in the related art. For example, the original video of the user has low-pitched and slow speech, and the fine-tuned TTS model generates broadcast audio that completely retains these characteristics.
[0045] In step 102, a stable video segment in the original video is determined, and based on the stable video segment, broadcast face data of the virtual human is determined. The video duration of the stable video segment is greater than or equal to a preset effective duration.
[0046] In the embodiments of the present application, the present application can filter high-quality segments from the original video in a preset period, extract stable face features, and solve the problem of noise interference such as user shaking and camera jitter in the original video, thereby providing high-quality input data for subsequent generation of clear, stable, and realistic virtual human faces.
[0047] Specifically, the present application can first filter noise segments to obtain stable video segments; then extract face data from the stable segments and optimize it to ensure that the face features are accurate and free of jitter, thereby providing a reliable basis for subsequent face generation.
[0048] The stable video segment is a noise-free and normally posed video segment (i.e., a high-quality video segment) filtered from the original video in a preset period. The filtering criteria rely on head pose estimation and eye estimation. The head pose estimation calculates the three-dimensional angle of the head by facial feature points (such as 68 key points) to exclude non-frontal pose segments such as "side head" and "large head tilt / tilt". The eye estimation determines the eye direction by iris positioning to exclude segments that do not conform to normal interaction, such as "squinting eyes" and "closed eyes". Through head pose estimation and eye estimation, the final filtered segment must meet the criteria of "no shaking, no jitter, normal pose / eye".
[0049] The preset effective duration is the minimum duration criterion for the stable video segment. The present application can take 1 minute as an example. The present application selects 1 minute as the preset effective duration because less than 1 minute may result in insufficient face feature extraction, which cannot cover the user's diverse facial poses and affect the accuracy of virtual human face generation. 1 minute can ensure the data volume while avoiding excessive filtering difficulty.
[0050] The broadcast face data is core face information (also referred to as face frame sequence IF) extracted from the stable video segment and used to drive the generation of the virtual human face. The generation logic is as follows: the face detection model (such as MTCNN) is used to extract the face region key feature point coordinates (i.e., face frame coordinates) from the stable video segment frame by frame to remove background interference; then the face frame coordinates are smoothed (such as sliding average) to solve the jitter problem of single-frame detection; and finally, the stable, continuous, and jitter-free face frame sequence is output, which contains the core features of the user's facial contour and facial feature ratio.
[0051] It can be understood that the screening of the stable video segment in the application actively denoises through pose / gaze estimation, reduces the interference of subsequent model training, and improves the face generation quality of the virtual person. Among them, the "smoothing processing" of the anchor face data in the application cannot be omitted, because "single frame recognition face frame exists data jitter", if not handled, the virtual person face generated subsequently will have the problem of "big and small, position offset", and after smoothing processing, the face frame sequence can be ensured to be stable and coherent.
[0052] In step 103, based on the anchor audio data, the anchor emotion data and the anchor face data, a face video of the virtual person is generated, so as to generate a virtual person video corresponding to the target user according to the face video of the virtual person.
[0053] In the embodiments of the application, the core target of the application is to generate a virtual person video that meets the "face realistic, emotion matching, lip synchronization" standard and is special for the target user. Specifically, first, the three core data of "sound, emotion and face" are fused by means of a special generator to generate a face video containing only the face of the virtual person; then the generated face video is fused with the original stable video segment (which contains user body actions and background) to complete the body and background dimensions, and finally a complete and natural virtual person video is output.
[0054] In the embodiments of the application, the face video generation process of the virtual person is as follows: the virtual person face video generated by the application is a video that only presents the face of the virtual person, which is generated by a speaker face generator based on expression enhancement, and the key is "three input data fusion + feature extraction + algorithm optimization".
[0055] Firstly, the generator needs to obtain three types of key data, each type of data corresponding to a core dimension of virtual person face generation, so as to ensure that the virtual person face is "like the user, accurate in lip shape, and matched in expression".
[0056] Anchor audio data (IA): provides audio features, mainly to ensure that the virtual person's lip shape is synchronized with the sound, and to prevent audio-visual misalignment. For example, when saying "hello", the virtual person's lip shape should match the pronunciation of "hello".
[0057] Anchor emotion data (IE): provides emotion categories such as happiness, sadness, anger, etc., mainly to ensure that the virtual person's expression matches the emotion of the text to be broadcast, and to avoid emotional misalignment. For example, when broadcasting sad text, the virtual person's expression should show a state of drooping mouth corners and furrowed brows.
[0058] Face data of the reporter (IF): Provide the facial features of the target user, such as the proportions of the facial contours, the relative positions of the facial features, etc., and the main role is to make the virtual person's face have a high degree of similarity with the target user. For example, if the user has a wide forehead and a sharp chin, the virtual person's face generated will retain these features.
[0059] Based on the above three types of input data, the present application needs to extract features that can directly drive face generation and build a mapping relationship between "features and facial movements". Specifically, from the reporting audio data, extract the lip movement driving features, that is, the lip movement parameters corresponding to different pronunciations (such as "a", "o", "e"), to provide driving basis for lip movement; from the reporting emotion data, extract the expression control features, and convert emotion categories such as "happy" and "sad" into specific facial muscle movement parameters, for example, "happy" corresponds to the parameters of the mouth corners up and the eyes squinting; from the face data of the reporter, extract the stable facial features, extract the static features of the user's facial basic contour and the positions of the facial features, etc., as the "basic template" of the virtual person's face.
[0060] Subsequently, the generator realizes feature fusion and video generation through "face encoder + audio encoder + decoder", and optimizes the generation effect through the loss function. The specific steps are as follows: first, encode the stable facial features and the lip movement driving features through the face encoder and the audio encoder respectively, then vector splice the encoded features and the expression control features to form a comprehensive feature matrix containing "lip movement + facial expression + basic face shape"; analyze the comprehensive feature matrix frame by frame through the decoder to generate a continuous sequence of virtual person's face images, and then combine them into a virtual person's face video; finally, ensure the generation effect through three types of loss functions, as follows:
[0061] Face reconstruction loss (L1, i.e. Lrecon): Calculate the difference between the generated face frame and the user's real face frame to ensure that the virtual person's face has a high degree of similarity with the user.
[0062] Lip movement synchronization loss (Lsync): Calculate the correlation between the audio features and the generated face frame features to ensure that the lip movement is synchronized with the sound.
[0063] Emotion synchronization loss (Lemo): Calculate the cross-entropy loss between the generated facial expression and the preset emotion category to ensure that the expression matches the emotion.
[0064] Among them, in the training process, in order to better reduce the difference between the generated video frame L g (i.e. the generated virtual person video) and the target video frame L g (i.e. the user's real recorded video). Use the face reconstruction loss function as its loss function, which can be
[0065] To better realize the synchronization between the generated video frame and the audio, a pre-trained lip-sync discriminator is used for audio-video synchronization, and the related lip-sync loss function is:
[0066]
[0067] Wherein, the main purpose of the lip-sync loss function is to compress the audio features and video frame (image) features through two encoding networks, and use L sync Optimize the two encoding networks, if the correlation between the two (audio, video synchronization) is strong P sync The larger L sync The smaller, the two (audio, video out of sync) is the opposite.
[0068] Wherein: v is the extracted audio features, s is the extracted video frame features. P is the correlation calculation, which can see the similarity of two mathematical vectors.
[0069] The cross-entropy loss calculated between the predicted emotion category of the generated frame by the emotion discriminator and the conditional emotion category is used to optimize the emotion discriminator. The emotion synchronization loss function can be specifically:
[0070] Wherein, y is the conditional emotion category, The emotion category predicted by the emotion discriminator for the generated frame.
[0071] The whole training generator related target function is:
[0072] L gen =αL recon +βL sync +γL emo
[0073] Wherein, α, β, γ are the weights of each loss component. Multiple weights are used to balance the values of each loss function and guide the learning direction of the model. Finally, the desired video
[0074] In the embodiments of the present application, the virtual human video corresponding to the target user is generated as follows: the virtual human video corresponding to the target user is the final output complete virtual human video, and the core is to complete the "limb and background" dimension, and solve the discomfort caused by only face video.
[0075] Specifically, the previously generated virtual human face video only contains the face, lacks the user's limb action (such as hand gesture, body posture) and background environment, and needs to be fused with "stable video segment" (high-quality segment selected from the target user's 3-minute original video, containing limb action and background).
[0076] The natural connection of "face-limb-background" is realized through image fusion technology, and the key operations include:
[0077] Timeline synchronization: Ensure that the playback progress of the face video and the stable video segment are consistent. For example, when the face video says "the first point", the user's limb in the stable segment just makes a "1" gesture.
[0078] Pose alignment: According to the head pose of the user in the stable video segment (such as slight head tilt), adjust the angle, size and position of the virtual face in the face video to avoid the contradiction of "head front, body side".
[0079] Pixel-level fusion: Replace the original face area in the stable video segment with the adjusted virtual face to ensure a natural transition at the fusion edge (such as the junction of face and hair, collar).
[0080] Finally, a complete virtual human video is generated, which has "personalized virtual face, user's original natural action, consistent background with the original scene", and meets the core requirements of "lip movement and audio synchronization, consistent expression and emotion".
[0081] In summary, according to the virtual human video generation method proposed in this application, based on the to-be-broadcast text and the original video pre-acquired by the target user at a preset time period, the broadcast audio data and broadcast emotion data of the virtual human are determined, and the stable video segment with a time length not less than a preset effective time length is selected from the original video to determine the broadcast face data. Then, the three types of data are combined to generate a face video to obtain a virtual human video corresponding to the target user. By synergistically fusing the broadcast audio, emotion and face data, the precise matching of lip movement and audio, expression and emotion is realized, making the virtual human performance more natural and lively, effectively avoiding the uncanny valley effect, and only a short original video is needed to complete personalized generation, greatly reducing the recording cost and technical threshold, and improving the efficiency and applicability of virtual human video generation.
[0082] As a possible implementation manner, Figure 2 A flowchart of a second virtual human video generation method is shown. Based on the above-mentioned embodiment, the stable video segment in the original video is determined, including the following steps:
[0083] Step 201: Determine the head deflection parameter and eye shift parameter of the target user in each frame of the original image in the original video through head estimation algorithm and eye estimation algorithm.
[0084] In the embodiments of the present application, the present application analyzes the original video frame by frame through two special algorithms, converts the two subjective judgment indexes of "whether the head posture is normal" and "whether the gaze is focused" into quantifiable parameters, provides an objective basis for subsequent "data screening", and solves the problem of inaccurate facial feature extraction caused by "user side head and oblique eye" in the original video.
[0085] Specifically, the present application can run the head estimation algorithm and the gaze estimation algorithm on each frame of the original image of the 3-minute original video, respectively, output the corresponding "head deflection parameter" (measuring the degree of head deflection from the front) and "gaze deflection parameter" (measuring the degree of gaze deflection from the front view) of each frame, and realize "frame-by-frame quantitative detection".
[0086] The head estimation algorithm is a technology for detecting the head posture of the target user, which calculates the deflection angle of the head in three-dimensional space (including left-right deflection, up-down pitch, and front-back rotation three dimensions) by recognizing the key feature points of the face (such as 68 facial feature points, including the corners of the eyes, the corners of the mouth, the angle of the jaw, etc.), and finally outputs the head deflection parameter. The core function is to judge whether the head is in a front or approximately front posture.
[0087] The gaze estimation algorithm is a technology for detecting the gaze direction of the target user, which calculates the deflection angle of the gaze from the "front view direction" (i.e. the front of the lens) (including horizontal left-right deflection and vertical up-down deflection two dimensions) by positioning the iris center, pupil edge and other key eye regions, and finally outputs the gaze deflection parameter. The core function is to judge whether the gaze is focused on the lens.
[0088] The original image is a single frame picture after the original video is split, and is the smallest processing unit of the head estimation algorithm and the gaze estimation algorithm. Each frame of image corresponds to the user posture and expression at a time point.
[0089] The head deflection parameter is the output result of the head estimation algorithm, which is a numerical value (such as an angle value, unit: degree) quantifying the degree of head deflection from the front, for example, "head deflection 10° to the left" corresponds to a specific deflection parameter value. The larger the parameter, the more serious the head deflection from the front.
[0090] The gaze deflection parameter is the output result of the gaze estimation algorithm, which is a numerical value (such as an angle value, unit: degree) quantifying the degree of gaze deflection from the front view direction, for example, "gaze deflection 5° to the right" corresponds to a specific deflection parameter value. The larger the parameter, the more dispersed the gaze and the less focused.
[0091] Among them, the "three-dimensional angle detection" of the head estimation algorithm in the application can eliminate non-frontal poses such as "side head", the head estimation algorithm of the application needs to cover "left-right deflection" (such as turning the head left / right), "up-down tilt" (such as lifting the head / lowing the head), and "forward-backward rotation" (such as head forward inclination / backward tilt), to ensure that there is no dead angle to detect the head posture; the "two-dimensional direction detection" of the eye estimation algorithm can eliminate "oblique eyes" in the application, and the eye estimation algorithm of the application focuses on the "horizontal + vertical" two dimensions to avoid unnatural virtual human eyes caused by eye up / down / left / right shift.
[0092] In step 202, the plurality of original images with the head deflection parameter less than or equal to the preset deflection threshold and the continuous frame number greater than or equal to the first frame number threshold are determined as the head stable video segment.
[0093] In the embodiments of the application, based on the head deflection parameter output by step 201, the video segment with "stable head posture (approximately frontal) and continuous" is screened out, and the noise frame with "large head deflection and frequent shaking" is eliminated, so that the problem of "incoherent facial feature extraction caused by unstable head posture" in the original video is solved.
[0094] Specifically, the application can set a "preset deflection threshold" (a criterion for judging whether the head is approximately frontal) and a "first frame number threshold" (a criterion for judging whether the segment is continuous and stable), and screen out an image sequence with "each frame head deflection parameter ≤ preset deflection threshold" and "continuous frame number ≥ first frame number threshold" from the original video. The sequence is the head stable video segment.
[0095] The preset deflection threshold is a judgment criterion for "whether the head is approximately frontal" set in advance, which is a fixed head deflection parameter threshold (such as 5°). If the head deflection parameter of a frame ≤ the threshold, it means that the head is in a frontal or approximately frontal posture (which meets the requirement of subsequent facial feature extraction); if the parameter > the threshold, it is determined as "side head, head up" and other unstable postures, which need to be eliminated.
[0096] The first frame number threshold is a judgment criterion for "continuous and stable head posture" set in advance by the application, which is a fixed continuous frame number threshold (which needs to be converted in combination with the frame rate of the original video, for example, if the video frame rate is 30 frames / second, and 0.5 seconds of continuous stability is required, then the first frame number threshold is 15 frames). Only when the number of frames that continuously meet "head deflection parameter ≤ preset deflection threshold" reaches the threshold, it is determined as a "head stable segment", so as to avoid screening out invalid segments with "single stable frame but overall shaking".
[0097] The head-stable video clip is an image sequence (video clip) selected from the original video, in which the head of each frame is in a front or approximately front pose without large deflection or frequent shaking, and can be used as a basic data source for subsequent eye screening.
[0098] In the present application, the core goal of the present application is to "quickly generate a virtual human", and therefore the threshold needs to balance the "screening strictness" and the "data volume". If the threshold is too small (such as 1°), the selected clip will be too short; if the threshold is too large (such as 10°), the head pose deviation will be large, and therefore the default setting in the file is a "small angle threshold" (such as 3-5°) to ensure that the head is approximately front. Meanwhile, the original video in the present application is 3 minutes, and the frame rate is usually 25-30 frames per second, and the first frame threshold needs to correspond to "short time continuous stability" (such as 0.3-0.5 seconds). For example, for a 30 frames per second video, the first frame threshold can be set to 10 frames (corresponding to 0.33 seconds) to avoid selecting a clip that is "stable in a single frame but shaking in front and back frames".
[0099] In step 203, from the head-stable video clip, a plurality of original images with an eye shift parameter less than or equal to a preset shift threshold and a continuous frame number greater than or equal to a second frame number threshold are selected as an eye-focused video clip.
[0100] In the embodiments of the present application, the present application further screens the "eye-focused and continuous" clip based on the head-stable video clip obtained in step 202, eliminates the "eye-divergent, squinting, and closed-eye" noise frames, and solves the problem of "eye not focusing leading to virtual human eye dullness".
[0101] Specifically, the present application can set a "preset shift threshold" (a criterion for judging whether the eyes are focused) and a "second frame number threshold" (a criterion for judging whether the eyes are continuously focused), and select an image sequence with "each frame eye shift parameter ≤ preset shift threshold" and "continuous frame number ≥ second frame number threshold" from the head-stable video clip, which is the eye-focused video clip.
[0102] The preset shift threshold is a judgment criterion for "whether the eyes are focused" and is a fixed eye shift parameter critical value (such as 3°). If the eye shift parameter of a certain frame ≤ the threshold, it means that the eyes are looking straight or approximately looking straight at the lens (meeting the requirement of natural virtual human eyes); if the parameter > the threshold, it is determined to be "squinting, eye-divergent", and needs to be eliminated.
[0103] The second frame number threshold is a pre-set judgment standard of "eye focus", which is a fixed continuous frame number threshold (consistent with the first frame number threshold, and needs to be converted according to the frame rate, for example, 10 frames for a video of 30 frames per second), and only when the frame number of the continuous frames meeting the condition of "eye offset parameter ≤ pre-set offset threshold" reaches the threshold, the "eye focus segment" is determined, so as to avoid screening out invalid segments with "single-frame focus but overall dispersion".
[0104] The eye focus video segment is an image sequence (video segment) further screened from the head stable video segment, which is "head stable and eye focus", and meets the conditions of "head approximately front, eye looking at the lens", and is a higher quality data source than the head stable video segment, which lays a foundation for subsequent screen of picture jitter.
[0105] In the present application, the threshold needs to be strictly set (such as 2-3°) to ensure the natural eye of the virtual person, only the frames of "looking at the lens" are reserved, so as to avoid the discomfort of "eye wandering" of the virtual person, and the present application is only used for screening of the head stable video segment, not the original video, which reflects the hierarchical logic of "filtering head first, then filtering eye", reduces invalid calculation, and improves the screening efficiency.
[0106] In step 204, based on the eye focus video segment, a stable video segment is determined.
[0107] In the embodiment of the present application, based on the eye focus video segment obtained in step 203, the unstable frames caused by "camera jitter and picture shaking" are removed by detecting the inter-frame pixel displacement, and finally the high-quality stable video segment with "head stable, eye focus and no picture jitter" is obtained.
[0108] Specifically, the present application first detects the inter-frame displacement of the eye focus video segment to obtain the pixel displacement of adjacent frames; then sets a "pre-set jitter threshold" (a standard for judging whether the picture is jitter-free) and a "third frame number threshold" (a standard for judging whether the segment is continuous and jitter-free), and screens out an image sequence with "adjacent frame pixel displacement ≤ pre-set jitter threshold" and "continuous frame number ≥ third frame number threshold", which is the final stable video segment.
[0109] The inter-frame displacement detection is a technology for detecting whether the picture is jittered, which calculates the offset distance of the pixels between adjacent frames (i.e. pixel displacement) by comparing the pixel positions of "adjacent two frames of original images" in the eye focus video segment (such as selecting the edge of the picture, the fixed point of the background, etc.), and the core function is to judge whether there is camera jitter (such as slight shaking when holding the camera).
[0110] Pixel displacement is the output result of inter-frame displacement detection, which is a value quantifying the degree of adjacent frame picture jitter (unit: pixel), for example, "adjacent frame background pixel offset 2 pixels" corresponds to a specific pixel displacement value, and the larger the parameter, the more serious the picture jitter.
[0111] The preset jitter threshold is a pre-set judgment standard for "whether the picture is jitter-free", which is a fixed pixel displacement threshold (such as 3 pixels). If the pixel displacement of adjacent frames is ≤ the threshold, it means that the picture is stable and has no obvious jitter (meets the requirements of subsequent face frame smoothing processing); if the parameter is > the threshold, it is determined as "camera jitter" and needs to be excluded.
[0112] The third frame threshold is a pre-set judgment standard for "picture continuous jitter-free", which is a fixed continuous frame threshold (needs to be converted in combination with the requirement of "at least 1 minute of high-quality segment", for example, for a 30 frames / second video, 1 minute corresponds to 1800 frames, and the third frame threshold needs to ensure that the total length of the final segment is ≥ 1 minute). Only when the number of consecutive frames that meet "pixel displacement ≤ preset jitter threshold" reaches the threshold, it is determined as "stable video segment".
[0113] The stable video segment is the final high-quality video segment after steps 201-204 filtering, which meets three conditions: head stability (approximately frontal); eye focus (looking directly at the lens); and picture jitter-free (adjacent frame pixel displacement is small). It is the core data source for subsequent extraction of broadcast face data (IF) and fusion of virtual face and body, and in this application, it needs to be clear that its length needs to be "greater than or equal to the preset effective length (at least 1 minute)".
[0114] In this application, in order to avoid the influence of user's own actions (such as slightly lifting hands) on the detection result, the algorithm selects "background fixed points" (such as wall texture, table edge) as reference pixels, only calculates the displacement of background pixels, and ensures that the detection result only reflects "camera jitter" and not user actions; and this application clearly requires that the stable video segment be "at least 1 minute", so the third frame threshold needs to be converted according to the frame rate (such as a 30 frames / second video, the third frame threshold ≥ 1800 frames), and at the same time allows multiple continuous segments to be selected, and the total length meets 1 minute, ensuring that the amount of data for subsequent face feature extraction is sufficient.
[0115] In specific implementation, this application can also introduce the judgment of user expression dimension to further optimize the filtering effect of stable video segments, so that the selected segments are more suitable for the subsequent virtual human expression generation requirements. For example, Figure 3A schematic diagram of determining a stable video segment is shown, and the core screening logic is: taking the original video of the target user as the processing object, on the basis of the original head posture estimation (judging whether the head is stable, approximately frontal), blinking estimation (judging whether the eyes are focused, without frequent closing of the eyes), an additional expression recognition link (identifying whether the user's facial expression is natural, without abnormal distortion, such as excessive exaggerated facial movements or meaningless expression shaking) is added; through the cooperative detection and screening of the three dimensions of head posture estimation, blinking estimation, and expression recognition, the noise frames of head deviation, eye distraction, and abnormal expression are removed from the original video, and finally the video segment meeting the three conditions of "stable head, focused eyes, and natural expression" is obtained, which is the stable video segment (also called self-selected video) used for subsequent processing in the present application.
[0116] As a possible implementation, Figure 4 A flowchart of a third virtual person video generation method is shown. On the basis of the above-mentioned embodiments, based on the stable video segment, the anchor face data of the virtual person is determined, including the following steps:
[0117] Step 301, for each frame image of the stable video segment, the face detection model is used to extract the key feature point coordinates of the face region of the target user.
[0118] In the embodiments of the present application, the face detection model is used to locate and extract the key feature points of the face from each frame image of the stable video segment, and the user's face contour and feature position are converted into quantifiable coordinate data, which provides the original feature basis for the subsequent virtual person face "similar to the user".
[0119] Specifically, the present application can apply the face detection model to the obtained stable video segment (which has met the conditions of stable head, focused eyes, and no shaking in the picture) frame by frame, identify and output the key feature point coordinates of the user's face in each frame image, and realize the conversion from "image to digital feature".
[0120] The face detection model is an algorithm model (such as MTCNN, Dlib, etc.) for locating key feature points of the face, which can accurately identify the feature point positions of the face contour (such as hairline, lower jaw line) and the five organs (eyes, nose, mouth, eyebrows), and output the two-dimensional coordinates (x, y) of each feature point in the image. The present application usually adopts a 68-standard facial feature point detection scheme.
[0121] The facial region key feature points are key position points of a face detection model for recognition, including: contour feature points (such as chin tip, left and right mandibular angles, etc.): defining the overall contour of the face; eye feature points (such as left and right corners of the eyes, pupil edges, etc.): defining the shape and position of the eyes; nose feature points (such as the tip of the nose, both sides of the nose, etc.): defining the shape of the nose; lip feature points (such as the upper and lower lip edges, corners of the mouth, etc.): defining the shape of the lips, providing a basis for subsequent lip synchronization; these points together constitute the "feature skeleton" of the user's face, determining the similarity between the virtual person and the user.
[0122] Since the frames of head offset, eye distraction and picture jitter have been filtered through steps 201-204, the feature point coordinates extracted by the present application are more stable, reducing noise interference; and the present application adopts the industry's common 68-point labeling system, ensuring that the feature points cover all key areas of the face, providing a unified feature dimension for subsequent smoothing processing and virtual person generation.
[0123] In step 302, the facial region key feature point coordinates of the stable video segment are adjusted by a smoothing processing algorithm to obtain the anchor face data of each frame.
[0124] In the embodiments of the present application, the smoothing processing algorithm is used to eliminate the slight fluctuations of the feature point coordinates between consecutive frames (caused by single-frame detection errors), ensuring that the facial feature point sequence is coherent and stable in the time dimension, and avoiding "high-frequency jitter" (such as the eyes and mouth becoming large and small suddenly) in the face of the generated virtual person.
[0125] Specifically, the smoothing processing algorithm (such as sliding average, Gaussian filtering) can be applied to the continuous frame feature point coordinates extracted in step 301 to correct them, so that the feature point coordinates of adjacent frames tend to be smooth, and the optimized coordinates of each frame output are the anchor face data.
[0126] The smoothing processing algorithm is an algorithm for eliminating fluctuations in the feature point coordinates of consecutive frames, and common algorithms include: sliding average algorithm (taking the average value of the same feature point coordinates of the current frame and the previous and next N frames as the optimized coordinates of the current frame); Gaussian filtering algorithm (weighting and averaging the coordinates of consecutive frames by a Gaussian function, with higher weight for recent frames and lower weight for distant frames); the core function is to filter random noise of single-frame detection and retain the overall trend of facial movement.
[0127] The facial region key feature point coordinates of consecutive frames are time series data output by step 301, that is, a set of feature point coordinates of each image in the order of play (such as 68 point coordinates of the first frame, 68 point coordinates of the second frame,..., 68 point coordinates of the Nth frame), since there are slight errors in single-frame detection, the coordinates of the same feature point of adjacent frames may fluctuate irregularly (such as a sudden shift of 1-2 pixels in the left eye coordinates of a frame).
[0128] The anchor face data is the optimized feature point coordinate sequence output by step 302, and is one of the core input data for generating a virtual human face, and includes: smoothed coordinates of 68 facial feature points in each frame of image; coherent motion trend of feature points over time (such as natural blinking and slight expression change); the data retains the personalized features (contour, facial feature ratio) of the user's face, and eliminates jitter noise, ensuring that the virtual human face moves naturally and smoothly.
[0129] It should be noted that if the smoothing is too much, the facial movement will be stiff (such as delayed blinking); if the smoothing is not enough, the jitter cannot be eliminated, and the smoothing processing algorithm balances between "noise removal" and "motion authenticity" by adjusting the size of the sliding window (such as 3-5 frames) or the Gaussian kernel parameter; the anchor face data will be one of the inputs of the "face generation module", and its stability directly affects the quality of the virtual human face generation, and the smoothed coordinates make it easier for the generator to learn the natural motion rules of the user's face.
[0130] In summary, the present application completes the conversion "from image to feature point" through the face detection model, ensures that the extracted feature points accurately reflect the user's facial morphology; eliminates single-frame noise through smoothing processing, completes the optimization "from original feature to stable feature", and ensures that the feature point sequence is coherent in the time dimension. The finally output anchor face data not only retains the personalized features of the user's face (ensuring that the virtual human "looks like the user"), but also has time continuity (ensuring that the virtual human face movement is "natural and not jittery"), laying a key foundation for subsequent fusion of audio and emotional features to generate a virtual human face video.
[0131] As a possible implementation manner, Figure 5 A flowchart of a fourth virtual human video generation method is shown. Based on the above-mentioned embodiments, based on the to-be-broadcast text and the original video of the target user, the broadcast audio data and the broadcast emotion data of the virtual human corresponding to the target user are determined, including the following steps:
[0132] Step 401, using the to-be-broadcast text and the target sound model of the target user, generating broadcast audio data corresponding to the to-be-broadcast text, the target sound model being obtained by training a personalized sound model based on the original video of the target user.
[0133] In the embodiments of the present application, the text is converted into audio with high matching of user voice, speech rate and intonation through the pre-trained target sound model, solving the problem of "generalized sound and lack of personalization" of traditional virtual humans.
[0134] Specifically, as shown in Figure 6 The present application provides a specific schematic diagram of obtaining a target sound model, as shown in Figure 6The application first acquires audio data of a user's original video, inputs the audio data into a pre-trained TTS model (i.e., a personalized voice model) to train a dedicated personalized TTS model (i.e., a target voice model), and then uses the model to convert any text to be broadcast into audio in the user's style. In other words, the application can extract original audio from a user's 3-minute original video, cut it into short sentences or phoneme segments; input these segments as "supervision signals" into the personalized voice model for fine-tuning (optimize personalized parameters through back propagation); after training is completed, the model can learn the user's voice features (such as deep tone color, slow speaking speed), and generate similar style voice for any text.
[0135] The text to be broadcast is the content text that needs to be broadcast by the virtual human (such as news scripts, teaching scripts, customer service scripts, etc.), which is the "semantic input" of audio generation and is irrelevant to the audio content in the original video.
[0136] The target voice model of the target user is a text-to-speech (TTS) model customized for the target user, which can simulate the user's voice characteristics such as tone, speaking speed, and intonation, and output voice with personalized features of the user. Its essence is to fine-tune a general personalized voice model based on user original audio data to obtain a dedicated model.
[0137] The personalized voice model is a pre-trained general text-to-speech (TTS) base model that has the ability to convert text into natural speech, but does not incorporate specific user voice characteristics (output as a general tone). It is usually trained based on a large amount of multi-person voice data and contains the basic network structure of speech synthesis (such as encoder, decoder, vocoder).
[0138] The original audio in the original video is pure audio data (without images) separated from the target user's 3-minute original video, which contains the user's tone, speaking speed, pronunciation habits, and other personalized voice features, and is the core "sample data" for training the target voice model.
[0139] The personalized parameters are parameters in the personalized voice model that control voice characteristics (such as pitch curve, speaking speed weight, formant frequency, etc.), which determine the tone and style of the synthesized voice. By training with user original audio, these parameters are adjusted to match user characteristics, and the target voice model is obtained.
[0140] The broadcast audio data is the audio file generated by the target voice model based on the text to be broadcast, which has a tone, speaking speed, and intonation highly consistent with the target user, and is the core data that drives the synchronization of the virtual human's mouth shape (corresponding to the input source of the mouth shape driving feature)
[0141] It can be seen that the application does not require the user to record the audio of the text to be broadcast, but can generate personalized voice through the text only, thereby solving the problems of "user recording in a noisy environment and non-standard pronunciation", and greatly reducing the operation threshold.
[0142] In step 402, a sentiment classification algorithm is used to perform sentiment analysis on the text to be broadcast, to determine the broadcast sentiment data corresponding to the text to be broadcast.
[0143] In the embodiments of the application, the semantic sentiment of the text to be broadcast is analyzed by the sentiment classification algorithm, and the corresponding sentiment category is output, to provide "emotion guidance" for virtual human expression generation, and to solve the problem of "disconnection between virtual human expression and text sentiment".
[0144] Specifically, the text to be broadcast is input into a pre-trained sentiment classification model, and the model outputs the sentiment category corresponding to the text, i.e., the broadcast sentiment data, based on the sentiment words (such as "joy" and "sadness") and tone (such as exclamatory sentences and interrogative sentences) in the text. For example, for a long text (such as multiple teaching contents), the algorithm performs "sentence-by-sentence sentiment analysis" to output the sentiment label corresponding to each sentence (for example, the first sentence "Welcome everyone" is "neutral", and the second sentence "Congratulations on winning" is "happy"), so that the virtual human expression changes in real time with the text sentiment, avoiding the stiffness of "single expression for the whole shot". The broadcast sentiment data will be converted into expression control features (such as "happy" corresponding to the mouth corner lifting parameter), input into the face generator, and finally reflected in the facial expression of the virtual human, to realize the accurate mapping of "text sentiment to expression action".
[0145] The sentiment classification algorithm is an algorithm model (such as a classification model based on BERT and LSTM) for analyzing the sentiment tendency of the text, which can divide the text into preset sentiment categories (such as basic emotions of happiness, sadness, anger, surprise, fear, and disgust). The core is to learn a large amount of text data labeled with sentiment categories, to master the language features of different sentiments (such as "too good" corresponding to happiness, and "regret" corresponding to sadness).
[0146] Sentiment analysis is the process of automatically identifying and classifying the sentiment tendency of the text, including: word-level analysis: identifying sentiment words (such as positive words and negative words) in the text; sentence-level analysis: judging the overall sentiment of the sentence based on the grammatical structure (such as "although it is raining, but very happy" as a whole); context analysis: sentence-by-sentence analysis of long texts (such as multiple scripts), to capture the changes in sentiment.
[0147] The broadcast sentiment data is the output result of the sentiment classification algorithm, which is the sentiment category label (such as "happy" and "sad") corresponding to the text to be broadcast, and can be divided into "sentence-by-sentence sentiment label" (the sentiment of each sentence) or "overall sentiment label" (the dominant sentiment of the whole text). This data is the core basis for generating virtual human expression control features, to ensure that the expression is consistent with the text sentiment.
[0148] In summary, the present application provides a "personalized voice" and a "matching text emotion" for a virtual person, which together form a two-dimensional feature of the virtual person's "speech content": the broadcast audio data ensures that the voice is consistent with the user; and the broadcast emotion data ensures that the expression matches the content. These two types of data, together with the broadcast face data generated by steps 301-302, form the three core inputs for virtual face generation, laying the foundation for the final generation of a virtual person video with "voice, expression pair, and accurate mouth shape".
[0149] For ease of understanding, as shown in Figure 7 , the present application provides a specific schematic diagram for obtaining a virtual person video.
[0150] Referring to Figure 7 , the flowchart shows the generation process of a digital virtual person video: first, the dialogue text is processed, on the one hand, the expression feature IE is obtained by using a text expression classifier, and on the other hand, the personalized audio IA is generated by using a personalized TTS model; at the same time, the self-selected video is processed by a face detector to generate a face video IF; then, the IE, IA, and IF are input into a speaker face generator based on expression enhancement to generate a face video; finally, the generated face video is fused with the self-selected video to obtain a digital virtual person video. The specific process of virtual person video generation can refer to the embodiment shown in Figures 1 to 6 , which will not be described here.
[0151] To implement the above-mentioned embodiments, the present application further provides a virtual person video generation device. Figure 8 A structure schematic diagram of a virtual person video generation device 800 provided by an embodiment of the present application. As shown in Figure 8 , the device comprises:
[0152] A first determination unit 810 is configured to determine broadcast audio data and broadcast emotion data of a virtual person corresponding to a target user based on a text to be broadcast and an original video of the target user, the original video being obtained in advance according to a preset time period;
[0153] A second determination unit 820 is configured to determine a stable video segment in the original video, and determine broadcast face data of the virtual person based on the stable video segment, the video duration of the stable video segment being greater than or equal to a preset effective duration;
[0154] A generation unit 830 is configured to generate a face video of the virtual person based on the broadcast audio data, the broadcast emotion data, and the broadcast face data, so as to generate a virtual person video corresponding to the target user according to the face video of the virtual person.
[0155] The method and device provided in the embodiments of the present application are introduced in the above embodiments of the present application. In order to realize the functions in the method provided in the embodiments of the present application, the electronic device can include a hardware structure, a software module, and realize the above functions in the form of hardware structure, software module, or hardware structure plus software module. Some of the above functions can be executed in the form of hardware structure, software module, or hardware structure plus software module.
[0156] Figure 9 is a block diagram of an electronic device 900 for implementing the virtual human video generation method according to an exemplary embodiment. For example, the electronic device 900 can be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0157] Referring to Figure 9 , the electronic device 900 can include a communication interface 901 capable of interacting with other devices, a processor 902 connected to the communication interface 901 to implement interaction with other devices, for running computer programs, and executing the method provided in one or more technical solutions described above; a memory 903, on which a computer program is stored. Specifically, the specific processing process of the processor 902 can refer to the virtual human video generation method described in the embodiments of the present disclosure.
[0158] Of course, in actual application, each component in the electronic device 900 is coupled together through a bus system 904. It can be understood that the bus system 904 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 904 also includes a power bus, a control bus and a status signal bus. However, in order to clearly illustrate the present application, various buses are marked as the bus system 904 in Figure 9 .
[0159] The memory 903 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device 900. Examples of these data include: any computer programs used to operate on the electronic device 900.
[0160] The method disclosed by the embodiments of the present application can be applied to the processor 902 or implemented by the processor 902. The processor 902 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 902. The processor 902 described above can be a general processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 902 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the hardware decoding processor can be directly embodied to complete the execution, or the combination of hardware and software modules in the decoding processor can be used to complete the execution. The software module can be located in the storage medium, which is located in the memory 903, and the processor 902 reads the information in the memory 903 to complete the steps of the above method in combination with the hardware.
[0161] In the exemplary embodiments, the electronic device 900 can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, micro controllers (MCUs), microprocessors (Microprocessors), or other electronic elements, for executing the above-described methods.
[0162] The embodiments of the present disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to make a computer execute the virtual human video generation method described in the above embodiments of the present disclosure.
[0163] The embodiments of the present disclosure also propose a computer program product comprising a computer program, which, when executed by a processor, executes the virtual human video generation method described in the above embodiments of the present disclosure.
[0164] Embodiments of the present disclosure also provide a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device, and send a signal to the processor, the signal comprising computer instructions stored in the memory, when the processor executes the computer instructions, causing the electronic device to perform the virtual human video generation method described in the above embodiments of the present disclosure.
[0165] It should be noted that the terms "first", "second", and the like in the description of the present disclosure and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0166] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0167] Any process or method descriptions or descriptions of the flow diagrams in the present specification can be understood as representing code modules, segments or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process, and the preferred embodiments of the present invention include additional implementations in which the order of steps can be different, including use of the same or different code modules, segments or portions of code, and the functions described can be performed in an order different from that shown or discussed, including substantially concurrently or in reverse order, as will be appreciated by those skilled in the art.
[0168] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processing module, or other systems that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (control method), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CD-ROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via the optical scanner of a device or device, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.
[0169] It should be understood that each part of the embodiments of the present application can be realized by hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized by hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.
[0170] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-described embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the programs are executed, they include one or a combination of the steps of the method embodiments.
[0171] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can exist physically independently, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of software function module. When the integrated module is realized in the form of software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0172] Although the embodiments of the present application have been shown and described above, it should be understood by those skilled in the art that the above embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.
Claims
1. A method for generating virtual human videos, characterized in that, The method includes: Based on the text to be broadcast and the original video of the target user, the broadcast audio data and broadcast emotion data of the virtual human corresponding to the target user are determined, wherein the original video is pre-acquired according to a preset time period; Determine stable video segments in the original video, and based on the stable video segments, determine the broadcast face data of the virtual human, wherein the video duration of the stable video segments is greater than or equal to a preset effective duration; Based on the broadcast audio data, the broadcast emotion data, and the broadcast facial data, a facial video of the virtual person is generated, and a virtual person video corresponding to the target user is generated based on the facial video of the virtual person.
2. The method according to claim 1, characterized in that, The process of determining stable video segments in the original video includes: The head deflection parameters and eye shift parameters of the target user in each frame of the original video are determined by the head estimation algorithm and the eye expression estimation algorithm, respectively. Multiple original images with head deflection parameters less than or equal to a preset deflection threshold and consecutive frame count greater than or equal to a first frame count threshold are identified as head-stabilized video segments. From the head-stabilized video clips, select multiple original images with eye-shift parameters less than or equal to a preset offset threshold and consecutive frame counts greater than or equal to a second frame count threshold, and determine them as eye-focused video clips. Based on the eye-focused video segment, the stable video segment is determined.
3. The method according to claim 2, characterized in that, The process of determining the stable video segment based on the eye-focusing video segment includes: Inter-frame displacement detection is performed on the eye-focusing video segment to determine the pixel displacement of the original image between adjacent frames; From the eye-focused video clip, select multiple original images that have pixel displacement between adjacent frames less than or equal to a preset jitter threshold and consecutive frame count greater than or equal to a third frame count threshold, and determine them as the stable video clip.
4. The method according to claim 2, characterized in that, The process of determining the virtual human's broadcast face data based on the stable video clip includes: For each frame of the stable video segment, the coordinates of key feature points in the target user's facial region are extracted using a face detection model; The coordinates of key feature points in the facial region of consecutive frames in the stable video segment are adjusted by a smoothing algorithm to obtain the broadcast face data for each frame.
5. The method according to claim 1, characterized in that, The process of determining the audio data and emotional data of the virtual human corresponding to the target user based on the text to be broadcast and the original video of the target user includes: Using the text to be played and the target user's target voice model, the audio data corresponding to the text to be played is generated. The target voice model is obtained by training a personalized voice model based on the target user's original video. Sentiment classification algorithms are used to perform sentiment analysis on the text to be broadcast, and the corresponding broadcast sentiment data is determined.
6. The method according to claim 5, characterized in that, Before generating the virtual human's audio data using the text to be broadcast and the target user's target voice model, the method includes: Obtain the original video of the target user; The original audio is extracted from the original video, and the personalized voice model is trained based on the original audio. The personalized parameters in the personalized voice model are adjusted to obtain the target voice model corresponding to the target user.
7. The method according to claim 1, characterized in that, The step of generating the virtual human's facial video based on the broadcast audio data, the broadcast emotion data, and the broadcast facial data includes: Based on the broadcast audio data, the broadcast emotion data, and the broadcast face data, the lip-sync driving features corresponding to the broadcast audio data, the facial expression control features corresponding to the broadcast emotion data, and the stable facial features corresponding to the broadcast face data are extracted respectively. Based on the lip-sync features, facial expression control features, and stable facial features, and combined with a preset face generation algorithm, a video of the virtual human's face is generated.
8. The method according to claim 1, characterized in that, The step of generating a virtual human video corresponding to the target user based on the virtual human's facial video includes: The virtual human's facial video is fused with the stable video clip to obtain the virtual human video corresponding to the target user.
9. A virtual human video generation device, characterized in that, The device includes: The first determining unit is used to determine the audio data and emotional data of the virtual human corresponding to the target user based on the text to be broadcast and the original video of the target user, wherein the original video is pre-acquired according to a preset time period; The second determining unit is used to determine a stable video segment in the original video, and based on the stable video segment, determine the broadcast face data of the virtual human, wherein the video duration of the stable video segment is greater than or equal to a preset effective duration; The generation unit is used to generate a face video of the virtual person based on the broadcast audio data, the broadcast emotion data, and the broadcast face data, so as to generate a virtual person video corresponding to the target user based on the face video of the virtual person.
10. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the method according to any one of claims 1 to 6.
11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.
12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.
Citation Information
Cited By
Long video generation method and device based on digital human, and storage medium
CN121397323A
Virtual-real fusion short video generation method and system
CN121603748A
Short video generation method and system based on virtual-real fusion
CN121603748B
