Digital human driving method and device based on secondary reasoning

Abstract

The invention provides a digital human driving method and device based on secondary reasoning, and relates to the technical field of digital humans.The method comprises the steps that firstly, through fixed reference frame reasoning, mouth shape stability and tooth form consistency are ensured; a sliding reference frame is used for reasoning, and the face angle and background are dynamically adjusted, so that the generated video is more natural and smoother; and finally, the visual effect of the digital human is further optimized by fusing the two reasoning results. In addition, an abnormal frame skipping strategy is introduced, abnormal frames in the video can be automatically detected and skipped, driving errors caused by the abnormal frames are avoided, and therefore driving robustness is improved. The objective of the invention is to solve the problems that the mouth shape action of the existing driving method has incoherence in the time dimension, and the surrounding background of the human face is inconsistent with the whole image.

Description

Technical Field

[0001] The present invention relates to the technical field of digital humans, and particularly to a digital human driving method and device based on secondary reasoning. Background Art

[0002] In the development process of digital human technology, realizing the natural expressions and lip-sync of digital humans has always been one of the key research points. With the progress of technology, current digital human driving methods can already achieve lip-sync to a certain extent, that is, the lip movements of digital humans can match the audio signals, enabling the visual effects of digital humans when speaking to be presented synchronously with the sound. However, despite the achievement of lip-sync, in terms of more detailed visual effects, such as the coherence of teeth in each frame and the stability of lips, the existing technologies still have deficiencies.

[0003] Traditional methods mainly drive digital humans based on 2D models. When generating videos using this method, each frame of the image is generated independently, lacking consideration for coherence in the time dimension. This independent generation method results in a lack of coherence in the lip movements of each frame in the video, specifically manifested as obvious inconsistencies in the movements of teeth and lips between adjacent frames. For example, when the digital human closes its mouth, the movement of the lips may jitter or have an unnatural transition, affecting the naturalness of the visual effect.

[0004] To address this problem, some researchers have tried to introduce temporal information and upgrade the 2D model to a 3D model. Through the 3D model, the temporal continuity of lip movements can be simulated to a certain extent, thereby improving the effect of digital human driving. However, this method also brings new challenges. First, the training of 3D models requires more computing resources and data support, which increases the difficulty and cost of model training. Second, 3D models may still have limitations when dealing with complex scenarios, such as subtle changes in the facial expressions of digital humans.

[0005] In addition, the method of fixing the face reference frame can, to a certain extent, solve the problem of inconsistency in the time dimension, but this method will cause the background around the face to remain fixed. In practical applications, the background around digital humans often changes dynamically, so a fixed background will lead to disharmony between the digital human and the background, thereby affecting the overall visual effect.

[0006] In summary, on the basis of achieving lip-sync, the existing digital human driving methods still face challenges such as how to improve the coherence, stability, and naturalness of lip movements, and how to maintain the consistency between the digital human and the background in a dynamic background.

[0007] In view of this, the present application is proposed. Summary of the Invention

[0008] The present invention provides a digital human driving method and device based on secondary reasoning, which can at least partially improve the above problems.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A digital human driving method based on secondary reasoning, which includes:

[0011] Obtain video data and audio data to be reasoned, and call a preset digital human mouth alignment model to perform alignment preprocessing on the video data and audio data to obtain a fixed reference frame reasoning result;

[0012] Screen out multiple sliding face reference frames from the video data to obtain a sliding reference frame reasoning result;

[0013] Perform fusion processing on the sliding reference frame reasoning result and the fixed reference frame reasoning result to obtain a fusion result, and perform fusion processing on the fusion result and the video data to generate a final driving video.

[0014] The present invention also provides a digital human driving device based on secondary reasoning, which includes:

[0015] A fixed reference frame reasoning unit, configured to obtain video data and audio data to be reasoned, and call a preset digital human mouth alignment model to perform alignment preprocessing on the video data and audio data to obtain a fixed reference frame reasoning result;

[0016] A sliding reference frame reasoning unit, configured to screen out multiple sliding face reference frames from the video data to obtain a sliding reference frame reasoning result;

[0017] A reasoning result fusion unit, configured to perform fusion processing on the sliding reference frame reasoning result and the fixed reference frame reasoning result to obtain a fusion result, and perform fusion processing on the fusion result and the video data to generate a final driving video.

[0018] In summary, the digital human driving method based on secondary reasoning combines two methods of fixed reference frame reasoning and sliding reference frame reasoning; by obtaining a face mask and performing erosion processing, a rough mask of the mouth area is obtained, and the fixed reference frame reasoning result is fused onto the sliding reference frame reasoning result, and then the fused result is fused with the original video, thereby achieving stable and natural mouth actions. Among them, fixed reference frame reasoning ensures the stability of the mouth shape and the consistency of the tooth shape by selecting five fixed faces as input references, and sliding reference frame reasoning smooths the face edge and background by dynamically selecting five reference frames, enabling the face angle and background to change in real time.

[0019] This method combines the advantages of fixed reference frames and sliding reference frames, introduces face fusion technology and abnormal frame skipping strategy, effectively solves the problems in the prior art, realizes the coherence, stability and naturalness of digital lip movements, and at the same time maintains the consistency between the face and the background. This method is applicable to a variety of digital human application scenarios, providing new ideas and solutions for the development of digital human technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic flowchart of a digital human driving method based on secondary inference provided by the first embodiment of the present invention;

[0021] Figure 2 is an input-output schematic diagram of a digital human lip alignment model provided by an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of sliding reference frame inference provided by an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of the video frame input-output queue of the sliding process provided by an embodiment of the present invention;

[0024] Figure 5 is a module schematic diagram of a digital human driving device based on secondary inference provided by the second embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] Referring to Figure 1 As shown, the first embodiment of the present invention discloses a digital human driving method based on secondary inference, which can be executed by a digital human driving device based on secondary inference (hereinafter referred to as the driving device), and particularly, by one or more processors in the driving device to implement the following method:

[0027] S1. Obtain video data and audio data to be inferred, and call a preset digital human lip alignment model to perform alignment preprocessing on the video data and audio data to obtain a fixed reference frame inference result;

[0028] Preferably, the digital human lip alignment model includes an audio encoder, an image encoder, and the backbone network and decoder of the deep learning model GAN.

[0029] Specifically, step S1 includes: obtaining video data to be inferred and a segment of audio data unrelated to the video data, and randomly screening out five fixed face reference frames from the video data;

[0030] Convert the audio data into mel-spectrum features, and use a fixed-length mek sliding window to extract the mel-spectrum segments within the window;

[0031] Screen out the face images corresponding to the time of the mel-spectrum segments from the video data, and input the face image, five fixed face reference frames, and the mel-spectrum segments into the digital mouth shape alignment model. Among them, the face image is a face image with the mouth covered and the mouth shape to be changed;

[0032] Use an audio encoder to encode the mel-spectrum segments to obtain high-dimensional audio features. At the same time, use an image encoder to encode the face image and the fixed face reference frames to obtain high-dimensional image features;

[0033] Connect the high-dimensional audio features and the high-dimensional image features, and input them into the deep learning model GAN to generate multiple fixed face images corresponding to the audio data, and obtain the fixed reference frame inference result.

[0034] In this embodiment, the digital mouth shape alignment model is a complex and efficient system, which consists of the backbone network and decoder of an audio encoder, an image encoder, and the deep learning model GAN. The design of this structure makes full use of the advantages of different encoders, can accurately extract features from audio and image data respectively, and at the same time, with the powerful generation ability of GAN, generate face images matching the audio data, so as to achieve mouth shape alignment. Among them, from Figure 2 it can be seen that the digital mouth shape alignment model has two input channels, namely the image channel and the audio channel, while the output has only the image channel.

[0035] Specifically, in the implementation process, first, it is necessary to obtain the video data to be inferred and an audio data that is not related to the video data. This design of non-correlation between audio and video enables this method to be flexibly applied to various scenarios. Even if the audio content does not match the original video content, it can generate natural and coherent mouth movements, greatly expanding the application range of digital human driving. Immediately afterwards, randomly screen out five fixed face reference frames from the video data. The selection of these five reference frames is crucial, and they provide a stable basis for subsequent mouth shape generation, ensuring the stability and consistency of the generated mouth shape. By randomly screening, the problem of poor generation effect caused by improper selection of reference frames can be avoided to a certain extent.

[0036] Subsequently, the audio data is processed and converted into mel-spectrum features. The mel-spectrum is a feature that can effectively represent audio frequency information. It can capture key information in the audio, such as pitch, rhythm, etc., which is of great guiding significance for lip generation. A fixed-length mel sliding window is used to extract the mel-spectrum segments within the window. This sliding window method can ensure the continuity and temporal order of audio features, providing accurate time alignment information for subsequent lip generation. And a face image corresponding to the time of the mel-spectrum segment is selected from the video data. This face image is the face image with the mouth covered and the lip to be changed. This face image, five fixed face reference frames, and the mel-spectrum segment are input into the digital human lip alignment model together.

[0037] Inside the model, the audio encoder encodes the mel-spectrum segment and converts it into high-dimensional audio features. At the same time, the image encoder encodes the face image and the fixed face reference frames to obtain high-dimensional image features. The collaborative work of these two encoders enables the effective extraction and fusion of audio and image features. The high-dimensional audio features and high-dimensional image features are concatenated and then input into the deep learning model GAN. Under the action of the backbone network and decoder of GAN, multiple fixed face images corresponding to the audio data are generated. Among them, the lip of the output face is synchronized with this audio segment. These generated face images have lip movements matching the input audio, thus obtaining the fixed reference frame inference result. This process not only achieves the alignment of lip and audio, but also ensures the stability and consistency of lip movements through the use of fixed face reference frames, providing high-quality input for subsequent secondary inference.

[0038] It can be seen that the input image of the digital human lip alignment model includes five selected reference faces and one face with the mouth covered and the lip to be changed. The input audio is a segment of audio selected by the user and not related to this video. The video is decomposed into each frame image. The purpose of the model is to change the lip of the current frame face according to the audio at the current moment. Finally, all the output frame images are recombined into a video to achieve the alignment of the digital human lip in the video and an unrelated audio segment.

[0039] S2. Select multiple sliding face reference frames from the video data to obtain the sliding reference frame inference result;

[0040] Specifically, step S2 includes: According to the preset sliding reference frame window, five sliding face reference frames are selected from the video data to obtain the sliding reference frame inference result. Among them, the five sliding face reference frames are the current frame, the previous frame of the current frame, the frame before the previous frame of the current frame, the next frame of the current frame, and the frame after the next frame of the current frame;

[0041] Among them, both the background and the face angle of the image in the sliding reference frame inference result change in real time, and the mouth shape and tooth shape of the image in the fixed reference frame inference result are stable.

[0042] In this embodiment, during each round of fixed reference frame inference, the five reference frames input to the model remain unchanged, thus having the characteristics of stable mouth shape and tooth shape; since the reference frames are fixed, but the face edge will jitter, because the model cannot make precise adjustments to the face angle that changes in real time, and there is no real-time reference information for the background, which will also cause the background around the face to be distorted. Therefore, it is necessary to further perform sliding reference frame inference.

[0043] In this embodiment, the core of step S2 is to screen out five sliding face reference frames from the video data according to the preset sliding reference frame window. The selection of these five reference frames has clear rules, which are the current frame, the previous frame of the current frame, the frame before the previous frame of the current frame, the next frame of the current frame, and the frame after the next frame of the current frame. This design of the sliding window enables the reference frames to be dynamically updated as the video plays during each round of inference, thereby providing rich temporal information for the model. Refer to Figure 3 It can be seen that, contrary to the fixed reference frame, the five reference faces selected for the sliding reference frame change every time the model infers, and the line segment range during sliding Figure 3 moves forward one frame each time.

[0044] Please refer to Figure 3 , specifically, the sliding reference frame slides forward one frame each time. When the image input to the model is the first frame, the five sliding reference frames are the 1st, 2nd, 3rd, 4th, and 5th frames. When the image of the face whose mouth shape needs to be changed input to the model is the second frame, the five reference frames input to the model are still the 1st, 2nd, 3rd, 4th, and 5th frames; until when the face image of the face whose mouth shape needs to be changed input to the model is the 4th frame, the five reference frames input to the model are the 2nd, 3rd, 4th, 5th, and 6th frames; when the face image of the face whose mouth shape needs to be changed input to the model is the 5th frame, the five reference frames input to the model are the 3rd, 4th, 5th, 6th, and 7th frames; when the face image of the face whose mouth shape needs to be changed input to the model is the 6th frame, the five reference frames input to the model are the 4th, 5th, 6th, 7th, and 8th frames, and so on.

[0045] When performing sliding reference frame inference, the model dynamically adjusts the face angle and background according to the information of the current frame and its front and back frames. This dynamic adjustment not only makes the face edge smoother and avoids the jitter phenomenon, but also makes the fusion of the background and the face more natural and avoids the distortion problem caused by the fixed background. In this way, while maintaining the coherence of the mouth shape movement, the sliding reference frame inference result significantly improves the overall visual effect of the digital human image.

[0046] Briefly speaking, during each round of fixed reference frame inference, the five reference frames input into the model remain unchanged, which ensures the stability of the mouth shape and tooth shape. However, this fixity also brings limitations. The model cannot make precise adjustments to the real-time changing face angles and cannot provide real-time changing background information. Therefore, the implementation of sliding reference frame inference is particularly important. By dynamically updating the reference frames, it provides the model with real-time changing face angles and background information, making the generated images more natural and coherent in these aspects. The sliding reference frame inference results obtained in this way have significant characteristics: both the background and face angles of the images change in real time. This forms a sharp contrast with the fixed reference frame inference results. Although the latter shows stability in the mouth shape and tooth shape, due to the fixed reference frames, jitter appears at the face edges, and the background is distorted due to the lack of real-time reference. The introduction of sliding reference frame inference effectively solves these problems, making the generated digital human images more natural and coherent in terms of background and face angles.

[0047] S3. Fuse the sliding reference frame inference result and the fixed reference frame inference result to obtain a fusion result, and fuse the fusion result with the video data to generate a final driving video.

[0048] Specifically, step S3 includes: The mathematical expression of the fusion process is: pre1 = pre1 * (1 - mask2) + pre2 * mask2, result = src * (1 - mask1) + pre1 * mask1, where result is the fusion result, pre1 is the sliding reference frame inference result, pre2 is the fixed reference frame inference result, mask1 is the face mask in the video data to be inferred, mask2 is the result after eroding mask1 by a preset number of pixels, and src is the face in the video data.

[0049] Preferably, the preset number is 20.

[0050] In this embodiment, since the background of the sliding reference frame changes in real time and the face angle also changes in real time, the model can better obtain the changing information, the generated face edges and background will be smoothed, and the overall video effect will be stable. However, since the tooth information of the reference frame also changes in real time, the mouth area and tooth shape generated by the model will show inconsistent changes. Therefore, it is necessary to fuse the sliding reference frame inference result and the fixed reference frame inference result.

[0051] Specifically, in this embodiment, the sliding reference frame inference result pre1 and the fixed reference frame inference result pre2 are fused to obtain a preliminary fusion result. The fusion formula is: pre1 = pre1 * (1 - mask2) + pre2 * mask2. Here, mask1 is the face mask in the video data to be inferred, and mask2 is the result after eroding mask1 by a preset number of pixels. Preferably, the preset number is 20 pixels. This is because the input image is uniformly resized to 512, and the distance between the mouth area and the face boundary is usually greater than 20 pixels. Therefore, eroding the face mask by 20 pixels can obtain the approximate area of the mouth.

[0052] The core of this fusion process is to mask the mouth area of pre2 through mask2, and integrate the stable and coherent mouth shape in the fixed reference frame inference result into the sliding reference frame inference result. This fusion method not only retains the real-time changes in the background and face angle in the sliding reference frame inference result, but also uses the stability of the fixed reference frame inference result to solve the problem of incoherence in the mouth area that may occur in the sliding reference frame inference.

[0053] Subsequently, the preliminary fusion result is further fused with the original video data src to obtain the final fusion result result. The fusion formula is: result = src * (1 - mask1) + pre1 * mask1. This process fuses the fused result (which already includes the stable mouth area and real-time changing background and face angle) with the original video data through mask1, so that the generated digital human image not only maintains the coherence and stability of the mouth movement, but also highly conforms to the background and the surrounding environment of the face in the original video. This fusion method effectively avoids problems such as distortion of the background around the face and jitter of the face edge, and significantly improves the visual effect of digital human driving.

[0054] Briefly speaking, obtain the face mask, erode the face mask by 20 pixels to obtain the approximate mask of the mouth area. Fuse the fixed reference frame inference result in step S1 to the face in the sliding reference frame inference result in step S2 through the mouth mask, and then fuse the obtained result into the original video. This realizes the complementary advantages of the two inference methods. The sliding reference frame inference result provides real-time changing background and face angle, making the digital human image more natural and smooth; the fixed reference frame inference result ensures the stability and coherence of the mouth shape and tooth shape. The fusion processing not only solves the limitations of a single inference method, but also further optimizes the effect of digital human driving, making it more realistic and natural visually. In addition, through the precise control of the preset pixel erosion value (such as 20 pixels), the size of the fusion area can be flexibly adjusted to better meet the requirements in different scenarios.

[0055] Preferably, when screening out five fixed face reference frames from the video data, abnormal frames in the video data are excluded, specifically:

[0056] Use a face detection model and a face key point detection model to perform abnormal detection processing on each video frame in the video data;

[0057] When the face detection model determines that the face is blocked by the hand in the video frame, or when the face key point detection model determines that there are exaggerated or expression and actions beyond the model's inference limit on the face in the video frame, this video frame is marked as an abnormal frame.

[0058] In this embodiment, due to insufficient training data and its own driving ability limitations, the digital mouth alignment model may not have a good driving effect on some abnormal frames. Abnormal frames include situations where the hand blocks the face occasionally in the video, the mouth suddenly opens very wide, the chin is extremely long, or strange expressions appear. To further improve the stability and reliability of digital human driving, especially when screening fixed face reference frames, an abnormal frame exclusion mechanism is introduced. This mechanism performs abnormal detection processing on each frame in the video data by using a face detection model and a face key point detection model, thereby ensuring the quality and representativeness of the selected fixed face reference frames.

[0059] Specifically, when screening out five fixed face reference frames from the video data, first use the face detection model to detect each frame. This model can accurately identify the position of the face in the video frame and determine whether there is a situation where the face is blocked by the hand. If the face detection model detects that the hand blocks the face, this frame will be marked as an abnormal frame. This detection mechanism can effectively avoid incorrect mouth shape generation caused by hand occlusion and ensure the clarity and integrity of the fixed face reference frames.

[0060] At the same time, the face key point detection model is also used to analyze each frame. This model can identify the key points of the face in the video frame, such as eyes, nose, mouth, etc., and determine whether the expressions and actions shown by these key points are exaggerated or beyond the model's inference limit. If the face key point detection model determines that there are exaggerated or expression and actions beyond the model's inference limit on the face in the video frame, this frame will also be marked as an abnormal frame. This detection mechanism can effectively avoid inaccurate mouth shape generation caused by abnormal expressions or actions and ensure the stability and consistency of the fixed face reference frames.

[0061] Through this dual detection mechanism, the present invention can effectively exclude abnormal frames in video data, thereby ensuring that the selected fixed face reference frames have high quality and representativeness. The introduction of this abnormal frame exclusion mechanism not only improves the reliability of the fixed face reference frames, but also further optimizes the effect of digital human driving. During the subsequent inference process, due to the guaranteed quality of the fixed face reference frames, the generation of the digital human's mouth shape will be more stable and coherent, avoiding mouth shape jitter or unnatural phenomena caused by abnormal frames. In addition, this abnormal frame exclusion mechanism also has significant beneficial effects. First of all, it can improve the robustness of digital human driving, and can generate stable and natural mouth shape movements even in complex video scenarios. Secondly, by excluding abnormal frames, the waste of computing resources when the model processes abnormal data is reduced, and the efficiency of the entire system is improved. Finally, this mechanism can also enhance the visual effect of digital human driving, making the generated digital human images more realistic and natural, and better meeting the requirements of various application scenarios.

[0062] Please refer to Figure 4 , preferably, during the sliding process of the sliding reference frame window, when it is determined that the video frame within the window is an abnormal frame, the sliding stops.

[0063] In this embodiment, during the sliding process of the sliding reference frame window, when the sliding reference frame window detects an abnormal frame, the sliding immediately stops, and its specific implementation method is a queue; the introduction of this stop mechanism can effectively avoid the influence of abnormal frames on the inference result of the sliding reference frame. Since the inference of the sliding reference frame depends on the information of consecutive frames within the window, the existence of abnormal frames may cause incorrect or unnatural changes in the generated face angle and background information. By stopping the sliding, it can be ensured that the frames within the window are all normal frames, thereby guaranteeing the accuracy and stability of the inference result of the sliding reference frame. On the contrary, when a normal frame is encountered, enqueueing and dequeueing operations are performed, and when an abnormal frame is encountered, the enqueueing and dequeueing operations stop.

[0064] This method avoids the ineffective processing of abnormal frames by stopping the sliding in a timely manner, reducing the waste of computing resources. This makes the entire system more efficient when processing complex video data and can generate high-quality digital human driving videos faster.

[0065] In summary, the digital human driving method based on secondary reasoning proposed by the present invention aims to solve the problems of insufficient coherence and poor stability of digital human mouth shape movements, as well as the inconsistency between the face and the background in the prior art. Through innovative reasoning mechanisms and fusion technologies, the visual effect and application value of digital human driving have been significantly improved. This method first performs reasoning with fixed reference frames. By using five fixed face reference frames, it ensures the stability of mouth shape movements and the consistency of tooth shapes. The use of these fixed reference frames provides a stable benchmark for digital human mouth shape generation, enabling the generated mouth shapes to maintain coherence in the time dimension and avoiding jitter or unnatural phenomena caused by real-time changes. However, the limitation of fixed reference frames is that the face edges and the background cannot be updated in real time, resulting in background distortion and face edge jitter problems.

[0066] To solve the above problems, sliding reference frame reasoning is further introduced. By dynamically selecting the current frame and its adjacent frames as reference frames, sliding reference frame reasoning can update the face angle and background information in real time, making the generated images more natural and smooth in terms of the background and face angle. This dynamic update mechanism effectively avoids background distortion and face edge jitter problems, significantly improving the overall visual effect of digital human images.

[0067] Furthermore, to fully utilize the advantages of fixed reference frames and sliding reference frames, a face fusion technology with two inferences is introduced. Through specific mathematical formulas, the stable mouth shape in the inference result of fixed reference frames is fused with the real-time changing background and face angle in the inference result of sliding reference frames. This fusion method not only retains the advantages of the two inference methods but also further optimizes the effect of digital human driving, making the generated digital human images more realistic and natural visually.

[0068] In addition, the digital human driving method based on secondary reasoning also involves an abnormal frame detection and processing mechanism. When screening fixed face reference frames, abnormal frames are excluded through a face detection model and a face key point detection model to ensure the quality and representativeness of the reference frames. During the sliding process of the sliding reference frame window, an abnormal frame detection mechanism is also used to stop the sliding in a timely manner to avoid the influence of abnormal frames on the inference result. This mechanism significantly improves the robustness of digital human driving, ensuring that stable and natural mouth shape movements can be generated even in complex video scenarios.

[0069] Briefly speaking, the digital human driving method based on secondary reasoning effectively solves the deficiencies in the prior art through fixed reference frame reasoning, sliding reference frame reasoning, and face fusion technology for two inferences, achieving the coherence, stability, and naturalness of the digital human's lip movement. At the same time, through the abnormal frame detection and processing mechanism, the robustness and visual effect of the system are further improved. This innovative digital human driving method not only provides new ideas for the development of digital human technology but also has important application value and broad development prospects, and is applicable to a variety of complex digital human application scenarios.

[0070] Please refer to Figure 5 , the second embodiment of the present invention provides a digital human driving device based on secondary reasoning, which includes:

[0071] A fixed reference frame reasoning unit 201, configured to obtain video data and audio data to be reasoned, call a preset digital human lip alignment model to perform alignment preprocessing on the video data and audio data, and obtain a fixed reference frame reasoning result;

[0072] A sliding reference frame reasoning unit 202, configured to screen out multiple sliding face reference frames from the video data and obtain a sliding reference frame reasoning result;

[0073] An inference result fusion unit 203, configured to perform fusion processing on the sliding reference frame reasoning result and the fixed reference frame reasoning result to obtain a fusion result, and perform fusion processing on the fusion result and the video data to generate a final driving video.

[0074] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A digital human driving method based on secondary reasoning, characterized in that, Including: Obtain video data and audio data to be inferred, and call a preset digital lip-sync alignment model to perform alignment preprocessing on the video data and audio data to obtain a fixed reference frame inference result; Select multiple sliding face reference frames from the video data to obtain a sliding reference frame inference result; Fuse the sliding reference frame inference result and the fixed reference frame inference result to obtain a fusion result, and fuse the fusion result with the video data to generate a final driving video.

2. The digital human driving method based on secondary reasoning according to claim 1, wherein, The digital lip-sync alignment model includes an audio encoder, an image encoder, and the backbone network and decoder of the deep learning model GAN.

3. The digital human driving method based on secondary reasoning according to claim 2, wherein Obtain video data and audio data to be inferred, and call a preset digital lip-sync alignment model to perform alignment preprocessing on the video data and audio data to obtain a fixed reference frame inference result. Specifically: Obtain video data to be inferred and a piece of audio data unrelated to the video data, and randomly select five fixed face reference frames from the video data; Convert the audio data into mel spectrogram features, and use a fixed-length mek sliding window to extract mel spectrogram segments within the window; Select a face image corresponding to the time of the mel spectrogram segment from the video data, and input the face image, five fixed face reference frames, and the mel spectrogram segment into the digital lip-sync alignment model, where the face image is a face image with the mouth covered and the lip shape to be changed; Use the audio encoder to encode the mel spectrogram segment to obtain high-dimensional audio features. At the same time, use the image encoder to encode the face image and the fixed face reference frames to obtain high-dimensional image features; Connect the high-dimensional audio features and the high-dimensional image features, and input them into the deep learning model GAN to generate multiple fixed face images corresponding to the audio data to obtain a fixed reference frame inference result.

4. The digital human driving method based on secondary reasoning according to claim 3, characterized in that Select multiple sliding face reference frames from the video data to obtain a sliding reference frame inference result. Specifically: According to a preset sliding reference frame window, select five sliding face reference frames from the video data to obtain a sliding reference frame inference result, where the five sliding face reference frames are the current frame, the previous frame of the current frame, the frame before the previous frame of the current frame, the next frame of the current frame, and the frame after the next frame of the current frame; Among them, the background and face angle of the image in the sliding reference frame inference result are changing in real time, and the lip shape and tooth shape of the image in the fixed reference frame inference result are stable.

5. The digital human driving method based on secondary reasoning according to claim 4, wherein During the sliding of the sliding reference frame window, when it is determined that the video frame within the window is an abnormal frame, stop sliding.

6. The digital human driving method based on secondary reasoning according to claim 1, wherein The mathematical expression for the fusion process is: pre1 = pre1 * (1 - mask2) + pre2 * mask2, result = src * (1 - mask1) + pre1 * mask1, where result is the fusion result, pre1 is the inference result of the sliding reference frame, pre2 is the inference result of the fixed reference frame, mask1 is the face mask in the video data to be inferred, mask2 is the result after eroding mask1 by a preset number of pixels, and src is the face in the video data.

7. The digital human driving method based on secondary reasoning according to claim 6, wherein The preset number is 20.

8. The digital human driving method based on secondary reasoning according to claim 3, wherein When screening out five fixed face reference frames from the video data, abnormal frames in the video data are excluded. Specifically: An abnormal detection process is performed on each video frame in the video data using a face detection model and a face key point detection model; When the face detection model determines that the face is covered by the hand in the video frame, or when the face key point detection model determines that there are exaggerated or actions and expressions beyond the inference limit of the model on the face in the video frame, this video frame is marked as an abnormal frame.

9. A digital human driving device based on secondary reasoning, characterized in that, It includes: A fixed reference frame inference unit, which is used to obtain the video data and audio data to be inferred, and call a preset digital lip synchronization model to perform alignment preprocessing on the video data and audio data to obtain the fixed reference frame inference result; A sliding reference frame inference unit, which is used to screen out multiple sliding face reference frames from the video data to obtain the sliding reference frame inference result; An inference result fusion unit, which is used to perform a fusion process on the sliding reference frame inference result and the fixed reference frame inference result to obtain a fusion result, and perform a fusion process on the fusion result and the video data to generate the final driving video.

Images