Digital human video action continuous splicing method and device for live streaming scene

Abstract

The application provides a digital human video action continuous splicing method and device for live streaming scene, and relates to the technical field of virtual digital human, which comprises the following steps: acquiring a digital human video in which a digital human alternately keeps still and performs actions, and constructing a frame track; when the state of the character is a still state, every frame is directly added to the tail of a playing queue; when the state of the character is a speaking state, a continuous video frame segment is processed by a video frame processing module, and then the processed continuous video frame segment is sequentially added to the tail of the playing queue; when the state of the character is a still state, the extracted video frame is a still frame; when the state of the character is a speaking state, the last frame of all the extracted video frames is a still frame; the content of the playing queue is displayed frame by frame, and the original audio is synchronously played. The application can reduce picture jitter when the action of the digital human is switched, and improve the stability and natural performance of the digital human in live streaming.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual digital humans, and in particular to a method and device for continuously splicing digital human video actions in a streaming live broadcast scene. Background Art

[0002] Currently, the video coherence of virtual characters has become a crucial factor affecting user experience in scenarios such as news broadcasting, online customer service, education and training, and live entertainment. To generate rich digital human videos, a common method in existing technologies is to construct a complete action sequence by splicing pre-produced video clips.

[0003] However, these solutions primarily focus on combining and reusing motion content. When switching between video clips, especially when transitioning between static and motion states, they often lack a smooth transition mechanism. This direct splicing approach can easily lead to noticeable jumps, jitters, or freezes in the final synthesized image, disrupting visual continuity and naturalness. Furthermore, when synthesizing videos, if the playback rhythm of the video frames isn't precisely aligned with the speech rhythm of the audio being synthesized, lip movements can easily become out of sync with the sound, severely impacting the realism of the digital human and the user's immersive experience.

[0004] Based on this, how to reduce the image jitter when the digital human switches movements and improve the stability and natural expressiveness of the digital human in live streaming has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of this, in order to solve the above technical problems, the present invention provides a method and device for continuously splicing digital human video movements in a streaming live scene.

[0006] The present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for continuously splicing digital human video actions in a live streaming scene, comprising:

[0008] Obtaining a pre-recorded digital human video, in which the digital human alternately remains still and performs actions;

[0009] Dividing the digital human video into still frame segments and action frame segments, and labeling each frame to obtain a frame track; the labels include still and action;

[0010] Setting a frame pointer on the frame track, wherein the frame pointer is used to slide back and forth on the frame track to locate and extract the video frame, and the frame pointer is initially pointed to the still frame segment;

[0011] During the video playing process, the frame pointer is controlled to slide according to the character state of the digital person, and when the character state is a static state, a frame is directly added to the tail of a playing queue every time, and when the character state is a speaking state, after a continuous video frame segment extracted is processed by a video frame processing module, the processed continuous video frame segment is sequentially added to the tail of the playing queue; when the character state is a static state, all the extracted video frames are static frames, and when the character state is a speaking state, the last frame of all the extracted video frames is a static frame; when there is corresponding original audio, the character state is a speaking state, and when there is no corresponding original audio, the character state is a static state; the original audio is divided into a plurality of audio segments, and the audio segments correspond to the continuous video frame segments one by one;

[0012] The content of the playing queue is displayed frame by frame, and the original audio is played synchronously.

[0013] Optionally, the frame pointer is controlled to slide according to the character state of the digital person, and specifically includes:

[0014] When the character state is a static state, the frame pointer is controlled to slide forward at a preset frame rate, and when the frame pointer slides to the edge of the current static frame segment, the frame pointer is controlled to slide reversely;

[0015] When the character state is a speaking state, each audio segment is acquired, and video frames are extracted from each audio segment in time sequence, wherein, for the current processed audio segment, if the audio segment is not the last audio segment, the frame pointer is controlled to slide forward at a preset frame rate, and when the frame pointer slides to the edge of the frame track, the frame pointer is controlled to slide reversely until a continuous video frame segment with a playing time length matching a time length of the audio segment is extracted, if the audio segment is the last audio segment, the frame pointer is controlled to slide into the nearest static frame segment, and after the frame pointer slides into the static frame segment, the frame pointer is controlled to slide according to the strategy when the character state is a static state until a continuous video frame segment with a playing time length matching a time length of the audio segment is extracted.

[0016] Optionally, each audio segment is acquired, and specifically includes:

[0017] Each audio segment divided in advance is acquired;

[0018] The dividing process of the audio segment includes:

[0019] The original audio is acquired; the original audio is complete audio data or an audio data stream with a determined end determined by any manner;

[0020] The original audio is divided into a plurality of audio segments, and the audio segments are numbered.

[0021] Optionally, the original audio is divided into a plurality of audio segments, specifically comprising:

[0022] The original audio is divided into a plurality of audio segments with equal audio length based on a time axis; or,

[0023] The original audio is divided into a plurality of audio segments with equal number of broadcast words based on audio content; or,

[0024] The original audio is divided into a plurality of audio segments with punctuation marks as division marks; or,

[0025] The original audio is divided into a plurality of audio segments with punctuation marks as division marks and based on a preset word threshold, so that the number of words in each audio segment is greater than the preset word threshold.

[0026] Optionally, the video frame processing module is specifically an AI technology-based video frame processing module, comprising:

[0027] A face detection module is configured to acquire face position information of each video frame in the current continuous video frame segment;

[0028] A lip shape generation module is configured to generate a corresponding lip shape map according to the current audio segment;

[0029] A map fusion processing module is configured to paste the lip shape map back to the corresponding video frame according to the face position information.

[0030] Optionally, the video frame processing module further comprises:

[0031] A super-resolution processing module is configured to improve the resolution of the video frame.

[0032] Optionally, the video frame processing module further comprises:

[0033] A style transfer processing module is configured to add a specific artistic style to the video frame; or,

[0034] An effect processing module is configured to add a specific special effect to the video frame.

[0035] Optionally, the frame track starts with one still frame segment and ends with one still frame segment.

[0036] In a second aspect, the present application further provides a digital human video action continuous splicing device for a live streaming scenario, comprising:

[0037] A digital human video acquisition module is configured to acquire a pre-recorded digital human video, in which the digital human alternately stays still and performs actions;

[0038] constructing a frame track module for dividing the digital human video into a still frame segment and a motion frame segment, and marking a label frame by frame to obtain a frame track; the label includes still and motion;

[0039] setting a frame pointer module for setting a frame pointer on the frame track, the frame pointer being used for sliding forward and backward on the frame track to realize positioning and extracting video frames, the frame pointer being initialized to point to the still frame segment;

[0040] a frame extraction and processing module for controlling the frame pointer to slide according to the character state of the digital human during video playing, and directly adding a frame to the tail of a playing queue every time a frame is extracted when the character state is a still state, and adding a processed continuous video frame segment to the tail of the playing queue in sequence after processing the continuous video frame segment extracted through a video frame processing module when the character state is a speaking state; wherein, when the character state is a still state, all the extracted video frames are still frames, and when the character state is a speaking state, the last frame of all the extracted video frames is a still frame; when there is a corresponding original audio, the character state is a speaking state, and when there is no corresponding original audio, the character state is a still state; the original audio is divided into a plurality of audio segments, and the audio segments correspond to the continuous video frame segments one by one;

[0041] a playing module for displaying the content of the playing queue frame by frame and playing the original audio synchronously.

[0042] The application adopts the above technical scheme, constructs a frame track in which a still frame segment and a motion frame segment are alternately arranged, extracts video frames from the frame track using different frame extraction strategies when the character state is a still state or a speaking state, and puts the extracted video frames to be played into a playing queue to directly play the continuous frames in the playing queue, so that the application can reduce picture jitter when the digital human motion is switched, improve the continuity of the digital human motion, and further improve the stability and natural performance of the digital human in live streaming, and is especially suitable for virtual digital human live streaming application scenarios that need long-time continuous broadcasting. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0044] Figure 1 is a flowchart of a digital human video motion continuous splicing method in a live streaming scenario provided by an embodiment of the present application.

[0045] Figure 2 is a frame track and play queue combined structure schematic diagram provided by an embodiment of the present application;

[0046] Figure 3 is a structure schematic diagram of a digital human video action continuous splicing device for a live streaming scene provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of the present application.

[0048] Figure 1 is a flow schematic diagram of a digital human video action continuous splicing method for a live streaming scene provided by an embodiment of the present application. Figure 2 is a frame track and play queue combined structure schematic diagram provided by an embodiment of the present application. As shown in Figure 1 and Figure 2 , the present flow includes:

[0049] Step 101: obtaining a pre-recorded digital human video, in which a digital human alternately stands still and performs actions.

[0050] Specifically, the digital human alternately stands still and performs actions, forming structured segments, the beginning segment and the ending segment of the digital human video are both still segments, and the time length of each segment is less than a preset time length threshold. In a specific example, when recording the digital human video, the digital human stands still for 10 seconds, then starts to perform actions such as waving hands, waving, and nodding for 10 seconds, and then stands still for 10 seconds, and so on, alternately, until the recording is completed.

[0051] Step 102: dividing the digital human video into still frame segments and action frame segments, and marking labels frame by frame to obtain a frame track; the labels include still and action. Establishing the frame track can realize sequentially arranging all video frames.

[0052] Specifically, the frames in the still frame segments are all still frames, and the frames in the action frame segments are all action frames, each frame segment contains a plurality of continuous frames, and by marking the labels frame by frame, it is convenient and fast to confirm whether any frame in the frame track is a still frame or an action frame.

[0053] As shown in Figure 2 , the frame track 21 is composed of a plurality of alternately arranged still frame segments and action frame segments Figure 2Take four still frame segments and three action frame segments as an example for illustration. And the frame track starts with a still frame segment and ends with a still frame segment, so as to stabilize the start and end of the system. The still frame segment is used for static display and static voice broadcast of the digital human without voice input, and the action frame segment is used for natural action display during the digital human speaking.

[0054] Step 103: Set a frame pointer 22 on the frame track, which can slide forward and backward on the frame track. The frame pointer is used to slide forward and backward on the frame track to realize positioning and extraction of video frames. When the frame pointer is initialized, it points to the still frame segment. When the frame track starts with a still frame segment and ends with a still frame segment, the frame pointer is initialized to point to the first frame segment of the frame track.

[0055] Step 104: During video playing, control the frame pointer to slide according to the character state of the digital human, and when the character state is a still state, add each extracted frame directly to the tail of the playing queue 24, and when the character state is a speaking state, after processing the extracted continuous video frame segment by the video frame processing module 23, add the processed continuous video frame segment to the tail of the playing queue 24 in order; wherein when the character state is a still state, all the extracted video frames are still frames, and when the character state is a speaking state, the last frame of all the extracted video frames is a still frame, so as to ensure that the video played at the end of speaking is a natural static picture; when there is corresponding original audio, the character state is a speaking state, and when there is no corresponding original audio, the character state is a still state; the original audio is divided into multiple audio segments, and the audio segments correspond to the continuous video frame segments one by one.

[0056] Step 105: Display the content of the playing queue frame by frame and play the original audio synchronously.

[0057] The playing queue can adopt a FIFO (First In, First Out) structure to ensure the ordered output of video frames, serving as the final digital human live broadcast playing source.

[0058] The application adopts the above technical scheme, constructs a frame track in which still frame segments and action frame segments are alternately arranged, uses different frame extraction strategies to extract video frames from the frame track when the character state is a still state or a speaking state, and puts the extracted to-be-played video frames into a playing queue. The application directly plays the continuous frames in the playing queue, so that the application can reduce the picture jitter when the digital human action switches, reduce the jitter amplitude by more than 85% through actual measurement, improve the continuity of the digital human action, and further improve the stability and natural expressiveness of the digital human in live streaming. The application can be widely applied to virtual human live broadcast, digital human broadcast, intelligent customer service and other scenes, and is especially suitable for virtual digital human live broadcast application scenes that need long-time continuous broadcast.

[0059] In addition, the whole process of the application does not need manual intervention, improves the degree of automation, supports real-time dynamic adjustment, is suitable for complex environments such as long-time live broadcast and multi-scene switching, and the system stability is improved by more than 90%.

[0060] In the embodiment of the application, the frame pointer is controlled to slide according to the character state of the digital human, and specifically can include:

[0061] (1) When the character state is a static state, the frame pointer is controlled to slide forward at a preset frame rate, and when it slides to the edge of the current static frame segment, it is reversed to slide, so as to ensure the stability of the picture.

[0062] Specifically, the preset frame rate can be 25 frames per second. And when the character state is a static state, the length of the play queue can be limited to 1.

[0063] It should be noted that since the frame pointer is initialized to point to the static frame segment, and at the end of the last audio segment (i.e. at the end of the speech), the frame pointer also points to the static frame segment, therefore, when the character state is a static state, the frame pointer is necessarily in the static frame segment.

[0064] (2) When the character state is a speech state, each audio segment is obtained, and video frames are extracted from each audio segment in time sequence, wherein, for the currently processed audio segment, if the audio segment is not the last audio segment, the frame pointer is controlled to slide forward at a preset frame rate, and when it slides to the edge of the frame track, it is reversed to slide, until a continuous video frame segment with a play duration matching the audio segment duration is extracted, if the audio segment is the last audio segment, the frame pointer is controlled to slide into the nearest static frame segment, and after sliding into the static frame segment, the frame pointer is controlled to slide according to the strategy when the character state is a static state, until a continuous video frame segment with a play duration matching the audio segment duration is extracted. The length of the continuous video frame segment is calculated according to the duration of the corresponding audio segment and the preset frame rate.

[0065] In this scheme, the position of the frame pointer is updated in real time, so that the extracted frame segment can follow the play progress. Among them, when the play queue is empty, the frame pointer is moved, the frame is continuously extracted, and the play queue is added.

[0066] The embodiment of the application adopts the above scheme, since at the end of the last audio segment (i.e. at the end of the speech), the frame pointer points to the static frame segment, so that from the visual effect, the video playing process is more smooth and natural. And by extracting video frames from each audio segment in time sequence, a continuous video frame segment with a play duration matching the audio segment duration is extracted, which is added to the play queue after being processed by the video frame processing module, so that the application can realize audio and video synchronous playing.

[0067] In the embodiment of the application, each audio segment is obtained, and specifically can include:

[0068] obtaining each audio segment divided in advance.

[0069] The dividing process of the audio segment includes:

[0070] (1) obtaining original audio; the original audio is complete audio data or an audio data stream determined by any method.

[0071] Specifically, the original audio can be obtained from an audio file, an audio acquisition device or a network stream, etc. In actual application scenarios, the audio stream is continuously segmented during the generation process, and the last segment can be confirmed until the generation is completed. The last segment is marked.

[0072] (2) dividing the original audio into multiple audio segments and numbering the audio segments.

[0073] In the embodiment of the application, the original audio is divided into multiple audio segments, which can specifically include:

[0074] The original audio is divided into multiple audio segments of equal audio length based on a time axis, for example, one segment is divided every 5 seconds. Or,

[0075] The original audio is divided into multiple audio segments of equal number of broadcast characters based on audio content, for example, one segment is divided every 20 characters. Or,

[0076] The original audio is divided into multiple audio segments using punctuation marks as division marks, that is, one segment is divided every time a punctuation mark is encountered. Or,

[0077] The original audio is divided into multiple audio segments using punctuation marks as division marks and based on a preset character number threshold, so that the number of characters in each audio segment is greater than the preset character number threshold. For example, more than 20 characters and a punctuation mark are divided into one segment.

[0078] In the embodiment of the application, the video frame processing module is specifically an AI technology-based video frame processing module, which includes multiple deep learning modules that work cooperatively, and specifically includes:

[0079] A face detection module for obtaining face position information of each video frame in the current continuous video frame segment.

[0080] A mouth shape generation module for generating a corresponding mouth shape map according to the current audio segment.

[0081] A map fusion processing module for pasting the mouth shape map back to the corresponding video frame according to the face position information.

[0082] The embodiment of the present invention adopts the above-mentioned solution. By detecting the facial position information, generating a lip map and pasting the lip map back to the corresponding video frame based on the facial position information, the present invention can improve the synchronization between lip shape and speech. The measured lip shape and speech synchronization error is ≤50ms, which enhances the realism of the digital human.

[0083] In addition, the video frame processing module may also include:

[0084] The super-resolution processing module is used to increase the resolution of video frames, specifically up to 4K. This improves image clarity and enhances the quality and visual quality of the output video.

[0085] In addition, the video frame processing module may also include:

[0086] The style transfer processing module is used to add specific artistic styles to video frames, such as oil painting and ink painting. The oil painting style can give the digital human's skin and clothing a brushstroke feel, such as Rembrandt's light and shadow effects. The ink painting style is suitable for digital humans with traditional Chinese style, creating a freehand effect of brushstrokes and white space. Alternatively, the special effects processing module is used to add specific special effects to video frames, such as dynamic mapping and virtual-real light and shadow matching. Dynamic mapping can overlay virtual decorations such as cat ears, orc features, and mechanical visors in real time. Virtual-real light and shadow matching can adjust the lighting direction and shadows of the digital human to make it blend naturally with the real scene. This can enrich the visual effects and further improve the quality and visual quality of the output video.

[0087] Based on a general inventive concept, the present invention also provides a device for continuously splicing digital human video actions in a streaming live scene. Figure 3 This is a structural diagram of a device for continuously splicing digital human video actions in a live streaming scene provided by an embodiment of the present invention. Figure 3 As shown, the device includes:

[0088] A digital human video acquisition module 31 is used to acquire pre-recorded digital human videos, in which the digital human alternates between being still and performing actions;

[0089] A frame track building module 32 is used to divide the digital human video into still frame segments and action frame segments, and mark labels frame by frame to obtain frame tracks; the labels include still and action;

[0090] A frame pointer module 33 is provided for setting a frame pointer on the frame track. The frame pointer is used to slide back and forth on the frame track to locate and extract the video frame. When the frame pointer is initialized, it points to a still frame segment.

[0091] The frame extraction and processing module 34 is configured to control the frame pointer to slide during video playing according to the character state of the digital human, and when the character state is a static state, add a frame directly to the tail of the playing queue every time a frame is extracted, and when the character state is a speaking state, add the processed continuous video frame segment to the tail of the playing queue in sequence after processing the extracted continuous video frame segment by the video frame processing module; when the character state is a static state, all the extracted video frames are static frames, and when the character state is a speaking state, the last frame of all the extracted video frames is a static frame; when there is corresponding original audio, the character state is a speaking state, and when there is no corresponding original audio, the character state is a static state; the original audio is divided into a plurality of audio segments, and the audio segments correspond to the continuous video frame segments in a one-to-one manner.

[0092] The playing module 35 is configured to display the content of the playing queue frame by frame and play the original audio synchronously.

[0093] Optionally, the frame extraction and processing module 34 can specifically include:

[0094] The first sliding unit is configured to control the frame pointer to slide forward at a preset frame rate when the character state is a static state, and slide reversely when the frame pointer slides to the edge of the current static frame segment.

[0095] The second sliding unit is configured to acquire each audio segment when the character state is a speaking state, and extract video frames from each audio segment in time sequence, wherein for the currently processed audio segment, if the audio segment is not the last audio segment, the frame pointer is controlled to slide forward at a preset frame rate, and slide reversely when the frame pointer slides to the edge of the frame track, until a continuous video frame segment with a playing time length matching the time length of the audio segment is extracted, if the audio segment is the last audio segment, the frame pointer is controlled to slide into the nearest static frame segment, and the frame pointer is controlled to slide according to the strategy when the character state is a static state after the frame pointer slides into the static frame segment, until a continuous video frame segment with a playing time length matching the time length of the audio segment is extracted.

[0096] Optionally, the second sliding unit can be specifically configured to:

[0097] Acquire each audio segment divided in advance.

[0098] The device can further include:

[0099] The original audio acquisition module is configured to acquire original audio; the original audio is complete audio data or an audio data stream with an end determined by any manner.

[0100] The audio division module is configured to divide the original audio into a plurality of audio segments, and number the audio segments.

[0101] Optionally, the audio division module can be specifically configured to:

[0102] divide the original audio into a plurality of audio segments with equal audio length based on a timeline; or

[0103] divide the original audio into a plurality of audio segments with equal number of words based on audio content; or

[0104] divide the original audio into a plurality of audio segments based on punctuation marks as division marks; or

[0105] divide the original audio into a plurality of audio segments based on punctuation marks as division marks and based on a preset word threshold, so that the number of words in each audio segment is greater than the preset word threshold.

[0106] Optionally, the video frame processing module is an AI technology-based video frame processing module, and includes:

[0107] a face detection module configured to obtain face position information of each video frame in the current continuous video frame segment.

[0108] a lip shape generation module configured to generate a corresponding lip shape map according to the current audio segment.

[0109] a map fusion processing module configured to paste the lip shape map back to the corresponding video frame according to the face position information.

[0110] Optionally, the video frame processing module further includes:

[0111] a super-resolution processing module configured to improve the resolution of the video frame.

[0112] Optionally, the video frame processing module further includes:

[0113] a style transfer processing module configured to add a specific artistic style to the video frame; or

[0114] a special effect processing module configured to add a specific special effect to the video frame.

[0115] Optionally, the frame track starts with a still frame segment and ends with a still frame segment.

[0116] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the content not described in detail in some embodiments can be referred to the same or similar content in other embodiments.

[0117] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0118] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0119] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0120] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0121] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0122] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for continuous splicing of digital human video actions of a live streaming scene, characterized in that, The method comprises the following steps: acquiring a pre-recorded digital human video in which a digital human alternately stays still and moves; dividing the digital human video into still frame segments and action frame segments, and labeling tags frame by frame to obtain a frame track; the tags include still and action; setting a frame pointer on the frame track, the frame pointer being used to slide forward and backward on the frame track to achieve positioning and extraction of video frames, the frame pointer being initialized to point to the still frame segment; during video playback, controlling the frame pointer to slide according to the character state of the digital human, and when the character state is a still state, adding a frame extracted each time directly to the tail of a playback queue, and when the character state is a speaking state, processing a continuous video frame segment extracted through a video frame processing module and then adding the processed continuous video frame segment to the tail of the playback queue in sequence; wherein when the character state is a still state, all the video frames extracted are still frames, and when the character state is a speaking state, the last frame of all the video frames extracted is a still frame; when there is corresponding original audio, the character state is a speaking state, and when there is no corresponding original audio, the character state is a still state; the original audio is divided into a plurality of audio segments, and the audio segments correspond to the continuous video frame segments one by one; displaying the content of the playback queue frame by frame and synchronously playing the original audio; controlling the frame pointer to slide according to the character state of the digital human, specifically comprising: when the character state is a still state, controlling the frame pointer to slide forward at a preset frame rate, and when the frame pointer slides to the edge of the current still frame segment, reversing the sliding direction; when the character state is a speaking state, acquiring each audio segment, and extracting video frames from each audio segment in time sequence, wherein for the current audio segment being processed, if the audio segment is not the last audio segment, controlling the frame pointer to slide forward at a preset frame rate, reversing the sliding direction when the frame pointer slides to the edge of the frame track, and continuing until a continuous video frame segment with a playback time length matching the time length of the audio segment is extracted, and if the audio segment is the last audio segment, controlling the frame pointer to slide into the nearest still frame segment, and then controlling the frame pointer to slide according to the strategy when the character state is a still state until a continuous video frame segment with a playback time length matching the time length of the audio segment is extracted; the video frame processing module is specifically an AI technology-based video frame processing module, comprising: a face detection module for acquiring face position information of each video frame in the current continuous video frame segment; a lip shape generation module for generating a corresponding lip shape map according to the current audio segment; a map fusion processing module for pasting the lip shape map back to the corresponding video frame according to the face position information.

2. The digital human video motion continuous stitching method of live streaming scene according to claim 1, characterized in that, acquiring each audio segment specifically comprises: acquiring each audio segment divided in advance; the division process of the audio segment comprises: acquiring the original audio; the original audio is complete audio data or an audio data stream with an end determined by any means; Dividing the original audio into a plurality of audio segments, and numbering the audio segments.

3. The method of claim 2, wherein the method further comprises: Dividing the original audio into a plurality of audio segments, specifically comprising: Dividing the original audio into a plurality of audio segments with equal audio length based on a time axis; or, Dividing the original audio into a plurality of audio segments with equal number of broadcasted characters based on audio content; or, Dividing the original audio into a plurality of audio segments with punctuation marks as division marks; or, Dividing the original audio into a plurality of audio segments with punctuation marks as division marks, and based on a preset character threshold, so that the number of characters in each audio segment is greater than the preset character threshold.

4. The method of claim 1, wherein the method further comprises: The video frame processing module further comprises: A super-resolution processing module for improving the resolution of the video frame.

5. The digital human video motion continuous stitching method of live streaming scene according to claim 4, characterized in that, The video frame processing module further comprises: A style transfer processing module for adding a specific artistic style to the video frame; or, A special effect processing module for adding a specific special effect to the video frame.

6. The method of claim 1, wherein, The frame track starts with one of the still frame segments and ends with one of the still frame segments.

7. A digital human video motion continuous splicing device for streaming live scenes, characterized in that, The digital human video action continuous splicing method and device for the live streaming scene according to any one of claims 1 to 6, comprising: An acquired digital human video module for acquiring a pre-recorded digital human video, in which the digital human alternately stays still and performs actions; A frame track construction module for dividing the digital human video into still frame segments and action frame segments, and marking labels frame by frame to obtain a frame track; the labels include still and action; A frame pointer setting module for setting a frame pointer on the frame track, the frame pointer being used to slide forward and backward on the frame track to realize positioning and extracting video frames, the frame pointer being initialized to point to the still frame segment; A frame extraction and processing module for controlling the frame pointer to slide according to the character state of the digital human during video playback, and when the character state is a still state, adding a frame directly to the tail of a playback queue every time a frame is extracted, and when the character state is a speaking state, processing the extracted continuous video frame segment through a video frame processing module, and then adding the processed continuous video frame segment to the tail of the playback queue in sequence; wherein, when the character state is a still state, all extracted video frames are still frames, and when the character state is a speaking state, the last frame of all extracted video frames is a still frame; when there is corresponding original audio, the character state is a speaking state, and when there is no corresponding original audio, the character state is a still state; the original audio is divided into a plurality of audio segments, and the audio segments correspond one-to-one to the continuous video frame segments; A playback module for displaying the content of the playback queue frame by frame and synchronously playing the original audio.

Images