Digital human video action continuous splicing method and device for streaming live broadcast scene

Abstract

The invention provides a digital human video action continuous splicing method and device for a streaming live broadcast scene, and relates to the technical field of virtual digital humans, and the method comprises the steps: obtaining a digital human video in which a digital human is still and does actions alternately, and constructing a frame track; when the person state is a static state, each frame is extracted and directly added to the tail part of the playing queue, and when the person state is a speaking state, after extracted continuous video frame segments are processed by a video frame processing module, the processed continuous video frame segments are sequentially added to the tail part of the playing queue; when the person state is a static state, the extracted video frames are all static frames, and when the person state is a speaking state, the last frame in all the extracted video frames is a static frame; and displaying the content of the playing queue frame by frame, and synchronously playing the original audio. According to the invention, the picture jitter during the action switching of the digital person can be reduced, and the stability and natural expressive force of the digital person in the streaming live broadcast are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual digital humans, and in particular to a method and device for continuously splicing digital human video actions in a streaming live broadcast scene. Background Art

[0002] Currently, the video coherence of virtual characters has become a crucial factor affecting user experience in scenarios such as news broadcasting, online customer service, education and training, and live entertainment. To generate rich digital human videos, a common method in existing technologies is to construct a complete action sequence by splicing pre-produced video clips.

[0003] However, these solutions primarily focus on combining and reusing motion content. When switching between video clips, especially when transitioning between static and motion states, they often lack a smooth transition mechanism. This direct splicing approach can easily lead to noticeable jumps, jitters, or freezes in the final synthesized image, disrupting visual continuity and naturalness. Furthermore, when synthesizing videos, if the playback rhythm of the video frames isn't precisely aligned with the speech rhythm of the audio being synthesized, lip movements can easily become out of sync with the sound, severely impacting the realism of the digital human and the user's immersive experience.

[0004] Based on this, how to reduce the image jitter when the digital human switches movements and improve the stability and natural expressiveness of the digital human in live streaming has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of this, in order to solve the above technical problems, the present invention provides a method and device for continuously splicing digital human video movements in a streaming live scene.

[0006] The present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for continuously splicing digital human video actions in a live streaming scene, comprising: Obtaining a pre-recorded digital human video, in which the digital human alternately remains still and performs actions; Dividing the digital human video into still frame segments and action frame segments, and labeling each frame to obtain a frame track; the labels include still and action; Setting a frame pointer on the frame track, wherein the frame pointer is used to slide back and forth on the frame track to locate and extract the video frame, and the frame pointer is initially pointed to the still frame segment; During video playback, the frame pointer is controlled to slide according to the state of the digital human. When the state of the digital human is still, each frame is directly added to the end of the playback queue after being extracted. When the state of the digital human is speaking, the continuous video frame segments extracted are processed by the video frame processing module, and the processed continuous video frame segments are added to the end of the playback queue in sequence. When the state of the digital human is still, the extracted video frames are all still frames. When the state of the digital human is speaking, the last frame of all the extracted video frames is a still frame. When there is corresponding original audio, the state of the digital human is speaking, and when there is no corresponding original audio, the state of the digital human is still. The original audio is divided into multiple audio segments, and the audio segments correspond one-to-one to the continuous video frame segments. The contents of the playback queue are displayed frame by frame, and the original audio is played synchronously.

[0007] Optionally, controlling the frame pointer to slide according to the character state of the digital human specifically includes: When the character is in a stationary state, controlling the frame pointer to slide forward at a preset frame rate, and sliding in the reverse direction when sliding to the edge of the current stationary frame segment; When the character is in a speaking state, each audio segment is obtained, and video frames are extracted for each audio segment in chronological order. For the audio segment currently being processed, if the audio segment is not the last audio segment, the frame pointer is controlled to slide forward at a preset frame rate. When sliding to the edge of the frame track, the frame pointer is controlled to slide in the reverse direction until a continuous video frame segment whose playback duration matches the duration of the audio segment is extracted. If the audio segment is the last audio segment, the frame pointer is controlled to slide into the nearest still frame segment. After sliding into the still frame segment, the frame pointer is controlled to slide according to the strategy when the character is in a still state until a continuous video frame segment whose playback duration matches the duration of the audio segment is extracted.

[0008] Optionally, obtain each audio segment, including: Obtaining pre-divided audio segments; The audio segmentation process includes: Acquire the original audio; the original audio is complete audio data, or an audio data stream whose end is determined by any method; The original audio is divided into a plurality of audio segments, and the audio segments are numbered.

[0009] Optionally, dividing the original audio into a plurality of audio segments specifically includes: Divide the original audio into a plurality of audio segments of equal length based on a time axis; or Divide the original audio into a plurality of audio segments with an equal number of broadcast words based on the audio content; or Using punctuation marks as division marks, dividing the original audio into a plurality of audio segments; or, The original audio is divided into a plurality of audio segments using punctuation marks as division identifiers and based on a preset word count threshold, so that the word count of each audio segment is greater than the preset word count threshold.

[0010] Optionally, the video frame processing module is specifically a video frame processing module based on AI technology, including: A face detection module is used to obtain face position information of each video frame in the current continuous video frame segment; A lip shape generation module, configured to generate a corresponding lip shape map according to the current audio segment; The map fusion processing module is used to paste the lip map back to the corresponding video frame according to the face position information.

[0011] Optionally, the video frame processing module further includes: The super-resolution processing module is used to improve the resolution of the video frame.

[0012] Optionally, the video frame processing module further includes: a style transfer processing module, configured to add a specific artistic style to the video frame; or The special effects processing module is used to add specific special effects to the video frames.

[0013] Optionally, the frame track starts with a still frame segment and ends with a still frame segment.

[0014] In a second aspect, the present invention further provides a device for continuously splicing digital human video actions in a live streaming scene, comprising: A digital human video acquisition module is used to acquire pre-recorded digital human videos, in which the digital human alternately remains still and performs actions; Constructing a frame track module for dividing the digital human video into still frame segments and action frame segments, and labeling each frame to obtain a frame track; the labels include still and action; Setting a frame pointer module, for setting a frame pointer on the frame track, the frame pointer is used to slide back and forth on the frame track to locate and extract the video frame, and the frame pointer is initialized to point to the still frame segment; The frame extraction and processing module is used to control the sliding of the frame pointer according to the character state of the digital human during video playback, and when the character state is still, each extracted frame is directly added to the end of the playback queue; when the character state is speaking, the extracted continuous video frame segments are processed by the video frame processing module and then added to the end of the playback queue in sequence; wherein, when the character state is still, the extracted video frames are all still frames, and when the character state is speaking, the last frame of all extracted video frames is a still frame; when there is corresponding original audio, the character state is speaking, and when there is no corresponding original audio, the character state is still; the original audio is divided into multiple audio segments, and the audio segments correspond one-to-one to the continuous video frame segments; The playback module is used to display the contents of the playback queue frame by frame and synchronously play the original audio.

[0015] The present invention adopts the above technical solution. By constructing a frame track in which static frame segments and action frame segments are alternately arranged, different frame extraction strategies are used to extract video frames from the frame track when the character is in a static state or a speaking state, and the extracted video frames to be played are placed in a play queue. The continuous frames in the play queue are directly played. This allows the present invention to reduce the image jitter when the digital human switches actions, improve the coherence of the digital human's actions, and thus enhance the stability and natural expressiveness of the digital human in streaming live broadcasts. It is particularly suitable for virtual digital human live broadcast application scenarios that require long-term continuous broadcasts. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a flow chart of a method for continuously splicing digital human video actions in a live streaming scene provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of a combined structure of a frame track and a play queue provided by an embodiment of the present invention; Figure 3 The present invention provides a schematic structural diagram of a device for continuously splicing digital human video movements in a live streaming scene. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.

[0019] Figure 1 The present invention provides a flowchart of a method for continuously splicing digital human video actions in a live streaming scene. Figure 2 This is a schematic diagram of a combined structure of a frame track and a play queue provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, this process includes: Step 101: Obtain a pre-recorded digital human video, in which the digital human alternates between being still and performing actions.

[0020] Specifically, the digital human alternates between being still and performing actions, forming structured segments. The beginning and ending segments of the digital human video are both still segments, and the duration of each segment is less than a preset duration threshold. In one specific example, when recording a digital human video, the digital human remains still for 10 seconds, then begins waving, beckoning, and nodding for 10 seconds, then remains still for another 10 seconds, and so on, alternating until the recording ends.

[0021] Step 102: Divide the digital human video into still frames and action frames, and label each frame to obtain a frame track; the labels include still and action. Establishing the frame track can achieve the ordering of all video frames.

[0022] Specifically, all frames in the still frame segment are still frames, and all frames in the action frame segment are action frames. Each frame segment contains several consecutive frames. By marking labels frame by frame, it is convenient to quickly confirm whether any frame in the frame track is a still frame or an action frame.

[0023] like Figure 2 As shown, the frame track 21 is composed of a plurality of still frame segments and action frame segments arranged alternately ( Figure 2 (This section uses four still frames and three action frames as an example.) Furthermore, the frame track begins and ends with a still frame, ensuring stable system startup and shutdown. Still frames are used for static presentations and static speech announcements of the digital human without voice input, while action frames are used to demonstrate the digital human's natural movements while speaking.

[0024] Step 103: A frame pointer 22 is set on the frame track, allowing the frame pointer to slide back and forth on the frame track. The frame pointer is used to slide back and forth on the frame track to locate and extract video frames. The frame pointer is initially pointed to a still frame segment. If the frame track begins and ends with a still frame segment, the frame pointer is initially pointed to the first frame segment on the frame track.

[0025] Step 104: During video playback, the frame pointer is controlled to slide according to the state of the digital human. When the state of the digital human is still, each frame is directly added to the end of the playback queue 24 after being extracted. When the state of the digital human is speaking, the extracted continuous video frame segments are processed by the video frame processing module 23, and the processed continuous video frame segments are added to the end of the playback queue 24 in sequence. When the state of the digital human is still, the extracted video frames are all still frames. When the state of the digital human is speaking, the last frame of all the extracted video frames is a still frame to ensure that the broadcast video is a natural static picture when the speech ends. When there is corresponding original audio, the state of the digital human is speaking, and when there is no corresponding original audio, the state of the digital human is still. The original audio is divided into multiple audio segments, and the audio segments correspond one-to-one to the continuous video frame segments.

[0026] Step 105: Display the contents of the play queue frame by frame and play the original audio synchronously.

[0027] The playback queue can adopt a FIFO (First In, First Out) structure to ensure the orderly output of video frames as the final digital human live broadcast source.

[0028] The present invention adopts the above technical solution. By constructing a frame track with alternating static and action frame segments, different frame extraction strategies are used to extract video frames from the frame track when the character is in a static or speaking state. The extracted video frames to be played are then placed in a play queue, and the continuous frames in the play queue are directly played. This allows the present invention to reduce the image jitter when the digital human switches between actions. The measured jitter amplitude has been reduced by more than 85%, improving the coherence of the digital human's movements, and further enhancing the stability and natural expressiveness of the digital human in live streaming. The present invention can be widely used in scenarios such as virtual human live broadcasts, digital human broadcasts, and intelligent customer service, and is particularly suitable for virtual digital human live broadcast applications that require long-term continuous broadcasts.

[0029] In addition, the entire process of the present invention does not require human intervention, improves the degree of automation, supports real-time dynamic adjustment, is suitable for complex environments such as long-term live broadcasts and multi-scene switching, and improves system stability by more than 90%.

[0030] In the embodiment of the present invention, controlling the frame pointer sliding according to the character state of the digital human may specifically include: (1) When the character is in a stationary state, the frame pointer is controlled to slide forward at a preset frame rate. When it slides to the edge of the current stationary frame segment, it slides in the opposite direction to ensure the stability of the picture.

[0031] Specifically, the preset frame rate may be 25 frames per second. Furthermore, when the character is in a stationary state, the length of the play queue may be limited to 1.

[0032] It should be noted that since the frame pointer points to the still frame segment when it is initialized, and when the last audio segment ends (that is, when the speech ends), the frame pointer also points to the still frame segment. Therefore, when the character is in a still state, the frame pointer must be in the still frame segment.

[0033] (2) When the character is in the speaking state, obtain each audio segment and extract video frames for each audio segment in chronological order. For the currently processed audio segment, if the audio segment is not the last audio segment, control the frame pointer to slide forward at a preset frame rate. When it slides to the edge of the frame track, it slides in the reverse direction until a continuous video frame segment whose playback duration matches the duration of the audio segment is extracted. If the audio segment is the last audio segment, control the frame pointer to slide into the nearest still frame segment. After sliding into the still frame segment, control the frame pointer to slide according to the strategy when the character is in the still state until a continuous video frame segment whose playback duration matches the duration of the audio segment is extracted. The length of the continuous video frame segment is calculated based on the duration of the corresponding audio segment and the preset frame rate.

[0034] In this solution, the frame pointer position is updated in real time so that the extracted frame segments can keep up with the playback progress. Real-time updating means that when the playback queue is empty, the frame pointer is moved, and frames are continuously extracted and added to the playback queue.

[0035] The embodiment of the present invention adopts the above-mentioned solution. Since the frame pointer points to a static frame segment at the end of the last audio segment (i.e., the end of the speech), the video playback process is visually smoother and more natural. Furthermore, by extracting video frames from each audio segment in chronological order, continuous video frames whose playback duration matches the audio segment duration are extracted, processed by the video frame processing module, and added to the playback queue, the present invention achieves synchronized audio and video playback.

[0036] In the embodiment of the present invention, obtaining each audio segment may specifically include: Get the pre-divided audio segments.

[0037] The audio segmentation process includes: (1) Obtain the original audio; the original audio is complete audio data, or an audio data stream whose end is determined by any method.

[0038] Specifically, the original audio can be obtained from audio files, audio capture devices, or network streams. In actual application scenarios, the audio stream is continuously segmented during the generation process. Only after the generation is completed can the last segment be determined and the last segment can be marked.

[0039] (2) Divide the original audio into multiple audio segments and number the audio segments.

[0040] In the embodiment of the present invention, dividing the original audio into multiple audio segments may specifically include: Divide the original audio into multiple audio segments of equal length based on the time axis, for example, divide the audio into one segment every 5 seconds. Or, Divide the original audio into multiple audio segments with the same number of words based on the audio content, for example, one segment for every 20 words. Or, Use punctuation marks as division markers to divide the original audio into multiple audio segments, that is, divide the audio into a segment each time a punctuation mark is encountered. Or, The original audio is divided into multiple audio segments based on a preset word count threshold, using punctuation marks as segmentation markers. Each audio segment is divided into multiple segments, so that the number of words in each segment exceeds the preset word count threshold. For example, if there are more than 20 words and a punctuation mark is encountered, the audio segment is divided into a segment.

[0041] In an embodiment of the present invention, the video frame processing module is specifically a video frame processing module based on AI technology, including multiple deep learning modules. The multiple deep learning modules work together, specifically including: The face detection module is used to obtain the face position information of each video frame in the current continuous video frame segment.

[0042] The lip shape generation module is used to generate the corresponding lip shape map according to the current audio segment.

[0043] The map fusion processing module is used to paste the mouth shape map back to the corresponding video frame based on the face position information.

[0044] The embodiment of the present invention adopts the above-mentioned solution. By detecting the facial position information, generating a lip map and pasting the lip map back to the corresponding video frame based on the facial position information, the present invention can improve the synchronization between lip shape and speech. The measured lip shape and speech synchronization error is ≤50ms, which enhances the realism of the digital human.

[0045] In addition, the video frame processing module may also include: The super-resolution processing module is used to increase the resolution of video frames, specifically up to 4K. This improves image clarity and enhances the quality and visual quality of the output video.

[0046] In addition, the video frame processing module may also include: The style transfer processing module is used to add specific artistic styles to video frames, such as oil painting and ink painting. The oil painting style can give the digital human's skin and clothing a brushstroke feel, such as Rembrandt's light and shadow effects. The ink painting style is suitable for digital humans with traditional Chinese style, creating a freehand effect of brushstrokes and white space. Alternatively, the special effects processing module is used to add specific special effects to video frames, such as dynamic mapping and virtual-real light and shadow matching. Dynamic mapping can overlay virtual decorations such as cat ears, orc features, and mechanical visors in real time. Virtual-real light and shadow matching can adjust the lighting direction and shadows of the digital human to make it blend naturally with the real scene. This can enrich the visual effects and further improve the quality and visual quality of the output video.

[0047] Based on a general inventive concept, the present invention also provides a device for continuously splicing digital human video actions in a streaming live scene. Figure 3 This is a structural diagram of a device for continuously splicing digital human video actions in a live streaming scene provided by an embodiment of the present invention. Figure 3 As shown, the device includes: The digital human video acquisition module 31 is used to acquire pre-recorded digital human videos, in which the digital human alternately remains still and performs actions; A frame track building module 32 is used to divide the digital human video into still frame segments and action frame segments, and mark labels frame by frame to obtain frame tracks; the labels include still and action; A frame pointer module 33 is provided for setting a frame pointer on the frame track. The frame pointer is used to slide back and forth on the frame track to locate and extract the video frame. When the frame pointer is initialized, it points to a still frame segment. The frame extraction and processing module 34 is used to control the frame pointer to slide according to the state of the digital human during video playback. When the human is in a stationary state, each extracted frame is directly added to the end of the playback queue. When the human is in a speaking state, the extracted continuous video frame segments are processed by the video frame processing module and then added to the end of the playback queue in sequence. When the human is in a stationary state, the extracted video frames are all stationary frames. When the human is in a speaking state, the last frame of all extracted video frames is a stationary frame. When there is corresponding original audio, the human is in a speaking state. When there is no corresponding original audio, the human is in a stationary state. The original audio is divided into multiple audio segments, and the audio segments correspond one-to-one with the continuous video frames. The playback module 35 is used to display the contents of the playback queue frame by frame and synchronously play the original audio.

[0048] Optionally, the frame extraction and processing module 34 may specifically include: The first sliding unit is used to control the frame pointer to slide forward at a preset frame rate when the character is in a static state, and to slide in the reverse direction when sliding to the edge of the current static frame segment.

[0049] The second sliding unit is used to obtain each audio segment when the character is in a speaking state, and extract video frames for each audio segment in chronological order. For the currently processed audio segment, if the audio segment is not the last audio segment, the frame pointer is controlled to slide forward at a preset frame rate. When sliding to the edge of the frame track, it slides in the opposite direction until a continuous video frame segment whose playback duration matches the audio segment duration is extracted. If the audio segment is the last audio segment, the frame pointer is controlled to slide into the nearest still frame segment. After sliding into the still frame segment, the frame pointer is controlled to slide according to the strategy when the character is in a still state until a continuous video frame segment whose playback duration matches the audio segment duration is extracted.

[0050] Optionally, the second sliding unit can be used to: Get the pre-divided audio segments.

[0051] The device may also include: The original audio acquisition module is used to obtain the original audio; the original audio is complete audio data, or an audio data stream whose end is determined by any method.

[0052] The audio segmentation module is used to divide the original audio into multiple audio segments and number the audio segments.

[0053] Optional audio partitioning module, specifically used for: Divide the original audio into multiple audio segments of equal length based on the time axis; or Divide the original audio into multiple audio segments with equal number of words based on the audio content; or Using punctuation marks as division marks, the original audio is divided into multiple audio segments; or, Using punctuation marks as division markers and based on a preset word count threshold, the original audio is divided into multiple audio segments, so that the number of words in each audio segment is greater than the preset word count threshold.

[0054] Optionally, the video frame processing module is specifically an AI-based video frame processing module, including: The face detection module is used to obtain the face position information of each video frame in the current continuous video frame segment.

[0055] The lip shape generation module is used to generate the corresponding lip shape map according to the current audio segment.

[0056] The map fusion processing module is used to paste the mouth shape map back to the corresponding video frame based on the face position information.

[0057] Optionally, the video frame processing module further includes: The super-resolution processing module is used to improve the resolution of video frames.

[0058] Optionally, the video frame processing module further includes: A style transfer processing module to add a specific artistic style to the video frames; or The special effects processing module is used to add specific special effects to video frames.

[0059] Optionally, the frame track starts with a still frame segment and ends with a still frame segment.

[0060] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0061] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0062] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0063] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0064] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0065] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0066] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for continuously splicing digital human video actions in a live streaming scene, characterized in that: include: Obtaining a pre-recorded digital human video, in which the digital human alternately remains still and performs actions; Dividing the digital human video into still frame segments and action frame segments, and labeling each frame to obtain a frame track; the labels include still and action; Setting a frame pointer on the frame track, wherein the frame pointer is used to slide back and forth on the frame track to locate and extract the video frame, and the frame pointer is initially pointed to the still frame segment; During video playback, the frame pointer is controlled to slide according to the state of the digital human. When the state of the digital human is still, each frame is directly added to the end of the playback queue after being extracted. When the state of the digital human is speaking, the continuous video frame segments extracted are processed by the video frame processing module, and the processed continuous video frame segments are added to the end of the playback queue in sequence. When the state of the digital human is still, the extracted video frames are all still frames. When the state of the digital human is speaking, the last frame of all the extracted video frames is a still frame. When there is corresponding original audio, the state of the digital human is speaking, and when there is no corresponding original audio, the state of the digital human is still. The original audio is divided into multiple audio segments, and the audio segments correspond one-to-one to the continuous video frame segments. The contents of the playback queue are displayed frame by frame, and the original audio is played synchronously.

2. The method for continuously splicing digital human video actions in a live streaming scene according to claim 1, characterized in that: Controlling the frame pointer to slide according to the character state of the digital human specifically includes: When the character is in a stationary state, controlling the frame pointer to slide forward at a preset frame rate, and sliding in the reverse direction when sliding to the edge of the current stationary frame segment; When the character is in a speaking state, each audio segment is obtained, and video frames are extracted for each audio segment in chronological order. For the audio segment currently being processed, if the audio segment is not the last audio segment, the frame pointer is controlled to slide forward at a preset frame rate. When sliding to the edge of the frame track, the frame pointer is controlled to slide in the reverse direction until a continuous video frame segment whose playback duration matches the duration of the audio segment is extracted. If the audio segment is the last audio segment, the frame pointer is controlled to slide into the nearest still frame segment. After sliding into the still frame segment, the frame pointer is controlled to slide according to the strategy when the character is in a still state until a continuous video frame segment whose playback duration matches the duration of the audio segment is extracted.

3. The method for continuously splicing digital human video actions in a live streaming scene according to claim 2, characterized in that: Get each audio segment, including: Obtaining pre-divided audio segments; The audio segmentation process includes: Acquire the original audio; the original audio is complete audio data, or an audio data stream whose end is determined by any method; The original audio is divided into a plurality of audio segments, and the audio segments are numbered.

4. The method for continuously splicing digital human video actions in a live streaming scene according to claim 3 is characterized in that: Dividing the original audio into a plurality of audio segments specifically includes: Divide the original audio into a plurality of audio segments of equal length based on a time axis; or Divide the original audio into a plurality of audio segments with an equal number of broadcast words based on the audio content; or Using punctuation marks as division marks, dividing the original audio into a plurality of audio segments; or, The original audio is divided into a plurality of audio segments using punctuation marks as division identifiers and based on a preset word count threshold, so that the word count of each audio segment is greater than the preset word count threshold.

5. The method for continuously splicing digital human video actions in a live streaming scene according to claim 1 is characterized in that: The video frame processing module is specifically a video frame processing module based on AI technology, including: A face detection module is used to obtain face position information of each video frame in the current continuous video frame segment; A lip shape generation module, configured to generate a corresponding lip shape map according to the current audio segment; The map fusion processing module is used to paste the lip map back to the corresponding video frame according to the face position information.

6. The method for continuously splicing digital human video actions in a live streaming scene according to claim 5, characterized in that: The video frame processing module also includes: The super-resolution processing module is used to improve the resolution of the video frame.

7. The method for continuously splicing digital human video actions in a live streaming scene according to claim 6, characterized in that: The video frame processing module also includes: a style transfer processing module, configured to add a specific artistic style to the video frame; or The special effects processing module is used to add specific special effects to the video frames.

8. The method for continuously splicing digital human video actions in a live streaming scene according to claim 1, characterized in that: The frame track starts with one of the still frame segments and ends with one of the still frame segments.

9. A device for continuously splicing digital human video actions in a live streaming scene, characterized in that: include: A digital human video acquisition module is used to acquire pre-recorded digital human videos, in which the digital human alternately remains still and performs actions; Constructing a frame track module for dividing the digital human video into still frame segments and action frame segments, and labeling each frame to obtain a frame track; the labels include still and action; Setting a frame pointer module, for setting a frame pointer on the frame track, the frame pointer is used to slide back and forth on the frame track to locate and extract the video frame, and the frame pointer is initialized to point to the still frame segment; The frame extraction and processing module is used to control the sliding of the frame pointer according to the character state of the digital human during video playback, and when the character state is still, each extracted frame is directly added to the end of the playback queue; when the character state is speaking, the extracted continuous video frame segments are processed by the video frame processing module and then added to the end of the playback queue in sequence; wherein, when the character state is still, the extracted video frames are all still frames, and when the character state is speaking, the last frame of all extracted video frames is a still frame; when there is corresponding original audio, the character state is speaking, and when there is no corresponding original audio, the character state is still; the original audio is divided into multiple audio segments, and the audio segments correspond one-to-one to the continuous video frame segments; The playback module is used to display the contents of the playback queue frame by frame and synchronously play the original audio.

Images