Video generation method, device, computing equipment and medium based on reference video

Abstract

The embodiment of the present application discloses a method, apparatus, computing device and medium for video generation based on a reference video, wherein the method comprises: dividing a reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments; obtaining driving data, processing the driving data into a plurality of driving voice segments, and determining the emotion types of the plurality of driving voice segments; determining a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments based on the duration, emotion type and whether it is a silent segment; using the plurality of driving voice segments to drive and lip-correct the corresponding target reference video segments, and connecting them to form a target video. The present application can generate a target video of any length and with coherent virtual anchor movements based on a short-duration reference video, effectively improving the realism and movement coherence of the virtual anchor voice-driven video generation based on the reference video.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and specifically to a method, apparatus, computing device, and medium for generating a video based on a reference video. Background Art

[0002] With the continuous advancement of artificial intelligence technology, technologies such as Wav2Lip, which use speech to drive images or videos to produce corresponding lip movements, head movements, and even subtle expressions, have rapidly developed, leading to the current research boom in virtual anchor video generation. Typically, a reference video is used as a whole input to generate the video.

[0003] However, in actual use, it was found that the existing generation method has the following problems: 1) The length of the generated virtual anchor video is limited. After a certain length of time, the lip synchronization decreases; 2) The mouth does not close when silent, which may be due to the noise in the training video. As a result, the generated virtual anchor video still has lips opening and closing and shaking movements even when silent, which seriously reduces the lip synchronization; 3) Affected by the length of the reference video, frame skipping will occur when generating a video with a longer duration, resulting in video incoherence; 4) The expression does not match the driving audio, resulting in distortion of the generated video. Summary of the Invention

[0004] In view of the above problems, the present application is proposed to provide a method, apparatus, computing device and medium for generating a video based on a reference video that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of an embodiment of the present application, a method for generating a video based on a reference video is provided, comprising:

[0006] Segmenting a reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments;

[0007] Acquiring driving data, processing the driving data into a plurality of driving speech segments, and determining emotion types of the plurality of driving speech segments;

[0008] Determine a corresponding target reference video segment for each driving speech segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment;

[0009] Multiple driving speech segments are used to drive and lip-modify the corresponding target reference video segments, and then connected to form a target video.

[0010] Furthermore, segmenting the reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments further includes:

[0011] Extracting speech channel data and visual channel data from the reference video;

[0012] Detecting the silent part in the voice channel data, performing preliminary segmentation based on the silent part, and obtaining preliminary segmented voiced segments and silent segments;

[0013] Based on the visual channel data corresponding to the initially segmented silent segment, the segmentation boundary is refined to obtain a finely segmented segmentation boundary;

[0014] Correcting the segmentation boundaries of the initially segmented voiced and silent segments according to the segmentation boundaries of the fine segmentation to obtain final voiced and silent segments;

[0015] The micro-expression recognition method is used to identify the emotion types of the sound clips and the silent clips. If a single reference video clip has multiple emotion types, the reference video clip is divided again so that each reference video clip has a single emotion type.

[0016] Furthermore, processing the driving data into a plurality of driving speech segments and determining the emotion types of the plurality of driving speech segments further includes:

[0017] If the driving data is audio, detecting silence in the audio, segmenting the audio into multiple driving speech segments based on the silence, and determining the emotion types of the multiple driving speech segments using an audio emotion analysis model; wherein the multiple driving speech segments include voiced segments and silent segments;

[0018] If the driving data is text, the text is divided into multiple sub-texts through a text sentiment analysis model and the sentiment types of the multiple sub-texts are determined. The multiple sub-texts are converted into multiple driving voice segments, and the sentiment types of the multiple driving voice segments are set accordingly according to the sentiment types of the multiple sub-texts.

[0019] Furthermore, determining a corresponding target reference video segment for each driving voice segment from the plurality of reference video segments based on the duration, the emotion type, and whether it is a silent segment further includes:

[0020] A first segment is selected from the plurality of driving speech segments, and a target reference video segment corresponding to the first segment is selected from the plurality of reference video segments based on the duration, emotion type, and whether the first segment is a silent segment; wherein the end position of the target reference video segment selected for the first segment is located at a boundary of the divided reference video segments; if the first segment is not the first segment among the plurality of driving speech segments, the target reference video segment selected for the first segment is also not the first segment among the plurality of reference video segments;

[0021] If there is a preceding segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments based on the duration, emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment;

[0022] If there is a subsequent segment of the first segment among multiple driving voice segments, the first segment is taken as the current segment, and the corresponding target reference video segment is determined for each subsequent segment from multiple reference video segments in turn based on the length, emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment.

[0023] Furthermore, determining a corresponding target reference video segment for each preceding segment from the plurality of reference video segments in sequence based on the duration, the emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment further includes:

[0024] Step 11: Using the starting position of the target reference video segment corresponding to the current segment as the preceding processing anchor point, selecting a segment adjacent to the current segment from the preceding segment, and updating the adjacent segment to the current segment;

[0025] Step 12: Based on the emotion type and whether it is a silent segment, a reference video segment whose ending position or starting position is the same as the previous processing anchor point is searched from multiple reference video segments as an intermediate reference video segment, and the offset direction variable is set to -1 or 1 based on whether the ending position or starting position is the same as the previous processing anchor point;

[0026] Step 13, calculating a floor-rounded ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor-rounded ratio;

[0027] Step 14: If the result of the floor rounding is an odd number and the offset direction variable is 1, the end position of the middle reference video segment is used as the first starting position, and the start position of the middle reference video segment is used as the first ending position;

[0028] If the result of the floor rounding is an odd number and the offset direction variable is -1, the starting position of the middle reference video segment is used as the first starting position, and the ending position of the middle reference video segment is used as the first ending position;

[0029] If the result of the floor rounding is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the first starting position, and the starting position of the middle reference video segment is used as the first ending position;

[0030] If the result of the floor rounding is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the first starting position, and the end position of the middle reference video segment is used as the first ending position;

[0031] Step 15: Select at least one continuous segment with variable direction and meeting a preset duration requirement from the intermediate reference video segments based on the duration difference, the first starting position, and the first ending position as a target reference video segment corresponding to the current segment;

[0032] Step 16: inserting the target reference video segment corresponding to the current segment into the head of the determined target reference video segment;

[0033] Step 17: If the preceding segment of the current segment exists in the plurality of driving voice segments, the process jumps to step 11.

[0034] Furthermore, determining a corresponding target reference video segment for each subsequent segment from the plurality of reference video segments in sequence based on the duration, the emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment further includes:

[0035] Step 21: Using the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, selecting a segment adjacent to the current segment from the subsequent segments, and updating the adjacent segment to the current segment;

[0036] Step 22: Based on the emotion type and whether it is a silent segment, a reference video segment whose ending position or starting position is the same as the subsequent processing anchor point is searched from multiple reference video segments as an intermediate reference video segment, and an offset direction variable is set to -1 or 1 based on whether the ending position or starting position is the same as the subsequent processing anchor point;

[0037] Step 23, calculating a floor-rounded ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor-rounded ratio;

[0038] Step 24: If the rounding result is an odd number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the ending position of the middle reference video segment is used as the second ending position;

[0039] If the result of the floor rounding is an odd number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second starting position, and the start position of the middle reference video segment is used as the second ending position;

[0040] If the rounding result is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the starting position of the middle reference video segment is used as the second ending position;

[0041] If the result of the floor rounding is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second starting position, and the end position of the middle reference video segment is used as the second ending position;

[0042] Step 25: Select at least one continuous segment with variable direction and meeting a preset duration requirement from the intermediate reference video segments based on the duration difference, the second starting position, and the second ending position as a target reference video segment corresponding to the current segment;

[0043] Step 26: insert the target reference video segment corresponding to the current segment into the end of the determined target reference video segment;

[0044] Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

[0045] According to another aspect of an embodiment of the present application, a video generation apparatus based on a reference video is provided, comprising:

[0046] A first processing module is adapted to segment a reference video into a plurality of reference video segments and determine emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments;

[0047] a second processing module adapted to acquire driving data, process the driving data into a plurality of driving speech segments, and determine emotion types of the plurality of driving speech segments;

[0048] a determination module adapted to determine a corresponding target reference video segment for each driving speech segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment;

[0049] The generation module is suitable for using multiple driving voice segments to drive and lip-modify the corresponding target reference video segments, and connect them to form a target video.

[0050] According to another aspect of an embodiment of the present application, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0051] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned video generation method based on the reference video.

[0052] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored. The executable instruction enables a processor to perform operations corresponding to the above-mentioned video generation method based on the reference video.

[0053] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned method for generating a video based on a reference video.

[0054] According to the technical solution provided by the embodiment of the present application, the reference video and the driving data are processed into sound segments and silent segments according to the content. Compared with the prior art method of generating a video by inputting the reference video as a whole, the single processing time of this solution is shorter, which can effectively avoid the lip-sound synchronization problem caused by generating a long video. Taking into account the duration, emotion type, and whether it is a silent segment, etc., the corresponding target reference video segment is determined for each driving voice segment according to segment optimization. In addition, the driving data is processed into sound segments and silent segments. On the one hand, it reduces the video driving of unnecessary silent segments and improves the video generation efficiency. On the other hand, it provides the possibility of selecting reference video segments. For the sound segment, a reference video segment of lip opening and closing can be selected. Selecting a reference video clip with closed lips for the silent clip can also maintain the emotional similarity between the driving voice clip and the reference video clip, narrowing the visual difference between the reference video and the target video, thereby improving the visual effect of voice driving. Moreover, since the initial driving of the silent clip is cancelled, the lip shaking phenomenon caused by the defects of the existing technology in the silent clip is avoided. This scheme can generate a target video of any length with coherent virtual anchor movements based on a short-length reference video, effectively improving the realism and movement coherence of the virtual anchor voice-driven video generation based on the reference video. At the same time, the segmented generation method is used to divide the generation task into multiple subtasks, and the dependencies between the subtasks are eliminated through reference frame selection, so that the efficiency of video generation can be improved through concurrent execution.

[0055] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the embodiments of the present application. The same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0057] Figure 1 A schematic diagram of a process of a video generation method based on a reference video according to an embodiment of the present application is shown;

[0058] Figure 2 A schematic diagram of segmentation of the initially segmented voiced segment and the silent segment is shown;

[0059] Figure 3 A schematic diagram showing selection of a corresponding target reference video segment for a current segment when the end position of the intermediate reference video segment is the same as the previous processing anchor point is shown;

[0060] Figure 4 A schematic diagram showing selection of a corresponding target reference video segment for a current segment when the starting position of the intermediate reference video segment is the same as the previous processing anchor point is shown;

[0061] Figure 5 A schematic diagram showing selection of a corresponding target reference video segment for a current segment when the starting position of the intermediate reference video segment is the same as the subsequent processing anchor point is shown;

[0062] Figure 6 A schematic diagram showing selection of a corresponding target reference video segment for a current segment when the end position of the intermediate reference video segment is the same as the subsequent processing anchor point is shown;

[0063] Figure 7 A structural block diagram of a video generation device based on a reference video according to an embodiment of the present application is shown;

[0064] Figure 8 A schematic structural diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] Figure 1FIG. 1 shows a flow chart of a method for generating a video based on a reference video according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0067] Step S101: Segment a reference video into multiple reference video segments, and determine the emotion types of the multiple reference video segments.

[0068] The embodiment of the present application proposes a method for generating a segmented video, and selects a video with a similar voice signal as a reference video for the target video. The image of the host in the target video is the same as the image of the host in the reference video. In the embodiment of the present application, the host in the target video is referred to as a virtual host. The reference video is segmented and divided into multiple reference video segments through silence detection. The multiple reference video segments include sound segments and silent segments. In step S101, the accurate segmentation of the reference video segments can be achieved through the double-layer segmentation processing of preliminary segmentation and fine segmentation.

[0069] Specifically, voice channel data and visual channel data are extracted from a reference video; the silent part in the voice channel data is detected, and preliminary segmentation is performed based on the silent part to obtain preliminary segmented voice segments and silent segments; the segmentation boundaries are refined based on the visual channel data corresponding to the preliminary segmented silent segments to obtain fine segmentation boundaries; the segmentation boundaries of the preliminary segmented voice segments and silent segments are corrected according to the fine segmentation boundaries to obtain final voice segments and silent segments; and the micro-expression recognition method is used to identify the emotional types of the voice segments and silent segments.

[0070] Extracting audio and visual channel data from a reference video can be accomplished using the open source library ffmpeg: Use the command ffmpeg -iinput.mp4 -vn-acodec copy audio.mp3 to extract audio channel data, and the command ffmpeg -iinput.mp4 -an-vcodec copy video.mp4 to extract visual channel data. Then, detect the silent portion in the extracted audio channel data, and perform a preliminary segmentation of the audio channel data based on the silent portion, dividing the data into a plurality of consecutive vocal segments and silent segments, i.e., the preliminary segmented vocal segments and silent segments. The vocal segments and silent segments are identified using the timestamp of the audio channel data, where the timestamp is in milliseconds (ms).

[0071] Considering that in actual application scenarios, there may be two problems: 1) Silent segments are often not completely without sound, such as Figure 2As shown; 2) The sound segment may also contain short silences. For this reason, it is necessary to set an amplitude threshold and a minimum duration threshold in silence detection, wherein the amplitude threshold is generally set to dBFS-16. dBFS (decibels relative to full scale) is a unit that describes the amplitude of an audio signal in a digital system. It represents the decibel value of the signal level relative to the full scale of the digital system. The full scale of dBFS is generally considered to be the maximum level that can be processed in a digital audio system. The shortest duration threshold can be set to 1500ms. Those skilled in the art can make appropriate adjustments to these two thresholds according to actual application needs.

[0072] According to the visual channel data (i.e., picture information) corresponding to the initially segmented silent segment, the segmentation boundary is refined, such as Figure 2 As shown: 1) If the mouth opens and closes in the tail sub-segment of a silent segment among multiple reference video segments, such sub-segments will be split and marked as sound segments, and merged with the subsequent sound segments; 2) If the mouth opens and closes in the head sub-segment of a silent segment among multiple reference video segments, such segments will be split and marked as sound segments, and if there is a preceding segment, it should be merged with the preceding sound segment. The segmentation boundaries of the initially segmented sound segments and silent segments are corrected according to the segmentation boundaries of the above-mentioned fine segmentation, thereby obtaining sound segments and silent segments. Among them, the judgment of the mouth opening and closing can be determined by detecting the changes in the position of the lip key points obtained using the program library Openface.

[0073] Using micro-expression recognition methods, the emotional types of both vocal and silent segments are identified. Emotional types may include neutral, happy, angry, sad, etc. Those skilled in the art may configure emotional types based on practical needs, such as adding, deleting, or further subdividing emotional types. Alternatively, if a single reference video segment has multiple emotional types, the reference video segment may be further segmented so that each segment has a single emotional type. For example, the reference video segment may be segmented again, ultimately into multiple consecutive segments, each with a single emotional type.

[0074] After multiple segmentations, the final silent segment is obtained, and a silent segment list is established. Each node in the silent segment list includes at least the segment's <starting position, ending position, duration, silent mark, and emotional type>; based on the silent segment, a corresponding sound segment list is established. Each node in the sound segment list also includes at least the segment's <starting position, ending position, duration, silent mark, and emotional type>.

[0075] Optionally, the voiced segments can be further divided into three categories: bidirectionally repeatable, unidirectionally repeatable, and non-repeatable, so that the corresponding target reference video segment can be determined as the driving voice segment based on whether it is repeatable.

[0076] Step S102 : Acquire driving data, process the driving data into a plurality of driving speech segments, and determine the emotion types of the plurality of driving speech segments.

[0077] The driving data is data used to reflect the video content of the target video to be generated, for example, the driving data may be a press release, advertising introduction information, etc. The driving data may be data in audio or text form.

[0078] If the driving data is audio, the system detects silence in the audio and segments it into multiple consecutive driving speech segments based on the silence. The system then uses an audio sentiment analysis model to determine the emotional type of each of these segments. The multiple driving speech segments include both vocal and silent segments, and the audio sentiment analysis model determines the emotional type of each vocal segment. Alternatively, it is not necessary to determine the emotional type of silent segments.

[0079] If the driving data is text, the text is divided into multiple sub-texts and the emotion types of the multiple sub-texts are determined through the text sentiment analysis model. Then, the multiple sub-texts are converted into multiple driving voice segments through speech synthesis or TTS (Text-to-Speech) technology, and the emotion types of the multiple driving voice segments are set accordingly according to the emotion types of the multiple sub-texts.

[0080] The emotion types of the multiple driving speech segments are consistent with the emotion types of the multiple reference video segments. The generated multiple driving speech segments are arranged in order from front to back. The multiple driving speech segments include <start position, end position, duration, silence flag, emotion type>. The total duration of the multiple driving speech segments is the same as the duration of the target video.

[0081] Step S103 : determining a corresponding target reference video segment for each driving voice segment from the plurality of reference video segments based on the duration, the emotion type, and whether it is a silent segment.

[0082] Considering that the flexibility of segmented driving may cause the virtual anchor's movements to be incoherent across segments, the embodiment of this application specifically proposes a cross-segment processing strategy. To ensure the coherence of the target video, the continuous processing method is selected by considering the reference video segment information of adjacent segments.

[0083] In step S103, a first segment can be selected from multiple driving voice segments, and a target reference video segment corresponding to the first segment can be selected from multiple reference video segments based on the duration, emotion type, and whether it is a silent segment of the first segment; if there is a preceding segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in turn based on the duration, emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment; if there is a subsequent segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each subsequent segment from the multiple reference video segments in turn based on the duration, emotion type, whether it is a silent segment, and the ending position of the target reference video segment corresponding to the current segment.

[0084] Among them, according to certain rules, a driving voice segment is selected from multiple driving voice segments as the first segment. For the selection of the first segment, the driving voice segment with the longest duration can be selected, or the longest sound segment can be selected from multiple driving voice segments to ensure that the best target reference video segment can be obtained. d , emotion type and whether it is a silent segment, select a reference video segment that meets the requirements from the reference video segments of the same type (emotion type, whether it is a silent segment) as the target reference video segment corresponding to the first segment, and the duration of the target reference video segment is recorded as T r , and record the starting position and ending position of the target reference video segment. Optionally, when there is no reference video segment with an emotion type that matches the emotion type of the first segment among the multiple reference video segments, a reference video segment with a neutral emotion type can be selected.

[0085] If the first segment is not the first segment among the multiple driving speech segments, the target reference video segment selected for it should not be the first segment among the multiple reference video segments. r ≤T d ≤(t+1)·T r, t is an integer, which increases from 0 upward until a reference video segment that meets the requirements is found as the intermediate reference video segment corresponding to the first segment. Among them, the end position of the target reference video segment selected for the first segment is located on the boundary of the divided reference video segments. In order to meet its requirements, after selecting the intermediate reference video segment corresponding to the first segment from multiple reference video segments, the ratio of the duration of the first segment to the duration of the selected intermediate reference video segment can be calculated as the integer result r; when the integer result r is an even number, the duration of T selected from the intermediate reference video segment with the starting position of the intermediate reference video segment as the selected starting position and the selected ending position is selected. d The continuous multiple direction-variable segments are used as the target reference video segments corresponding to the first segment; when the rounded result r is an odd number, the duration of T is selected from the intermediate reference video segment with the starting position of the intermediate reference video segment as the selected starting position and the ending position as the selected ending position. d At least one continuous direction-variable segment is used as a target reference video segment corresponding to the first segment.

[0086] Specifically, the starting position of the intermediate reference video segment is recorded as S, and the ending position is recorded as E. When the rounding result r is an even number, the starting position S of the intermediate reference video segment is used as the starting position, and the starting position S of the intermediate reference video segment is used as the ending position. A video segment with a duration of T is selected from the intermediate reference video segment. d The continuous multiple direction-variable segments are used as the target reference video segments corresponding to the first segment; when the rounded result r is an odd number, the starting position S of the intermediate reference video segment is used as the selection starting position, the ending position E of the intermediate reference video segment is used as the selection ending position, and a duration of T is selected from the intermediate reference video segment. d At least one continuous direction-variable segment is used as a target reference video segment corresponding to the first segment.

[0087] (1) If there is a preceding segment of the first segment among the multiple driving speech segments, the first segment is used as the current segment, and the corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in accordance with the following steps 11 to 17:

[0088] Step 11: Use the starting position of the target reference video segment corresponding to the current segment as the preceding processing anchor point, select the adjacent segment to the current segment from the preceding segment, and update the adjacent segment to the current segment.

[0089] The preceding processing anchor point refers to the anchor point used to select the reference video segment for the preceding segment. Update the adjacent segment of the current segment to the current segment, that is, the new current segment, and obtain the duration T of the current segment. d , emotional type, whether it is a silent clip.

[0090] Step 12, based on the emotion type and whether it is a silent segment, query a reference video segment whose end position or starting position is the same as the previous processing anchor point from multiple reference video segments as the intermediate reference video segment, and set the offset direction variable to -1 or 1 based on whether the end position or starting position is the same as the previous processing anchor point.

[0091] Based on the emotion type and whether it is a silent segment, the same type of reference video segments are selected from multiple reference video segments. Then, the reference video segment with the same ending position or starting position as the previous processing anchor point is searched from the same type of reference video segments as the intermediate reference video segment. The starting position of the intermediate reference video segment is recorded as S, its ending position is recorded as E, and its duration is recorded as T. r For ease of processing, the offset direction variable dir can be set to -1 or 1, depending on whether the ending or starting position is the same as the previous processing anchor point. If the ending position of the intermediate reference video segment is the same as the previous processing anchor point, dir is -1; if the starting position of the intermediate reference video segment is the same as the previous processing anchor point, dir is 1. Optionally, if no reference video segment among multiple reference video segments has an emotional type that matches the emotional type of the current segment, a reference video segment with a neutral emotional type can be selected.

[0092] Step 13: Calculate the floor value of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculate the duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor value.

[0093] In the embodiment of the present application, r is used to represent the rounded-down result, and g is used to represent the time difference. Specifically, g=T d -r*T r , where T d Indicates the duration of the current segment, T r Indicates the duration of the intermediate reference video segment.

[0094] Step 14. If the rounding-down result is an odd number and the offset direction variable is 1, the end position of the intermediate reference video clip is used as the first starting position, and the starting position of the intermediate reference video clip is used as the first ending position; if the rounding-down result is an odd number and the offset direction variable is -1, the starting position of the intermediate reference video clip is used as the first starting position, and the end position of the intermediate reference video clip is used as the first ending position; if the rounding-down result is an even number and the offset direction variable is 1, the starting position of the intermediate reference video clip is used as the first starting position, and the starting position of the intermediate reference video clip is used as the first ending position; if the rounding-down result is an even number and the offset direction variable is -1, the end position of the intermediate reference video clip is used as the first starting position, and the end position of the intermediate reference video clip is used as the first ending position.

[0095] Step 15 : Select at least one direction-variable but continuous segment that meets a preset duration requirement from the intermediate reference video segments based on the duration difference, the first starting position, and the first ending position as the target reference video segment corresponding to the current segment.

[0096] Figure 3 FIG2 shows a schematic diagram of selecting a corresponding target reference video segment for a current segment when the end position of the intermediate reference video segment is the same as the previous processing anchor point. Figure 4 The diagram shows a selection diagram of the target reference video segment corresponding to the current segment when the starting position of the intermediate reference video segment is the same as the preceding processing anchor point. Specifically, when r is an odd number and dir is 1, the ending position E of the intermediate reference video segment is used as the first starting position, and the starting position S of the intermediate reference video segment is used as the first ending position, and multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or ending position of each segment should be S or E, and for two adjacent segments, the ending position of the preceding segment is the same as the starting position of the succeeding segment. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), if r≥3, the reference video segment is selected as pair ([S,E], [E,S]); stage 2), then select fragment [S,E], fragment and snippets When r>3, the above two stages can be exchanged in order. When r≥5, the fragments determined in stage 1) can be placed between any two pairs of fragments.

[0097] When r is an odd number and dir is -1, the starting position S of the middle reference video segment is used as the first starting position, and the ending position E of the middle reference video segment is used as the first ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, where the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of the two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), select segments [S, E], segment [S, E], and segment [E] in turn. and snippets Stage 2), if r≥3, select the reference video segment as For ([E, S], [S, E]), when certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stages 1) and 2).

[0098] When r is an even number and dir is 1, the starting position S of the middle reference video segment is used as the first starting position, and the starting position S of the middle reference video segment is used as the first ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of the two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), selecting the segment and Stage 2), select Reference video clip pairs, each pair contains a clip [S, E] and a clip [E, S]. When r>0, the order of the above two stages can be swapped. When r≥4, the clip determined in stage 1) can be placed between any two pairs of clips.

[0099] When r is an even number and dir is -1, the end position E of the middle reference video segment is used as the first starting position, and the end position E of the middle reference video segment is used as the first ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of the two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), the reference video segment is selected as For ([E,S], [S,E]) segments; stage 2), select the segment and snippets When certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stages 1) and 2).

[0100] Step 16: Insert the target reference video segment corresponding to the current segment into the head of the determined target reference video segment.

[0101] Step 17: If the preceding segment of the current segment exists in the plurality of driving voice segments, the process jumps to step 11.

[0102] If there are preceding segments of the current segment in the multiple driving speech segments, steps 11 to 16 are repeated until corresponding target reference video segments are determined for all preceding segments.

[0103] (2) If there is a subsequent segment of the first segment among the multiple driving speech segments, the first segment is used as the current segment, and the corresponding target reference video segment is determined for each subsequent segment from the multiple reference video segments in sequence according to the following steps 21 to 27:

[0104] Step 21: Use the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, select a segment adjacent to the current segment from the subsequent segments, and update the adjacent segment to the current segment.

[0105] The subsequent processing anchor point refers to the anchor point used to select the reference video segment for the subsequent segment. Update the adjacent segment of the current segment to the current segment, that is, the new current segment, and obtain the duration T of the current segment. d , emotional type, whether it is a silent clip.

[0106] Step 22, based on the emotion type and whether it is a silent segment, query a reference video segment whose end position or starting position is the same as the subsequent processing anchor point from multiple reference video segments as the intermediate reference video segment, and set the offset direction variable to -1 or 1 based on whether the end position or starting position is the same as the subsequent processing anchor point.

[0107] Based on the emotion type and whether it is a silent segment, the same type of reference video segments are selected from multiple reference video segments. Then, the reference video segments with the same ending position or starting position as the anchor point of the subsequent processing are searched from the reference video segments of the same type as the intermediate reference video segments. The starting position of the intermediate reference video segment is recorded as S, its ending position is recorded as E, and its duration is recorded as T. r For ease of processing, the offset direction variable dir can be set to -1 or 1, depending on whether the ending or starting position is the same as the subsequent processing anchor point. If the ending position of the intermediate reference video segment is the same as the subsequent processing anchor point, dir is -1; if the starting position of the intermediate reference video segment is the same as the subsequent processing anchor point, dir is 1. Optionally, if no reference video segment among multiple reference video segments has an emotional type that matches the emotional type of the current segment, a reference video segment with a neutral emotional type can be selected.

[0108] Step 23 , calculate the floor result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculate the duration difference based on the duration of the current segment, the duration of the intermediate reference video segment and the floor result.

[0109] Among them, r is used to represent the rounded result, and g is used to represent the duration difference. Specifically, g=T d -r*T r , where T d Indicates the duration of the current segment, T r Indicates the duration of the intermediate reference video segment.

[0110] Step 24: if the rounding-down result is an odd number and the offset direction variable is 1, the starting position of the intermediate reference video clip is used as the second starting position, and the ending position of the intermediate reference video clip is used as the second ending position; if the rounding-down result is an odd number and the offset direction variable is -1, the ending position of the intermediate reference video clip is used as the second starting position, and the starting position of the intermediate reference video clip is used as the second ending position; if the rounding-down result is an even number and the offset direction variable is 1, the starting position of the intermediate reference video clip is used as the second starting position, and the starting position of the intermediate reference video clip is used as the second ending position; if the rounding-down result is an even number and the offset direction variable is -1, the ending position of the intermediate reference video clip is used as the second starting position, and the ending position of the intermediate reference video clip is used as the second ending position.

[0111] Step 25 : Select at least one direction-variable but continuous segment that meets the preset duration requirement from the intermediate reference video segments based on the duration difference, the second starting position, and the second ending position as the target reference video segment corresponding to the current segment.

[0112] Figure 5 FIG2 shows a schematic diagram of selecting a corresponding target reference video segment for a current segment when the starting position of the intermediate reference video segment is the same as the subsequent processing anchor point. Figure 6The diagram shows a selection diagram of selecting a corresponding target reference video segment for the current segment when the ending position of the intermediate reference video segment is the same as the subsequent processing anchor point. Specifically, when r is an odd number and dir is 1, the starting position S of the intermediate reference video segment is used as the second starting position, and the ending position E of the intermediate reference video segment is used as the second ending position, and multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or ending position of each segment should be S or E, and for two adjacent segments, the ending position of the preceding segment is the same as the starting position of the subsequent segment. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), if r≥3, the reference video segment is selected as pair ([S,E], [E,S]); stage 2), then select fragment [S,E], fragment and snippets When r>3, the above two stages can be exchanged in order. When r≥5, the fragments determined in stage 1) can be placed between any two pairs of fragments.

[0113] When r is an odd number and dir is -1, the end position E of the middle reference video segment is used as the second starting position, and the start position S of the middle reference video segment is used as the second ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, where the start position or end position of each segment should be S or E, and the end position of the preceding segment is the same as the start position of the succeeding segment of the two adjacent segments. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), select segments [S, E], segment [S, E], and segment [E] in turn. and snippets Stage 2), if r≥3, select the reference video segment as For ([E, S], [S, E]), when certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stages 1) and 2).

[0114] When r is an even number and dir is 1, the starting position S of the middle reference video segment is used as the second starting position, and the starting position S of the middle reference video segment is used as the second ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of the two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), selecting the segment and Stage 2), select Reference video clip pairs, each pair contains a clip [S, E] and a clip [E, S]. When r>0, the order of the above two stages can be swapped. When r≥4, the clip determined in stage 1) can be placed between any two pairs of clips.

[0115] When r is an even number and dir is -1, the end position E of the middle reference video segment is used as the second starting position, and the end position E of the middle reference video segment is used as the second ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of the two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), the reference video segment is selected as For ([E,S], [S,E]) segments; stage 2), select the segment and snippets When certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stages 1) and 2).

[0116] Step 26: Insert the target reference video segment corresponding to the current segment into the end of the determined target reference video segment.

[0117] Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

[0118] If there are subsequent segments of the current segment in the multiple driving speech segments, steps 21 to 26 are repeated until corresponding target reference video segments are determined for all subsequent segments.

[0119] Based on the timestamp of the target reference video clip corresponding to each driving speech segment, obtain the target reference video segment corresponding to each driving speech segment. Obtain the corresponding video frame number based on the timestamp. Record the video frame number as index, where index = time_stamp * frame_rate / 1000, where time_stamp represents the timestamp and frame_rate represents the frame rate of the target reference video segment. Obtain the frame image based on the video frame number and save it as a video file.

[0120] Step S104: Utilize the plurality of driving speech segments to drive and lip-modify the corresponding target reference video segments, and connect them to form a target video.

[0121] For the target reference video segment determined in step S103, the corresponding driving voice segment is used for driving and lip correction, and then connected to form a complete target video. Specifically, the target video is generated through the steps of lip correction of the sound segment, processing of the silent segment, and video synthesis.

[0122] 1) Lip Correction for Voiced Clips: For voiced clips, the segmented driving speech segment and its corresponding target reference video segment are simultaneously fed into a lip correction network. The network then corrects the virtual host's lip movements in the target reference video segment based on the driving speech segment, generating a corrected target reference video segment. Common models such as Wav2Lip can be used for the lip correction network.

[0123] 2) Silence segment processing: add a voice channel to the target reference video segment that is a silence segment. For example, using the program library ffmpeg, this can be achieved through the following command: ffmpeg -i ref_x.mp4 -f lavfi-ianullsrc -map 0:v-map 1:a-shortest -y target_x.mp4.

[0124] 3) Video synthesis: The target reference video segments obtained in the above steps are sequentially connected in order from front to back to form a complete target video. For example, using the program library ffmpeg, the connection can be completed by using the -f concat option.

[0125] Through the embodiments of the present application, a target video of any length with coherent virtual anchor movements can be generated based on a short-duration reference video, effectively improving the coherence and realism of the virtual anchor's movements in scenes such as broadcasting and dialogue, and solving the problem of decreased lip synchronization in the generation of long-duration videos in the prior art.

[0126] According to the video generation method based on reference video provided by the embodiment of the present application, the reference video and driving data are processed into sound segments and silent segments according to the content. Compared with the prior art method of generating video by taking the reference video as a whole input, since the single processing time of this scheme is shorter, the lip-sound synchronization problem caused by the generation of long-duration videos can be effectively avoided; considering the duration, emotion type and whether it is a silent segment, etc., the corresponding target reference video segment is determined for each driving voice segment according to segment optimization; and the driving data is processed into sound segments and silent segments. On the one hand, it reduces the video driving of unnecessary silent segments and improves the video generation efficiency. On the other hand, it provides the possibility of selecting reference video segments, and can select reference lip opening and closing for sound segments. Video clips, selecting reference video clips with closed lips for silent clips, and maintaining the emotional similarity between the driving voice clip and the reference video clip, narrowing the visual difference between the reference video and the target video, thereby improving the visual effect of voice driving, and because the initial drive of the silent clip is cancelled, the lip shaking phenomenon caused by the defects of the existing technology in the silent clip is avoided; this scheme can generate a target video of any length and with coherent virtual anchor movements based on a short-length reference video, effectively improving the realism and movement coherence of the virtual anchor voice-driven video generation based on the reference video, and at the same time adopting a segmented generation method to divide the generation task into multiple subtasks, and the dependence between the subtasks is eliminated through reference frame selection, so that the efficiency of video generation can be improved through concurrent execution.

[0127] Figure 7 FIG. 1 shows a structural block diagram of a video generation device based on a reference video according to an embodiment of the present application. Figure 7 As shown, the apparatus includes: a first processing module 710 , a second processing module 720 , a determination module 730 and a generation module 740 .

[0128] The first processing module 710 is adapted to: segment a reference video into a plurality of reference video segments, and determine the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments.

[0129] The second processing module 720 is adapted to: acquire driving data, process the driving data into a plurality of driving speech segments, and determine the emotion types of the plurality of driving speech segments.

[0130] The determination module 730 is adapted to determine a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments according to duration, emotion type, and whether it is a silent segment.

[0131] The generation module 740 is adapted to: utilize multiple driving speech segments to drive and lip-modify corresponding target reference video segments, and connect them to form a target video.

[0132] Optionally, the first processing module 710 is further adapted to: extract voice channel data and visual channel data from a reference video; detect the silent part in the voice channel data, perform preliminary segmentation based on the silent part, and obtain preliminary segmented sound segments and silent segments; refine the segmentation boundaries based on the visual channel data corresponding to the preliminary segmented silent segments to obtain fine segmentation boundaries; correct the segmentation boundaries of the preliminary segmented sound segments and silent segments according to the fine segmentation boundaries to obtain final sound segments and silent segments; use a micro-expression recognition method to identify the emotion types of the sound segments and silent segments; wherein, if a single reference video segment has multiple emotion types, the reference video segment is further divided so that each reference video segment after division has a single emotion type.

[0133] Optionally, the second processing module 720 is further adapted to: if the driving data is audio, detecting the silent part in the audio, dividing the audio into multiple driving voice segments based on the silent part, and determining the emotional types of the multiple driving voice segments through an audio sentiment analysis model; wherein the multiple driving voice segments include sound segments and silent segments; if the driving data is text, dividing the text into multiple sub-texts and determining the emotional types of the multiple sub-texts through a text sentiment analysis model, converting the multiple sub-texts into multiple driving voice segments, and setting the emotional types of the multiple driving voice segments corresponding to the emotional types of the multiple sub-texts.

[0134] Optionally, the determination module 730 is further adapted to: select a first segment from multiple driving voice segments, and select a target reference video segment corresponding to the first segment from multiple reference video segments based on the duration, emotion type, and whether the first segment is a silent segment; wherein the end position of the target reference video segment selected for the first segment is located on the boundary of the divided reference video segments; if the first segment is not the first segment among the multiple driving voice segments, the target reference video segment selected for it is also not the first segment among the multiple reference video segments; if there is a preceding segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and based on the duration, emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment, the corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in turn; if there is a subsequent segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and based on the duration, emotion type, whether it is a silent segment, and the ending position of the target reference video segment corresponding to the current segment, the corresponding target reference video segment is determined for each subsequent segment from the multiple reference video segments in turn.

[0135] Optionally, the determination module 730 is further adapted to:

[0136] Step 11: Using the starting position of the target reference video segment corresponding to the current segment as the preceding processing anchor point, selecting a segment adjacent to the current segment from the preceding segment, and updating the adjacent segment to the current segment;

[0137] Step 12: Based on the emotion type and whether it is a silent segment, a reference video segment whose ending position or starting position is the same as the previous processing anchor point is searched from multiple reference video segments as an intermediate reference video segment, and the offset direction variable is set to -1 or 1 based on whether the ending position or starting position is the same as the previous processing anchor point;

[0138] Step 13, calculating a floor-rounded ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor-rounded ratio;

[0139] Step 14: If the result of the floor rounding is an odd number and the offset direction variable is 1, the end position of the middle reference video segment is used as the first starting position, and the start position of the middle reference video segment is used as the first ending position;

[0140] If the result of the floor rounding is an odd number and the offset direction variable is -1, the starting position of the middle reference video segment is used as the first starting position, and the ending position of the middle reference video segment is used as the first ending position;

[0141] If the result of the floor rounding is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the first starting position, and the starting position of the middle reference video segment is used as the first ending position;

[0142] If the result of the floor rounding is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the first starting position, and the end position of the middle reference video segment is used as the first ending position;

[0143] Step 15: Select at least one continuous segment with variable direction and meeting a preset duration requirement from the intermediate reference video segments based on the duration difference, the first starting position, and the first ending position as a target reference video segment corresponding to the current segment;

[0144] Step 16: inserting the target reference video segment corresponding to the current segment into the head of the determined target reference video segment;

[0145] Step 17: If the preceding segment of the current segment exists in the plurality of driving voice segments, the process jumps to step 11.

[0146] Optionally, the determination module 730 is further adapted to:

[0147] Step 21: Using the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, selecting a segment adjacent to the current segment from the subsequent segments, and updating the adjacent segment to the current segment;

[0148] Step 22: Based on the emotion type and whether it is a silent segment, a reference video segment whose ending position or starting position is the same as the subsequent processing anchor point is searched from multiple reference video segments as an intermediate reference video segment, and an offset direction variable is set to -1 or 1 based on whether the ending position or starting position is the same as the subsequent processing anchor point;

[0149] Step 23, calculating a floor-rounded ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor-rounded ratio;

[0150] Step 24: If the rounding result is an odd number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the ending position of the middle reference video segment is used as the second ending position;

[0151] If the result of the floor rounding is an odd number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second starting position, and the start position of the middle reference video segment is used as the second ending position;

[0152] If the rounding result is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the starting position of the middle reference video segment is used as the second ending position;

[0153] If the result of the floor rounding is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second starting position, and the end position of the middle reference video segment is used as the second ending position;

[0154] Step 25: Select at least one continuous segment with variable direction and meeting a preset duration requirement from the intermediate reference video segments based on the duration difference, the second starting position, and the second ending position as a target reference video segment corresponding to the current segment;

[0155] Step 26: insert the target reference video segment corresponding to the current segment into the end of the determined target reference video segment;

[0156] Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

[0157] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0158] According to the video generation device based on reference video provided by the embodiment of the present application, the reference video and driving data are processed into sound segments and silent segments according to the content. Compared with the prior art method of generating video by taking the reference video as a whole input, since the single processing time of this scheme is shorter, the lip-sound synchronization problem caused by the generation of long-duration videos can be effectively avoided; taking into account the duration, emotion type and whether it is a silent segment, etc., the corresponding target reference video segment is determined for each driving voice segment according to segment optimization; and the driving data is processed into sound segments and silent segments. On the one hand, it reduces the video driving of unnecessary silent segments and improves the video generation efficiency. On the other hand, it provides the possibility of selecting reference video segments, and can select reference lip opening and closing for sound segments. Video clips, selecting reference video clips with closed lips for silent clips, and maintaining the emotional similarity between the driving voice clip and the reference video clip, narrowing the visual difference between the reference video and the target video, thereby improving the visual effect of voice driving, and because the initial drive of the silent clip is cancelled, the lip shaking phenomenon caused by the defects of the existing technology in the silent clip is avoided; this scheme can generate a target video of any length and with coherent virtual anchor movements based on a short-length reference video, effectively improving the realism and movement coherence of the virtual anchor voice-driven video generation based on the reference video, and at the same time adopting a segmented generation method to divide the generation task into multiple subtasks, and the dependence between the subtasks is eliminated through reference frame selection, so that the efficiency of video generation can be improved through concurrent execution.

[0159] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which enables a processor to perform operations corresponding to the video generation method based on a reference video in any of the above method embodiments.

[0160] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the video generation method based on a reference video in any of the above method embodiments.

[0161] Figure 8 A schematic structural diagram of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0162] like Figure 8 As shown, the computing device may include: a processor 802 , a communications interface 804 , a memory 806 , and a communication bus 808 .

[0163] Processor 802, communication interface 804, and memory 806 communicate with each other via communication bus 808. Communication interface 804 is used to communicate with other devices, such as clients or other server network elements. Processor 802 is used to execute program 810, which may specifically perform the steps described in the aforementioned embodiment of the method for generating a video based on a reference video for a computing device.

[0164] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0165] Processor 802 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0166] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0167] Program 810 can be specifically configured to cause processor 802 to execute the reference video-based video generation method described in any of the aforementioned method embodiments. The specific implementation of each step in program 810 can be found in the corresponding descriptions of the corresponding steps and units in the aforementioned reference video-based video generation embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes of the devices and modules described above can refer to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0168] The algorithm and display provided herein are not inherently relevant to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the embodiments of the present application are not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the embodiments of the present application described herein, and the description of the specific languages above is for the purpose of disclosing the best mode of implementation of the embodiments of the present application.

[0169] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0170] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed embodiments of the present application require more features than the features explicitly recited in each claim. More precisely, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the embodiments of the present application.

[0171] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0172] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features that are included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0173] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for performing a part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0174] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The embodiments of the present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A video generation method based on a reference video, characterized in that: include: Segmenting a reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments; Acquiring driving data, processing the driving data into a plurality of driving speech segments, and determining emotion types of the plurality of driving speech segments; Determine a corresponding target reference video segment for each driving speech segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment; Using multiple driving speech segments to drive and lip-modify the corresponding target reference video segments, and then connecting them to form a target video; The step of dividing the reference video into a plurality of reference video segments and determining the emotion types of the plurality of reference video segments further includes: Extracting speech channel data and visual channel data from the reference video; Detecting a silent portion in the voice channel data, and performing preliminary segmentation based on the silent portion to obtain preliminary segmented voiced segments and silent segments; Refining the segmentation boundary according to the visual channel data corresponding to the initially segmented silent segment to obtain a finely segmented segmentation boundary; Correcting the segmentation boundaries of the initially segmented voiced and silent segments according to the segmentation boundaries of the fine segmentation to obtain final voiced and silent segments; The emotion types of the sound segment and the silent segment are identified using a micro-expression recognition method; wherein, if a single reference video segment has multiple emotion types, the reference video segment is further divided so that each divided reference video segment has a single emotion type.

2. The method according to claim 1, characterized in that The step of processing the driving data into a plurality of driving speech segments and determining the emotion types of the plurality of driving speech segments further includes: If the driving data is audio, detecting a silent portion in the audio, segmenting the audio into a plurality of driving speech segments based on the silent portion, and determining the emotion types of the plurality of driving speech segments using an audio emotion analysis model; wherein the plurality of driving speech segments include voiced segments and silent segments; If the driving data is text, the text is divided into multiple sub-texts and the emotion types of the multiple sub-texts are determined through a text sentiment analysis model, the multiple sub-texts are converted into multiple driving voice segments, and the emotion types of the multiple driving voice segments are set accordingly according to the emotion types of the multiple sub-texts.

3. The method according to any one of claims 1-2, characterized in that The step of determining a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments based on duration, emotion type, and whether the segment is a silent segment further includes: A first segment is selected from a plurality of driving speech segments, and a target reference video segment corresponding to the first segment is selected from a plurality of reference video segments based on the duration, emotion type, and whether the first segment is a silent segment; wherein the end position of the target reference video segment selected for the first segment is located at a boundary of the divided reference video segments; if the first segment is not the first segment of the plurality of driving speech segments, the target reference video segment selected for the first segment is also not the first segment of the plurality of reference video segments; If a preceding segment of the first segment exists in the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments based on the duration, emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment; If there is a subsequent segment of the first segment among multiple driving voice segments, the first segment will be taken as the current segment, and the corresponding target reference video segment will be determined for each subsequent segment from multiple reference video segments in turn based on the length, emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment.

4. The method according to claim 3, characterized in that The step of determining a corresponding target reference video segment for each preceding segment from a plurality of reference video segments in sequence based on the duration, the emotion type, whether the segment is a silent segment, and the starting position of the target reference video segment corresponding to the current segment further includes: Step 11: Using the starting position of the target reference video segment corresponding to the current segment as a preceding processing anchor point, selecting a segment adjacent to the current segment from the preceding segment, and updating the adjacent segment as the current segment; Step 12: Based on the emotion type and whether it is a silent segment, a reference video segment whose ending position or starting position is the same as the preceding processing anchor point is searched from multiple reference video segments as an intermediate reference video segment, and an offset direction variable is set to -1 or 1 based on whether the ending position or starting position is the same as the preceding processing anchor point; Step 13, calculating a floor-rounded ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor-rounded ratio; Step 14: If the rounding result is an odd number and the offset direction variable is 1, the end position of the intermediate reference video segment is used as the first starting position, and the start position of the intermediate reference video segment is used as the first ending position; If the rounding-down result is an odd number and the offset direction variable is -1, the starting position of the intermediate reference video segment is used as the first starting position, and the ending position of the intermediate reference video segment is used as the first ending position; If the rounding-down result is an even number and the offset direction variable is 1, the starting position of the intermediate reference video segment is used as the first starting position, and the starting position of the intermediate reference video segment is used as the first ending position; If the rounding-down result is an even number and the offset direction variable is -1, the end position of the intermediate reference video segment is used as the first starting position, and the end position of the intermediate reference video segment is used as the first ending position; Step 15: Select at least one continuous segment with variable direction and meeting a preset duration requirement from the intermediate reference video segments based on the duration difference, the first starting position, and the first ending position as a target reference video segment corresponding to the current segment; Step 16: inserting the target reference video segment corresponding to the current segment into the head of the determined target reference video segment; Step 17: If the preceding segment of the current segment exists in the plurality of driving voice segments, the process jumps to step 11.

5. The method according to claim 3, characterized in that The step of determining a corresponding target reference video segment for each subsequent segment from a plurality of reference video segments based on the duration, the emotion type, whether the segment is a silent segment, and the end position of the target reference video segment corresponding to the current segment further includes: Step 21: Using the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, selecting a segment adjacent to the current segment from the subsequent segments, and updating the adjacent segment as the current segment; Step 22: Based on the emotion type and whether it is a silent segment, a reference video segment whose ending position or starting position is the same as the post-processing anchor point is searched from multiple reference video segments as an intermediate reference video segment, and an offset direction variable is set to -1 or 1 based on whether the ending position or starting position is the same as the post-processing anchor point. Step 23, calculating a floor-rounded ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment, and the floor-rounded ratio; Step 24: If the rounding result is an odd number and the offset direction variable is 1, the starting position of the intermediate reference video segment is used as the second starting position, and the ending position of the intermediate reference video segment is used as the second ending position; If the rounding-down result is an odd number and the offset direction variable is -1, the end position of the intermediate reference video segment is used as the second starting position, and the start position of the intermediate reference video segment is used as the second ending position; If the rounding-down result is an even number and the offset direction variable is 1, the starting position of the intermediate reference video segment is used as the second starting position, and the starting position of the intermediate reference video segment is used as the second ending position; If the rounding-down result is an even number and the offset direction variable is -1, the end position of the intermediate reference video segment is used as the second starting position, and the end position of the intermediate reference video segment is used as the second ending position; Step 25: Select at least one continuous segment with variable direction and meeting a preset duration requirement from the intermediate reference video segments based on the duration difference, the second starting position, and the second ending position as a target reference video segment corresponding to the current segment; Step 26: insert the target reference video segment corresponding to the current segment into the end of the determined target reference video segment; Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

6. A video generation device based on a reference video, characterized in that: include: A first processing module is adapted to segment a reference video into a plurality of reference video segments and determine emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments; a second processing module adapted to acquire driving data, process the driving data into a plurality of driving speech segments, and determine emotion types of the plurality of driving speech segments; a determination module adapted to determine a corresponding target reference video segment for each driving speech segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment; A generation module is adapted to use a plurality of driving speech segments to drive and lip-modify the corresponding target reference video segments, and connect them to form a target video; Wherein, the first processing module is further adapted to: Extracting speech channel data and visual channel data from the reference video; Detecting a silent portion in the voice channel data, and performing preliminary segmentation based on the silent portion to obtain preliminary segmented voiced segments and silent segments; Refining the segmentation boundary according to the visual channel data corresponding to the initially segmented silent segment to obtain a finely segmented segmentation boundary; Correcting the segmentation boundaries of the initially segmented voiced and silent segments according to the segmentation boundaries of the fine segmentation to obtain final voiced and silent segments; The emotion types of the sound segment and the silent segment are identified using a micro-expression recognition method; wherein, if a single reference video segment has multiple emotion types, the reference video segment is further divided so that each divided reference video segment has a single emotion type.

7. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video generation method based on a reference video according to any one of claims 1 to 5.

8. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and wherein the executable instruction enables a processor to execute operations corresponding to the method for generating a video based on a reference video according to any one of claims 1 to 5.

9. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the method for generating a video based on a reference video according to any one of claims 1 to 5.

Images