Video generation method and device based on reference video, computing equipment and medium

Abstract

The embodiment of the invention discloses a video generation method and device based on a reference video, computing equipment and a medium, and the method comprises the steps: segmenting the reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments; obtaining driving data, processing the driving data into a plurality of driving voice segments, and determining emotion types of the plurality of driving voice segments; determining a corresponding target reference video clip for each driving voice clip from the plurality of reference video clips according to the duration, the emotion type and whether the driving voice clip is a mute clip; and performing driving and mouth shape correction on the corresponding target reference video clips by using the plurality of driving voice clips, and connecting to form a target video. According to the method and the device, the target video with any duration and coherent virtual anchor actions can be generated according to the reference video with the short duration, and the fidelity and action coherence of virtual anchor voice driven video generation based on the reference video are effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and specifically to a method, apparatus, computing device, and medium for generating a video based on a reference video. Background Art

[0002] With the continuous advancement of artificial intelligence technology, technologies such as Wav2Lip that use voice to drive images or videos to generate corresponding lip movements, head movements, and even subtle expressions have developed rapidly, forming the current research boom in virtual anchor video generation. Usually, the reference video is used as a whole input to generate the video.

[0003] However, in actual use, it was found that the existing generation method has the following problems: 1) The duration of the generated virtual anchor video is limited. After a certain duration, the lip synchronization decreases; 2) The mouth does not close when silent, which may be due to the noise in the training video. As a result, the generated virtual anchor video still has opening and closing lips and shaking movements even when silent, which seriously reduces the lip synchronization; 3) Affected by the length of the reference video, frame skipping will occur when generating a video with a longer duration, resulting in video incoherence; 4) The expression does not match the driving audio, resulting in distortion of the generated video. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a method, apparatus, computing device and medium for generating a video based on a reference video that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of an embodiment of the present application, a video generation method based on a reference video is provided, comprising:

[0006] Segmenting the reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments;

[0007] Acquire driving data, process the driving data into a plurality of driving speech segments, and determine emotion types of the plurality of driving speech segments;

[0008] Determine a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments according to the duration, the emotion type, and whether it is a silent segment;

[0009] A plurality of driving speech segments are used to drive and lip-modify the corresponding target reference video segments, and then connected to form a target video.

[0010] Furthermore, segmenting the reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments further comprises:

[0011] Extracting speech channel data and visual channel data from a reference video;

[0012] Detect the silent part in the voice channel data, perform preliminary segmentation based on the silent part, and obtain preliminary segmented voiced segments and silent segments;

[0013] According to the visual channel data corresponding to the initially segmented silent segment, the segmentation boundary is refined to obtain a finely segmented segmentation boundary;

[0014] The segmentation boundaries of the initially segmented voiced segments and the silent segments are modified according to the segmentation boundaries of the fine segmentation to obtain the final voiced segments and the silent segments;

[0015] The micro-expression recognition method is used to identify the emotion types of the sound clips and the silent clips; if a single reference video clip has multiple emotion types, the reference video clip is divided again so that each of the divided reference video clips has a single emotion type.

[0016] Furthermore, processing the driving data into a plurality of driving speech segments, and determining the emotion types of the plurality of driving speech segments further comprises:

[0017] If the driving data is audio, then the silent part in the audio is detected, and the audio is divided into multiple driving voice segments according to the silent part, and the emotional types of the multiple driving voice segments are determined by the audio emotion analysis model; wherein the multiple driving voice segments include voiced segments and silent segments;

[0018] If the driving data is text, the text is divided into multiple sub-texts and the emotion types of the multiple sub-texts are determined through a text sentiment analysis model, the multiple sub-texts are converted into multiple driving voice segments, and the emotion types of the multiple driving voice segments are set accordingly according to the emotion types of the multiple sub-texts.

[0019] Furthermore, determining a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments according to the duration, the emotion type, and whether it is a silent segment further includes:

[0020] A first segment is selected from a plurality of driving speech segments, and a target reference video segment corresponding to the first segment is selected from a plurality of reference video segments according to the duration, emotion type, and whether the first segment is a silent segment; wherein the end position of the target reference video segment selected for the first segment is located on the boundary of the divided reference video segments; if the first segment is not the first segment among the plurality of driving speech segments, the target reference video segment selected for it is also not the first segment among the plurality of reference video segments;

[0021] If there is a preceding segment of the first segment among the multiple driving voice segments, the first segment is taken as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in turn according to the duration, the emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment;

[0022] If there is a subsequent segment of the first segment among multiple driving voice segments, the first segment is taken as the current segment, and the corresponding target reference video segment is determined for each subsequent segment from the multiple reference video segments in turn based on the length, emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment.

[0023] Furthermore, determining a corresponding target reference video segment for each preceding segment from a plurality of reference video segments in sequence according to the duration, the emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment further includes:

[0024] Step 11, taking the starting position of the target reference video segment corresponding to the current segment as the preceding processing anchor point, selecting a segment adjacent to the current segment from the preceding segment, and updating the adjacent segment to the current segment;

[0025] Step 12, according to the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the previous processing anchor point from multiple reference video segments as an intermediate reference video segment, and set the offset direction variable to -1 or 1 according to whether the end position or the start position is the same as the previous processing anchor point;

[0026] Step 13, calculating a round-down result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating the duration difference according to the duration of the current segment, the duration of the intermediate reference video segment and the round-down result;

[0027] Step 14, if the result of rounding down is an odd number and the offset direction variable is 1, the end position of the middle reference video segment is used as the first start position, and the start position of the middle reference video segment is used as the first end position;

[0028] If the result of rounding down is an odd number and the offset direction variable is -1, the starting position of the middle reference video segment is used as the first starting position, and the ending position of the middle reference video segment is used as the first ending position;

[0029] If the result of rounding down is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the first starting position, and the starting position of the middle reference video segment is used as the first ending position;

[0030] If the result of rounding down is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the first starting position, and the end position of the middle reference video segment is used as the first ending position;

[0031] Step 15, selecting at least one segment with variable direction but continuous and meeting preset duration requirements from the intermediate reference video segments according to the duration difference, the first starting position and the first ending position as the target reference video segment corresponding to the current segment;

[0032] Step 16, inserting the target reference video segment corresponding to the current segment into the head of the determined target reference video segment;

[0033] Step 17: If there is a preceding segment of the current segment in the multiple driving voice segments, jump to step 11.

[0034] Furthermore, determining a corresponding target reference video segment for each subsequent segment from a plurality of reference video segments in sequence according to the duration, the emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment further includes:

[0035] Step 21, taking the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, selecting an adjacent segment to the current segment from the subsequent segments, and updating the adjacent segment to the current segment;

[0036] Step 22, according to the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the subsequent processing anchor point from multiple reference video segments as an intermediate reference video segment, and set the offset direction variable to -1 or 1 according to whether the end position or the start position is the same as the subsequent processing anchor point;

[0037] Step 23, calculating a round-down result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating the duration difference according to the duration of the current segment, the duration of the intermediate reference video segment and the round-down result;

[0038] Step 24, if the result of rounding down is an odd number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the ending position of the middle reference video segment is used as the second ending position;

[0039] If the result of rounding down is an odd number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second start position, and the start position of the middle reference video segment is used as the second end position;

[0040] If the rounding result is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the starting position of the middle reference video segment is used as the second ending position;

[0041] If the result of rounding down is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second starting position, and the end position of the middle reference video segment is used as the second end position;

[0042] Step 25, selecting at least one segment with variable direction but continuous and meeting the preset duration requirement from the intermediate reference video segments according to the duration difference, the second starting position and the second ending position as the target reference video segment corresponding to the current segment;

[0043] Step 26, inserting the target reference video segment corresponding to the current segment into the end of the determined target reference video segment;

[0044] Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

[0045] According to another aspect of an embodiment of the present application, a video generation device based on a reference video is provided, including:

[0046] The first processing module is adapted to divide the reference video into a plurality of reference video segments and determine the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments;

[0047] a second processing module adapted to obtain driving data, process the driving data into a plurality of driving speech segments, and determine emotion types of the plurality of driving speech segments;

[0048] A determination module, adapted to determine a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment;

[0049] The generation module is suitable for using multiple driving voice segments to drive and lip-modify the corresponding target reference video segments, and connect them to form a target video.

[0050] According to another aspect of the embodiment of the present application, there is provided a computing device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0051] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned video generation method based on the reference video.

[0052] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored. The executable instruction enables a processor to perform operations corresponding to the above-mentioned video generation method based on a reference video.

[0053] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned video generation method based on a reference video.

[0054] According to the technical solution provided in the embodiment of the present application, the reference video and the driving data are processed into sound segments and silent segments according to the content. Compared with the prior art method of generating a video by inputting the reference video as a whole, the present solution has a shorter single processing time and can effectively avoid the lip-sound asynchrony problem caused by generating a long-duration video. The duration, emotion type, and whether it is a silent segment are comprehensively considered, and the corresponding target reference video segment is determined for each driving voice segment according to segment optimization. In addition, the driving data is processed into sound segments and silent segments. On the one hand, unnecessary video driving of silent segments is reduced, and the video generation efficiency is improved. On the other hand, it provides the possibility of selecting a reference video segment, and a reference video segment of lip opening and closing can be selected for the sound segment. Selecting a reference video clip with closed lips for the silent clip can also maintain the emotional similarity between the driving voice clip and the reference video clip, narrowing the visual difference between the reference video and the target video, thereby improving the visual effect of voice-driven, and because the initial drive of the silent clip is cancelled, the lip shaking phenomenon of the silent clip caused by the defects of the prior art is avoided; this scheme can generate a target video of any length with coherent virtual host movements based on a short-duration reference video, effectively improving the realism and movement coherence of the virtual host voice-driven video generation based on the reference video, and at the same time, the generation task is divided into multiple subtasks by a segmented generation method, and the dependencies between the subtasks are eliminated by reference frame selection, so that the efficiency of video generation can be improved through concurrent execution.

[0055] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0057] Figure 1 A schematic diagram of a process of a video generation method based on a reference video according to an embodiment of the present application is shown;

[0058] Figure 2 A schematic diagram of segmentation of the initially segmented voiced segments and silent segments is shown;

[0059] Figure 3 A schematic diagram showing a selection of a corresponding target reference video segment for a current segment when the end position of the intermediate reference video segment is the same as the previous processing anchor point;

[0060] Figure 4 A schematic diagram showing a selection of a corresponding target reference video segment for a current segment when the starting position of the intermediate reference video segment is the same as the previous processing anchor point;

[0061] Figure 5 A schematic diagram showing a selection of a corresponding target reference video segment for a current segment when the starting position of the intermediate reference video segment is the same as the subsequent processing anchor point;

[0062] Figure 6 A schematic diagram showing the selection of a corresponding target reference video segment for a current segment when the end position of the intermediate reference video segment is the same as the subsequent processing anchor point;

[0063] Figure 7 A structural block diagram of a video generation device based on a reference video according to an embodiment of the present application is shown;

[0064] Figure 8 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] Figure 1FIG. 4 shows a flow chart of a method for generating a video based on a reference video according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0067] Step S101: segment a reference video into a plurality of reference video segments, and determine the emotion types of the plurality of reference video segments.

[0068] The embodiment of the present application proposes a method for generating a segmented video, and selects a video with a similar voice signal as a reference video for the target video. The image of the anchor in the target video finally generated is the same as the image of the anchor in the reference video. In the embodiment of the present application, the anchor in the target video is called a virtual anchor. The reference video is segmented, and the reference video is divided into multiple reference video segments through silence detection. The multiple reference video segments include sound segments and silent segments. In step S101, the accurate segmentation of the reference video segments can be achieved through the double-layer segmentation processing of preliminary segmentation and fine segmentation.

[0069] Specifically, voice channel data and visual channel data are extracted from a reference video; the silent part in the voice channel data is detected, and a preliminary segmentation is performed based on the silent part to obtain preliminary segmented voice segments and silent segments; the segmentation boundary is refined based on the visual channel data corresponding to the preliminary segmented silent segments to obtain fine segmentation boundary; the segmentation boundary of the preliminary segmented voice segments and silent segments is corrected according to the fine segmentation boundary to obtain the final voice segments and silent segments; the micro-expression recognition method is used to identify the emotional type of the voice segments and silent segments.

[0070] Extract the voice channel data and the visual channel data from the reference video, which can be accomplished using the open source library ffmpeg: use the command ffmpeg-iinput.mp4-vn-acodec copy audio.mp3 to extract the voice channel data, and use the command ffmpeg-iinput.mp4-an-vcodec copy video.mp4 to extract the visual channel data. Then detect the silent part in the extracted voice channel data, perform a preliminary segmentation of the voice channel data according to the silent part, and segment it into a plurality of continuous voice segments and silent segments, i.e., the preliminary segmented voice segments and silent segments, and use the timestamp of the voice channel data to identify the voice segments and silent segments, and the unit of the timestamp is milliseconds (ms).

[0071] Considering that in actual application scenarios, there may be two problems: 1) Silent segments are often not completely without sound, such as Figure 2As shown; 2) The sound clip may also contain short silences. For this reason, it is necessary to set an amplitude threshold and a minimum duration threshold in silence detection, wherein the amplitude threshold is generally set to dBFS-16. dBFS (decibels relative to full scale) is a unit that describes the amplitude of an audio signal in a digital system. It represents the decibel value of the signal level relative to the full scale of the digital system. The full scale of dBFS is generally considered to be the maximum level that can be handled in a digital audio system. The shortest duration threshold can be set to 1500ms. Those skilled in the art can make appropriate adjustments to these two thresholds according to actual application needs.

[0072] According to the visual channel data (i.e., picture information) corresponding to the initially segmented silent segment, the segmentation boundary is refined, such as Figure 2 As shown: 1) If the mouth opens and closes in the tail sub-segment of a silent segment among multiple reference video segments, such sub-segments are split and marked as sound segments, and merged with the subsequent sound segments; 2) If the mouth opens and closes in the head sub-segment of a silent segment among multiple reference video segments, such segments are split and marked as sound segments, and if there is a preceding segment, it should be merged with the preceding sound segment. The segmentation boundaries of the initially segmented sound segments and silent segments are corrected according to the segmentation boundaries of the above-mentioned fine segmentation, thereby obtaining sound segments and silent segments. Among them, the judgment of the mouth opening and closing can be determined by detecting the changes in the positions of the lip key points obtained using the program library Openface.

[0073] The micro-expression recognition method is used to identify the emotion types of the sound clips and the silent clips, and the emotion types may include neutral, happy, angry, sad, etc. Those skilled in the art may set the emotion types according to actual needs, such as adding or deleting the types of emotion types or further subdividing them. Optionally, if a single reference video clip has multiple emotion types, the reference video clip is divided again so that each reference video clip after division has a single emotion type. For example, the reference video clip may be divided twice, and finally divided into multiple continuous clips, and each clip has a single emotion type.

[0074] After multiple segmentations, the final silent segment is obtained, and a silent segment list is established. Each node in the silent segment list includes at least the segment's <starting position, ending position, duration, silent mark, and emotional type>. Based on the silent segment, a corresponding sound segment list is established, and each node in the sound segment list also includes at least the segment's <starting position, ending position, duration, silent mark, and emotional type>.

[0075] Optionally, the voiced segments can be further divided into three categories: bidirectionally repeatable, unidirectionally repeatable, and non-repeatable, so that the corresponding target reference video segment can be determined as the driving voice segment according to whether it is repeatable.

[0076] Step S102: Acquire driving data, process the driving data into a plurality of driving speech segments, and determine the emotion types of the plurality of driving speech segments.

[0077] The driving data is data used to reflect the video copy content of the target video to be generated, for example, the driving data may be news releases, advertising introduction information, etc. The driving data may be data in audio or text form.

[0078] If the driving data is audio, the silent part in the audio is detected, and the audio is divided into a plurality of continuous driving voice segments according to the silent part, and the emotion types of the plurality of driving voice segments are determined by the audio emotion analysis model. The plurality of driving voice segments include vocal segments and silent segments, and the emotion type of each vocal segment can be determined by the audio emotion analysis model. Optionally, for silent segments, it is not necessary to determine their emotion types.

[0079] If the driving data is text, the text is divided into multiple sub-texts and the emotion types of the multiple sub-texts are determined through a text sentiment analysis model. Then, the multiple sub-texts are converted into multiple driving voice segments through speech synthesis or TTS (Text-to-Speech) technology, and the emotion types of the multiple driving voice segments are set accordingly according to the emotion types of the multiple sub-texts.

[0080] The emotion types of the multiple driving voice segments are consistent with the emotion types of the multiple reference video segments. The multiple driving voice segments are arranged in order from front to back, and the multiple driving voice segments include <starting position, ending position, duration, silence mark, emotion type>. The total duration of the multiple driving voice segments is the same as the duration of the target video.

[0081] Step S103 , determining a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments according to the duration, the emotion type, and whether it is a silent segment.

[0082] Considering that the flexibility of segmented driving may cause the virtual anchor's actions to be incoherent across segments, the embodiment of the present application specifically proposes a cross-segment processing strategy. To ensure the coherence of the target video, the reference video segment information of adjacent segments is considered and a continuous processing method is selected.

[0083] In step S103, a first segment can be selected from multiple driving voice segments, and a target reference video segment corresponding to the first segment can be selected from multiple reference video segments based on the duration, emotion type, and whether it is a silent segment of the first segment; if there is a preceding segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in turn based on the duration, emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment; if there is a subsequent segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each subsequent segment from the multiple reference video segments in turn based on the duration, emotion type, whether it is a silent segment, and the ending position of the target reference video segment corresponding to the current segment.

[0084] Among them, according to certain rules, a certain driving speech segment is selected from multiple driving speech segments as the first segment. For the selection of the first segment, the driving speech segment with the longest duration can be selected, or the longest sound segment can be selected from multiple driving speech segments to ensure that the best target reference video segment can be obtained. d , emotion type, and whether it is a silent segment, a reference video segment that meets the requirements is selected from the reference video segments of the same type (emotion type, whether it is a silent segment) as the target reference video segment corresponding to the first segment, and the duration of the target reference video segment is recorded as T r , and record the starting position and ending position of the target reference video segment. Optionally, when there is no reference video segment whose emotion type matches the emotion type of the first segment among the multiple reference video segments, a reference video segment with a neutral emotion type can be selected.

[0085] If the first segment is not the first segment among the multiple driving speech segments, then the target reference video segment selected for it should not be the first segment among the multiple reference video segments. r ≤T d ≤(t+1)·T r, t is an integer, which increases upward from 0 until a reference video segment that meets the requirements is found as the intermediate reference video segment corresponding to the first segment. Among them, the end position of the target reference video segment selected for the first segment is located on the boundary of the divided reference video segments. In order to meet its requirements, after selecting the intermediate reference video segment corresponding to the first segment from multiple reference video segments, the ratio of the duration of the first segment to the duration of the selected intermediate reference video segment can be calculated. The result r of the rounding down can be calculated; when the rounding down result r is an even number, a time length T is selected from the intermediate reference video segment with the starting position of the intermediate reference video segment as the selected starting position and the selected ending position as the starting position. d The continuous multiple directional variable segments are used as the target reference video segments corresponding to the first segment; when the rounded result r is an odd number, the duration of T is selected from the intermediate reference video segment with the starting position of the intermediate reference video segment as the selected starting position and the ending position as the selected ending position. d At least one continuous direction-variable segment is used as a target reference video segment corresponding to the first segment.

[0086] Specifically, the starting position of the intermediate reference video segment is recorded as S, and the ending position is recorded as E. When the rounding result r is an even number, the starting position S of the intermediate reference video segment is used as the starting position, and the starting position S of the intermediate reference video segment is used as the ending position. A video clip with a duration of T is selected from the intermediate reference video segment. d The continuous multiple variable direction segments are used as the target reference video segments corresponding to the first segment; when the rounded result r is an odd number, the starting position S of the intermediate reference video segment is used as the selection starting position, the ending position E of the intermediate reference video segment is used as the selection ending position, and a duration T is selected from the intermediate reference video segment. d At least one continuous direction-variable segment is used as a target reference video segment corresponding to the first segment.

[0087] (1) If there is a preceding segment of the first segment among the multiple driving speech segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in sequence according to the following steps 11 to 17:

[0088] Step 11: Use the starting position of the target reference video segment corresponding to the current segment as a preceding processing anchor point, select an adjacent segment to the current segment from the preceding segment, and update the adjacent segment to the current segment.

[0089] The preceding processing anchor point refers to the anchor point used to select the reference video segment for the preceding segment. Update the adjacent segment of the current segment to the current segment, that is, the new current segment, and obtain the duration T of the current segment. d , emotion type, whether it is a silent clip.

[0090] Step 12, based on the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the previous processing anchor point from multiple reference video segments as the intermediate reference video segment, and set the offset direction variable to -1 or 1 based on whether the end position or start position is the same as the previous processing anchor point.

[0091] According to the emotion type and whether it is a silent segment, the same type of reference video segments are selected from multiple reference video segments. Then, the reference video segment with the same end position or start position as the anchor point of the previous processing is queried from the reference video segment of the same type as the intermediate reference video segment. The start position of the intermediate reference video segment is recorded as S, the end position is recorded as E, and the duration is recorded as T. r , for ease of processing, the offset direction variable dir can be set to -1 or 1 depending on whether the end position or the start position is the same as the previous processing anchor point. Wherein, when the end position of the intermediate reference video segment is the same as the previous processing anchor point, dir is -1; when the start position of the intermediate reference video segment is the same as the previous processing anchor point, dir is 1. Optionally, when there is no reference video segment whose emotion type matches the emotion type of the current segment among the multiple reference video segments, a reference video segment with a neutral emotion type can be selected.

[0092] Step 13, calculate the floor result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculate the duration difference based on the duration of the current segment, the duration of the intermediate reference video segment and the floor result.

[0093] In the embodiment of the present application, r is used to represent the rounding result, and g is used to represent the time difference. Specifically, g=T d -r*T r , where T d Indicates the duration of the current segment, T r Indicates the duration of the intermediate reference video segment.

[0094] Step 14, if the rounding result is an odd number and the offset direction variable is 1, the end position of the middle reference video clip is used as the first starting position, and the starting position of the middle reference video clip is used as the first ending position; if the rounding result is an odd number and the offset direction variable is -1, the starting position of the middle reference video clip is used as the first starting position, and the end position of the middle reference video clip is used as the first ending position; if the rounding result is an even number and the offset direction variable is 1, the starting position of the middle reference video clip is used as the first starting position, and the starting position of the middle reference video clip is used as the first ending position; if the rounding result is an even number and the offset direction variable is -1, the end position of the middle reference video clip is used as the first starting position, and the end position of the middle reference video clip is used as the first ending position.

[0095] Step 15, selecting at least one segment with variable direction but continuous and meeting preset duration requirements from the intermediate reference video segments as the target reference video segment corresponding to the current segment according to the duration difference, the first starting position and the first ending position.

[0096] Figure 3 FIG. 1 shows a schematic diagram of selecting a corresponding target reference video segment for a current segment when the end position of the intermediate reference video segment is the same as the previous processing anchor point. Figure 4 A schematic diagram of selecting a target reference video segment corresponding to the current segment when the starting position of the intermediate reference video segment is the same as the previous processing anchor point is shown. Specifically, when r is an odd number and dir is 1, the ending position E of the intermediate reference video segment is used as the first starting position, and the starting position S of the intermediate reference video segment is used as the first ending position, and multiple segments that meet the preset time length requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or the ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), if r≥3, select the reference video segment as pair ([S,E], [E,S]); stage 2), then select fragment [S,E], fragment and snippets When r>3, the above two stages can be exchanged in order. When r≥5, the fragments determined in stage 1) can be placed between any two pairs of fragments.

[0097] When r is an odd number and dir is -1, the starting position S of the middle reference video segment is used as the first starting position, and the ending position E of the middle reference video segment is used as the first ending position. Multiple segments that meet the preset duration requirements are selected as the target reference video segments corresponding to the current segment, where the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), select segments [S, E], and and snippets Stage 2), if r ≥ 3, select the reference video segment as For ([E, S], [S, E]), when certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stage 1) and stage 2).

[0098] When r is an even number and dir is 1, the starting position S of the middle reference video segment is used as the first starting position, and the starting position S of the middle reference video segment is used as the first ending position. Multiple segments that meet the preset duration requirements are selected as the target reference video segments corresponding to the current segment, where the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), selecting the segment and Stage 2), select reference video clip pairs, each pair contains a clip [S, E] and a clip [E, S]. When r>0, the above two stages can be swapped in order. When r≥4, the clips determined in stage 1) can be placed between any two pairs of clips.

[0099] When r is an even number and dir is -1, the end position E of the middle reference video segment is used as the first starting position, and the end position E of the middle reference video segment is used as the first ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, where the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), selecting the reference video segment as For ([E,S], [S,E]) fragment; stage 2), select the fragment and snippets When certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stage 1) and stage 2).

[0100] Step 16: insert the target reference video segment corresponding to the current segment into the head of the determined target reference video segment.

[0101] Step 17: If there is a preceding segment of the current segment in the multiple driving voice segments, jump to step 11.

[0102] If there are preceding segments of the current segment in the multiple driving speech segments, steps 11 to 16 are repeated until corresponding target reference video segments are determined for all preceding segments.

[0103] (2) If there is a subsequent segment of the first segment among the multiple driving speech segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each subsequent segment from the multiple reference video segments in sequence according to the following steps 21 to 27:

[0104] Step 21, taking the end position of the target reference video segment corresponding to the current segment as the subsequent processing anchor point, selecting an adjacent segment to the current segment from the subsequent segments, and updating the adjacent segment to the current segment.

[0105] The subsequent processing anchor point refers to the anchor point used to select the reference video segment for the subsequent segment. Update the adjacent segment of the current segment to the current segment, that is, the new current segment, and obtain the duration T of the current segment. d , emotion type, whether it is a silent clip.

[0106] Step 22, based on the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the subsequent processing anchor point from multiple reference video segments as an intermediate reference video segment, and set the offset direction variable to -1 or 1 based on whether the end position or start position is the same as the subsequent processing anchor point.

[0107] According to the emotion type and whether it is a silent segment, the same type of reference video segments are selected from multiple reference video segments. Then, the reference video segments with the same end position or start position as the anchor point of the subsequent processing are queried from the same type of reference video segments as the intermediate reference video segments. The starting position of the intermediate reference video segment is recorded as S, its end position is recorded as E, and its duration is recorded as T. r , for ease of processing, the offset direction variable dir can be set to -1 or 1 depending on whether the end position or the start position is the same as the post-processing anchor point. Wherein, when the end position of the intermediate reference video segment is the same as the post-processing anchor point, dir is -1; when the start position of the intermediate reference video segment is the same as the post-processing anchor point, dir is 1. Optionally, when there is no reference video segment whose emotion type matches the emotion type of the current segment among multiple reference video segments, a reference video segment with a neutral emotion type can be selected.

[0108] Step 23, calculate the floor result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculate the duration difference based on the duration of the current segment, the duration of the intermediate reference video segment and the floor result.

[0109] Among them, r is used to represent the rounded result, and g is used to represent the duration difference. Specifically, g=T d -r*T r , where T d Indicates the duration of the current segment, T r Indicates the duration of the intermediate reference video segment.

[0110] Step 24, if the rounding result is an odd number and the offset direction variable is 1, the starting position of the middle reference video clip is used as the second starting position, and the ending position of the middle reference video clip is used as the second ending position; if the rounding result is an odd number and the offset direction variable is -1, the ending position of the middle reference video clip is used as the second starting position, and the starting position of the middle reference video clip is used as the second ending position; if the rounding result is an even number and the offset direction variable is 1, the starting position of the middle reference video clip is used as the second starting position, and the starting position of the middle reference video clip is used as the second ending position; if the rounding result is an even number and the offset direction variable is -1, the ending position of the middle reference video clip is used as the second starting position, and the ending position of the middle reference video clip is used as the second ending position.

[0111] Step 25, selecting at least one segment with variable direction but continuous and meeting preset duration requirements from the intermediate reference video segments according to the duration difference, the second starting position and the second ending position as the target reference video segment corresponding to the current segment.

[0112] Figure 5 A schematic diagram showing a selection of a corresponding target reference video segment for a current segment when the starting position of the intermediate reference video segment is the same as the subsequent processing anchor point is shown. Figure 6A schematic diagram of selecting a target reference video segment corresponding to the current segment when the end position of the intermediate reference video segment is the same as the subsequent processing anchor point is shown. Specifically, when r is an odd number and dir is 1, the starting position S of the intermediate reference video segment is used as the second starting position, and the end position E of the intermediate reference video segment is used as the second end position, and multiple segments that meet the preset time length requirements are selected as the target reference video segments corresponding to the current segment, wherein the starting position or end position of each segment should be S or E, and the end position of the preceding segment of two adjacent segments is the same as the starting position of the succeeding segment. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), if r ≥ 3, select the reference video segment as pair ([S,E], [E,S]); stage 2), then select fragment [S,E], fragment and snippets When r>3, the above two stages can be exchanged in order. When r≥5, the fragments determined in stage 1) can be placed between any two pairs of fragments.

[0113] When r is an odd number and dir is -1, the end position E of the middle reference video segment is used as the second starting position, and the start position S of the middle reference video segment is used as the second ending position. Multiple segments that meet the preset duration requirements are selected as the target reference video segments corresponding to the current segment, where the start position or end position of each segment should be S or E, and the end position of the preceding segment and the start position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), select segments [S, E], and then select segments [S, E] and [E, S]. and snippets Stage 2), if r ≥ 3, select the reference video segment as For ([E, S], [S, E]), when certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stage 1) and stage 2).

[0114] When r is an even number and dir is 1, the starting position S of the middle reference video segment is used as the second starting position, and the starting position S of the middle reference video segment is used as the second ending position. Multiple segments that meet the preset duration requirements are selected as the target reference video segments corresponding to the current segment, where the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), selecting the segment and Stage 2), select reference video clip pairs, each pair contains a clip [S, E] and a clip [E, S]. When r>0, the above two stages can be swapped in order. When r≥4, the clips determined in stage 1) can be placed between any two pairs of clips.

[0115] When r is an even number and dir is -1, the end position E of the middle reference video segment is used as the second starting position, and the end position E of the middle reference video segment is used as the second ending position. Multiple segments that meet the preset time requirements are selected as the target reference video segments corresponding to the current segment, where the starting position or ending position of each segment should be S or E, and the ending position of the preceding segment and the starting position of the succeeding segment of two adjacent segments are the same. In view of this situation, the determination of the target reference video segment can be divided into two stages: Stage 1), selecting the reference video segment as For ([E,S], [S,E]) fragment; stage 2), select the fragment and snippets When certain conditions are met, different reference video segments can be generated by changing the order of generating segments in stage 1) and stage 2).

[0116] Step 26: insert the target reference video segment corresponding to the current segment into the end of the determined target reference video segment.

[0117] Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

[0118] If there are subsequent segments of the current segment in the multiple driving speech segments, steps 21 to 26 are repeatedly performed until corresponding target reference video segments are determined for all subsequent segments.

[0119] According to the timestamp of the target reference video segment corresponding to each driving voice segment, the target reference video segment corresponding to each driving voice segment is obtained, and the corresponding video frame number is obtained according to the timestamp, and the video frame number is recorded as index, index = time_stamp*frame_rate / 1000, where time_stamp represents the timestamp and frame_rate represents the frame rate of the target reference video segment. According to the video frame number, the frame image is obtained and saved as a video file.

[0120] Step S104: using a plurality of driving voice segments to drive and lip-modify the corresponding target reference video segments, and connecting them to form a target video.

[0121] For the target reference video segment determined in step S103, the corresponding driving voice segment is used for driving and lip shape correction, and connected to form a complete target video. Specifically, the target video is generated through the steps of lip shape correction of the sound segment, silent segment processing and video synthesis.

[0122] 1) Lip shape correction of voice clips: For voice clips, the segmented driving voice clips and their corresponding target reference video clips are simultaneously input into the lip shape correction network, which corrects the lip shape of the virtual anchor in the target reference video clip according to the driving voice clip to obtain the corrected target reference video clip. Among them, the lip shape correction network can use common models such as Wav2Lip.

[0123] 2) Silent segment processing: add a voice channel to the target reference video segment which is a silent segment. For example, using the program library ffmpeg, this can be achieved through the following command: ffmpeg -i ref_x.mp4 -f lavfi-ianullsrc -map 0:v-map 1:a-shortest -y target_x.mp4.

[0124] 3) Video synthesis: The target reference video clips obtained in the above steps are sequentially connected in order from front to back to form a complete target video. For example, using the program library ffmpeg, the connection can be completed through the -f concat option.

[0125] Through the embodiments of the present application, a target video of any length with coherent virtual anchor movements can be generated based on a short-duration reference video, which effectively improves the coherence and realism of the virtual anchor's movements in scenes such as broadcasting and dialogue, and solves the problem of decreased lip synchronization in the generation of long-duration videos in the prior art.

[0126] According to the video generation method based on reference video provided in the embodiment of the present application, the reference video and the driving data are processed into sound segments and silent segments according to the content. Compared with the prior art method of generating a video by inputting the reference video as a whole, the present solution has a shorter single processing time, which can effectively avoid the lip-sound asynchrony problem caused by generating a long video. The duration, emotion type, whether it is a silent segment, etc. are comprehensively considered, and the corresponding target reference video segment is determined for each driving voice segment according to segment optimization. In addition, the driving data is processed into sound segments and silent segments. On the one hand, it reduces unnecessary video driving of silent segments and improves the video generation efficiency. On the other hand, it provides the possibility of selecting reference video segments, and can select reference lip opening and closing for sound segments. Video clips, selecting reference video clips with closed lips for silent clips, and maintaining the emotional similarity between the driving voice clip and the reference video clip, narrowing the visual difference between the reference video and the target video, thereby improving the visual effect of voice-driven, and because the initial drive of the silent clip is cancelled, the lip shaking phenomenon of the silent clip caused by the defects of the prior art is avoided; this scheme can generate a target video of any length and with coherent virtual host movements based on a short-duration reference video, effectively improving the realism and movement coherence of the virtual host voice-driven video generation based on the reference video, and at the same time, the segmented generation method is used to divide the generation task into multiple subtasks, and the dependencies between the subtasks are eliminated through reference frame selection, so that the efficiency of video generation can be improved through concurrent execution.

[0127] Figure 7 FIG. 4 shows a structural block diagram of a video generation device based on a reference video according to an embodiment of the present application. Figure 7 As shown, the device includes: a first processing module 710 , a second processing module 720 , a determination module 730 and a generation module 740 .

[0128] The first processing module 710 is adapted to: segment the reference video into a plurality of reference video segments, and determine the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments.

[0129] The second processing module 720 is adapted to: acquire driving data, process the driving data into a plurality of driving speech segments, and determine the emotion types of the plurality of driving speech segments.

[0130] The determination module 730 is adapted to determine a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments according to the duration, the emotion type, and whether it is a silent segment.

[0131] The generation module 740 is adapted to: utilize a plurality of driving speech segments to drive and lip-modify the corresponding target reference video segments, and connect them to form a target video.

[0132] Optionally, the first processing module 710 is further adapted to: extract voice channel data and visual channel data from a reference video; detect silent parts in the voice channel data, and perform preliminary segmentation based on the silent parts to obtain preliminary segmented sound segments and silent segments; refine the segmentation boundaries based on the visual channel data corresponding to the preliminary segmented silent segments to obtain finely segmented segmentation boundaries; correct the segmentation boundaries of the preliminary segmented sound segments and silent segments according to the finely segmented segmentation boundaries to obtain final sound segments and silent segments; use a micro-expression recognition method to identify the emotion types of the sound segments and silent segments; wherein, if a single reference video segment has multiple emotion types, the reference video segment is divided again so that each reference video segment after division has a single emotion type.

[0133] Optionally, the second processing module 720 is further adapted to: if the driving data is audio, detecting the silent part in the audio, dividing the audio into multiple driving voice segments based on the silent part, and determining the emotion types of the multiple driving voice segments through an audio sentiment analysis model; wherein the multiple driving voice segments include voiced segments and silent segments; if the driving data is text, dividing the text into multiple sub-texts and determining the emotion types of the multiple sub-texts through a text sentiment analysis model, converting the multiple sub-texts into multiple driving voice segments, and setting the emotion types of the multiple driving voice segments correspondingly according to the emotion types of the multiple sub-texts.

[0134] Optionally, the determination module 730 is further adapted to: select a first segment from multiple driving voice segments, and select a target reference video segment corresponding to the first segment from multiple reference video segments based on the duration, emotion type and whether it is a silent segment of the first segment; wherein the end position of the target reference video segment selected for the first segment is located on the boundary of the divided reference video segments; if the first segment is not the first segment among the multiple driving voice segments, the target reference video segment selected for it is also not the first segment among the multiple reference video segments; if there is a preceding segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and the corresponding target reference video segment is determined from the multiple reference video segments in turn for each preceding segment based on the duration, emotion type, whether it is a silent segment and the starting position of the target reference video segment corresponding to the current segment; if there is a subsequent segment of the first segment among the multiple driving voice segments, the first segment is used as the current segment, and the corresponding target reference video segment is determined from the multiple reference video segments in turn for each subsequent segment based on the duration, emotion type, whether it is a silent segment and the end position of the target reference video segment corresponding to the current segment.

[0135] Optionally, the determination module 730 is further adapted to:

[0136] Step 11, taking the starting position of the target reference video segment corresponding to the current segment as the preceding processing anchor point, selecting a segment adjacent to the current segment from the preceding segment, and updating the adjacent segment to the current segment;

[0137] Step 12, according to the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the previous processing anchor point from multiple reference video segments as an intermediate reference video segment, and set the offset direction variable to -1 or 1 according to whether the end position or the start position is the same as the previous processing anchor point;

[0138] Step 13, calculating a round-down result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating the duration difference according to the duration of the current segment, the duration of the intermediate reference video segment and the round-down result;

[0139] Step 14, if the result of rounding down is an odd number and the offset direction variable is 1, the end position of the middle reference video segment is used as the first start position, and the start position of the middle reference video segment is used as the first end position;

[0140] If the result of rounding down is an odd number and the offset direction variable is -1, the starting position of the middle reference video segment is used as the first starting position, and the ending position of the middle reference video segment is used as the first ending position;

[0141] If the result of rounding down is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the first starting position, and the starting position of the middle reference video segment is used as the first ending position;

[0142] If the result of rounding down is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the first starting position, and the end position of the middle reference video segment is used as the first ending position;

[0143] Step 15, selecting at least one segment with variable direction but continuous and meeting preset duration requirements from the intermediate reference video segments according to the duration difference, the first starting position and the first ending position as the target reference video segment corresponding to the current segment;

[0144] Step 16, inserting the target reference video segment corresponding to the current segment into the head of the determined target reference video segment;

[0145] Step 17: If there is a preceding segment of the current segment in the multiple driving voice segments, jump to step 11.

[0146] Optionally, the determination module 730 is further adapted to:

[0147] Step 21, taking the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, selecting an adjacent segment to the current segment from the subsequent segments, and updating the adjacent segment to the current segment;

[0148] Step 22, according to the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the subsequent processing anchor point from multiple reference video segments as an intermediate reference video segment, and set the offset direction variable to -1 or 1 according to whether the end position or the start position is the same as the subsequent processing anchor point;

[0149] Step 23, calculating a round-down result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating the duration difference according to the duration of the current segment, the duration of the intermediate reference video segment and the round-down result;

[0150] Step 24, if the result of rounding down is an odd number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the ending position of the middle reference video segment is used as the second ending position;

[0151] If the result of rounding down is an odd number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second start position, and the start position of the middle reference video segment is used as the second end position;

[0152] If the rounding result is an even number and the offset direction variable is 1, the starting position of the middle reference video segment is used as the second starting position, and the starting position of the middle reference video segment is used as the second ending position;

[0153] If the result of rounding down is an even number and the offset direction variable is -1, the end position of the middle reference video segment is used as the second starting position, and the end position of the middle reference video segment is used as the second end position;

[0154] Step 25, selecting at least one segment with variable direction but continuous and meeting the preset duration requirement from the intermediate reference video segments according to the duration difference, the second starting position and the second ending position as the target reference video segment corresponding to the current segment;

[0155] Step 26, inserting the target reference video segment corresponding to the current segment into the end of the determined target reference video segment;

[0156] Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

[0157] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0158] According to the video generation device based on reference video provided by the embodiment of the present application, the reference video and the driving data are processed into sound segments and silent segments according to the content. Compared with the prior art method of generating a video by inputting the reference video as a whole, the single processing time of this scheme is shorter, so the lip-sound asynchrony problem caused by generating a long video can be effectively avoided; the duration, emotion type, whether it is a silent segment, etc. are comprehensively considered, and the corresponding target reference video segment is determined for each driving voice segment according to segment optimization; and the driving data is processed into sound segments and silent segments. On the one hand, it reduces the video driving of unnecessary silent segments and improves the video generation efficiency. On the other hand, it provides the possibility of selecting reference video segments, and can select reference lip opening and closing for sound segments. Video clips, selecting reference video clips with closed lips for silent clips, and maintaining the emotional similarity between the driving voice clip and the reference video clip, narrowing the visual difference between the reference video and the target video, thereby improving the visual effect of voice-driven, and because the initial drive of the silent clip is cancelled, the lip shaking phenomenon of the silent clip caused by the defects of the prior art is avoided; this scheme can generate a target video of any length and with coherent virtual host movements based on a short-duration reference video, effectively improving the realism and movement coherence of the virtual host voice-driven video generation based on the reference video, and at the same time, the segmented generation method is used to divide the generation task into multiple subtasks, and the dependencies between the subtasks are eliminated through reference frame selection, so that the efficiency of video generation can be improved through concurrent execution.

[0159] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which enables a processor to perform operations corresponding to the video generation method based on a reference video in any of the above-mentioned method embodiments.

[0160] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the video generation method based on a reference video in any of the above method embodiments.

[0161] Figure 8 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0162] like Figure 8 As shown, the computing device may include: a processor (processor) 802 , a communication interface (Communications Interface) 804 , a memory (memory) 806 , and a communication bus 808 .

[0163] The processor 802, the communication interface 804, and the memory 806 communicate with each other via a communication bus 808. The communication interface 804 is used to communicate with other devices such as a client or other server network elements. The processor 802 is used to execute a program 810, which can specifically execute the relevant steps in the above-mentioned video generation method based on a reference video for a computing device.

[0164] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0165] The processor 802 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0166] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0167] The program 810 can be specifically used to enable the processor 802 to execute the video generation method based on the reference video in any of the above method embodiments. The specific implementation of each step in the program 810 can refer to the corresponding descriptions in the corresponding steps and units in the above video generation embodiment based on the reference video, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiments, which will not be repeated here.

[0168] The algorithm and display provided herein are not inherently related to any specific computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application embodiment described herein, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application embodiment.

[0169] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0170] Similarly, it should be understood that in order to streamline the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present application require more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the embodiment of the present application.

[0171] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0172] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the embodiments of the present application and to form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0173] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0174] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A video generation method based on a reference video, characterized in that: include: Segmenting the reference video into a plurality of reference video segments, and determining the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments; Acquiring driving data, processing the driving data into a plurality of driving speech segments, and determining emotion types of the plurality of driving speech segments; Determine a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments according to the duration, the emotion type, and whether it is a silent segment; A plurality of driving speech segments are used to drive and lip-modify the corresponding target reference video segments, and then connected to form a target video.

2. The method according to claim 1, characterized in that The step of dividing the reference video into a plurality of reference video segments and determining the emotion types of the plurality of reference video segments further comprises: Extracting speech channel data and visual channel data from the reference video; Detecting a silent part in the voice channel data, and performing preliminary segmentation according to the silent part to obtain preliminary segmented voiced segments and silent segments; According to the visual channel data corresponding to the initially segmented silent segment, the segmentation boundary is refined to obtain a finely segmented segmentation boundary; The segmentation boundaries of the initially segmented voiced segments and the silent segments are modified according to the segmentation boundaries of the fine segmentation to obtain the final voiced segments and the silent segments; The emotion types of the sound segment and the silent segment are identified by using a micro-expression recognition method; wherein, if a single reference video segment has multiple emotion types, the reference video segment is divided again so that each of the divided reference video segments has a single emotion type.

3. The method according to claim 1, characterized in that The step of processing the driving data into a plurality of driving speech segments and determining the emotion types of the plurality of driving speech segments further comprises: If the driving data is audio, a silent part in the audio is detected, the audio is segmented into a plurality of driving voice segments according to the silent part, and the emotion types of the plurality of driving voice segments are determined by an audio emotion analysis model; wherein the plurality of driving voice segments include a vocal segment and a silent segment; If the driving data is text, the text is divided into multiple sub-texts and the emotion types of the multiple sub-texts are determined through a text sentiment analysis model, the multiple sub-texts are converted into multiple driving voice segments, and the emotion types of the multiple driving voice segments are set accordingly according to the emotion types of the multiple sub-texts.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment further includes: A first segment is selected from a plurality of driving speech segments, and a target reference video segment corresponding to the first segment is selected from a plurality of reference video segments according to the duration, emotion type, and whether the first segment is a silent segment; wherein the end position of the target reference video segment selected for the first segment is located on the boundary of the divided reference video segments; if the first segment is not the first segment among the plurality of driving speech segments, the target reference video segment selected for the first segment is also not the first segment among the plurality of reference video segments; If there is a preceding segment of the first segment in the multiple driving voice segments, the first segment is used as the current segment, and a corresponding target reference video segment is determined for each preceding segment from the multiple reference video segments in turn according to the duration, the emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment; If there is a subsequent segment of the first segment among multiple driving voice segments, the first segment is taken as the current segment, and the corresponding target reference video segment is determined for each subsequent segment from multiple reference video segments in turn based on the length, emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment.

5. The method according to claim 4, characterized in that The step of determining a corresponding target reference video segment for each preceding segment from a plurality of reference video segments in sequence according to the duration, the emotion type, whether it is a silent segment, and the starting position of the target reference video segment corresponding to the current segment further comprises: Step 11, taking the starting position of the target reference video segment corresponding to the current segment as a preceding processing anchor point, selecting a segment adjacent to the current segment from the preceding segment, and updating the adjacent segment to the current segment; Step 12, according to the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the preceding processing anchor point from multiple reference video segments as an intermediate reference video segment, and according to whether the end position or the start position is the same as the preceding processing anchor point, set the offset direction variable to -1 or 1; Step 13, calculating a round-down result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment and the round-down result; Step 14: if the rounding result is an odd number and the offset direction variable is 1, the end position of the intermediate reference video segment is used as the first starting position, and the start position of the intermediate reference video segment is used as the first ending position; If the rounding result is an odd number and the offset direction variable is -1, the starting position of the intermediate reference video segment is used as the first starting position, and the ending position of the intermediate reference video segment is used as the first ending position; If the rounding result is an even number and the offset direction variable is 1, the starting position of the intermediate reference video segment is used as the first starting position, and the starting position of the intermediate reference video segment is used as the first ending position; If the rounding-down result is an even number and the offset direction variable is -1, the end position of the intermediate reference video segment is used as the first starting position, and the end position of the intermediate reference video segment is used as the first end position; Step 15, selecting at least one segment with variable direction but continuous and meeting preset duration requirements from the intermediate reference video segments according to the duration difference, the first starting position and the first ending position as a target reference video segment corresponding to the current segment; Step 16, inserting the target reference video segment corresponding to the current segment into the head of the determined target reference video segment; Step 17: If there is a preceding segment of the current segment in the multiple driving voice segments, jump to step 11.

6. The method according to claim 4, characterized in that The step of determining a corresponding target reference video segment for each subsequent segment from a plurality of reference video segments in sequence according to the duration, the emotion type, whether it is a silent segment, and the end position of the target reference video segment corresponding to the current segment further comprises: Step 21, taking the end position of the target reference video segment corresponding to the current segment as a subsequent processing anchor point, selecting an adjacent segment to the current segment from the subsequent segments, and updating the adjacent segment to the current segment; Step 22, according to the emotion type and whether it is a silent segment, query a reference video segment whose end position or start position is the same as the post-processing anchor point from multiple reference video segments as an intermediate reference video segment, and set the offset direction variable to -1 or 1 according to whether the end position or the start position is the same as the post-processing anchor point; Step 23, calculating a round-down result of the ratio of the duration of the current segment to the duration of the intermediate reference video segment, and calculating a duration difference based on the duration of the current segment, the duration of the intermediate reference video segment and the round-down result; Step 24: if the rounding result is an odd number and the offset direction variable is 1, the starting position of the intermediate reference video segment is used as the second starting position, and the ending position of the intermediate reference video segment is used as the second ending position; If the rounding-down result is an odd number and the offset direction variable is -1, the end position of the intermediate reference video segment is used as the second start position, and the start position of the intermediate reference video segment is used as the second end position; If the rounding-down result is an even number and the offset direction variable is 1, the starting position of the intermediate reference video segment is used as the second starting position, and the starting position of the intermediate reference video segment is used as the second ending position; If the rounding result is an even number and the offset direction variable is -1, the end position of the intermediate reference video segment is used as the second starting position, and the end position of the intermediate reference video segment is used as the second end position; Step 25, selecting at least one segment with variable direction but continuous and meeting preset duration requirements from the intermediate reference video segments according to the duration difference, the second starting position and the second ending position as a target reference video segment corresponding to the current segment; Step 26, inserting the target reference video segment corresponding to the current segment into the end of the determined target reference video segment; Step 27: If there is a subsequent segment of the current segment in the multiple driving voice segments, jump to step 21.

7. A video generation device based on a reference video, characterized in that: include: The first processing module is adapted to divide the reference video into a plurality of reference video segments and determine the emotion types of the plurality of reference video segments; wherein the plurality of reference video segments include sound segments and silent segments; a second processing module adapted to acquire driving data, process the driving data into a plurality of driving speech segments, and determine emotion types of the plurality of driving speech segments; A determination module, adapted to determine a corresponding target reference video segment for each driving voice segment from a plurality of reference video segments based on duration, emotion type, and whether it is a silent segment; The generation module is suitable for using multiple driving voice segments to drive and lip-modify the corresponding target reference video segments, and connect them to form a target video.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video generation method based on the reference video as described in any one of claims 1-6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the video generation method based on a reference video as described in any one of claims 1 to 6.

10. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the method for generating a video based on a reference video according to any one of claims 1 to 6.

Images