Video frame insertion method, video generation method, storage medium and electronic equipment

Abstract

The invention provides a video frame insertion method, a video generation method, a storage medium and electronic equipment, and is applied to the technical field of artificial intelligence. According to the invention, the modulation residual error is generated between the initial motion potential vector and the termination motion potential vector based on the time sequence self-attention and multi-scale convolution interpolation network, and the modulation residual error and the linear interpolation vector are added to form the continuous and smooth intermediate motion potential vector; the seamless connection of the video motion potential vector sequences driven by multiple segments of audios is realized, so that the problems of unnatural frame insertion, transition jump and insufficient real-time performance when the digital human video is switched among different states are effectively solved, and the continuity and real-time interaction experience of video generation are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video frame interpolation method, a video generation method, a storage medium, and an electronic device. Background Technology

[0002] With the development of generative artificial intelligence technology, digital human video dialogue systems based on single-image human models are widely used in scenarios such as online customer service, virtual anchors, and real-time assistants. These systems generate digital human videos by inputting reference images and audio, enabling real-time interaction.

[0003] However, existing technologies have several shortcomings: First, they rely on external frame interpolation models that are not specifically optimized, resulting in slow transition speeds and unstable effects, which affect real-time performance; second, they use image-level caching, which requires re-encoding each time a new video is generated, resulting in high latency of the first frame and reduced response speed; third, they generally use a serial generation process, which requires generating all video frames before transitioning, increasing the latency of the first frame viewing on the user's end; and fourth, when switching between Idle and Talk states, there is a lack of natural transition animations, and the direct jump causes visual jumps, affecting the continuity and realism of the video.

[0004] Therefore, how to optimize frame interpolation technology to improve the smoothness and interactive experience of digital human videos has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a video frame interpolation method, a video generation method, a storage medium, and an electronic device to overcome or at least partially solve the above problems. The technical solution is as follows:

[0006] A video frame interpolation method includes:

[0007] Obtain the initial motion potential vector and the final motion potential vector;

[0008] Linear interpolation is performed on the initial motion potential vector and the final motion potential vector to obtain a linear interpolation vector corresponding to at least one interpolation frame time point;

[0009] A conditional feature sequence is generated using the initial motion potential vector, the final motion potential vector, and the temporal embedding vectors at each interpolated time point;

[0010] The conditional feature sequence is input into the interpolation network, so that the interpolation network processes the conditional feature sequence based on temporal self-attention and multi-scale one-dimensional convolution, and outputs the residual vector corresponding to each interpolation frame time point;

[0011] The linear interpolation vector corresponding to each interpolation time point is added to the corresponding modulated residual vector to generate at least one intermediate motion potential vector located between the starting motion potential vector and the ending motion potential vector.

[0012] Optionally, generating a conditional feature sequence using the initial motion latent vector, the final motion latent vector, and the temporal embedding vectors at each interpolated time point includes:

[0013] Determine at least one interpolation time point within the specified interpolation interval;

[0014] The temporal embedding vector corresponding to each interpolated frame time point is obtained through a positional encoding function;

[0015] The starting motion potential vector, the ending motion potential vector, and the temporal embedding vector of each interpolated time point are concatenated to obtain the conditional features of each interpolated time point.

[0016] The condition feature sequence is obtained by utilizing the condition features of all interpolated time points.

[0017] Optionally, before adding the linear interpolation vector corresponding to each interpolation time point to the corresponding modulated residual vector, the method further includes:

[0018] The residual vector corresponding to each interpolation time point is multiplied by the value of a boundary constraint shape function at that interpolation time point to obtain the modulated residual vector, wherein the boundary constraint shape function has a value of zero at the start and end of the interpolation interval.

[0019] Optionally, the boundary constraint shape function is: ,in, This refers to the frame insertion time point.

[0020] Optionally, the training process of the interpolation network includes:

[0021] Obtain a continuous motion latent vector sequence generated by the FLOAT model;

[0022] From the continuous motion potential vector sequence, the motion potential vector corresponding to the starting frame is selected as the starting motion potential vector sample, and the motion potential vector corresponding to the ending frame is selected as the ending motion potential vector sample.

[0023] Select the motion potential vector of at least one intermediate frame located between the start frame and the end frame from the continuous motion potential vector sequence as a supervision sample;

[0024] Using the initial motion latent vector sample and the final motion latent vector sample as input, and the supervision sample as the ground truth label, the network containing the temporal self-attention module and the multi-scale one-dimensional convolution module is trained to obtain the trained interpolation network.

[0025] Optionally, when training the interpolation network, the loss function used includes at least one of reconstruction loss, first-order smoothing loss, second-order smoothing loss, and residual energy constraint loss.

[0026] A video generation method, comprising:

[0027] Obtain a prototype reference image of the character and at least two audio inputs;

[0028] An identity code representing identity features is generated based on the aforementioned prototype reference image;

[0029] Based on the identity encoding and the first audio segment, a first motion latent vector sequence corresponding to the first audio segment is generated using the FLOAT model;

[0030] Based on the identity encoding and the second audio segment, a second motion latent vector sequence corresponding to the second audio segment is generated using the FLOAT model;

[0031] According to the video frame interpolation method, the last frame motion potential vector of the first motion potential vector sequence is used as the starting motion potential vector, and the first frame motion potential vector of the second motion potential vector sequence is used as the ending motion potential vector. Interpolation is performed to generate a transition motion potential vector sequence, wherein the transition motion potential vector sequence contains multiple intermediate motion potential vectors.

[0032] The transitional motion potential vector sequence is inserted between the first motion potential vector sequence and the second motion potential vector sequence to obtain a continuous target motion potential vector sequence;

[0033] Based on the target motion potential vector sequence and the identity encoding, a continuous video stream is synthesized.

[0034] Optionally, after obtaining the prototype reference image and at least two audio inputs, the method further includes:

[0035] Based on the prototype reference image of the character and the blank audio, a sequence of motion potential vectors in the idle state is generated using the FLOAT model;

[0036] The step of generating a first motion latent vector sequence corresponding to the first audio segment using the FLOAT model based on the identity encoding and the first audio segment includes:

[0037] When the first audio segment is a streaming audio segment, when the first audio segment is received, the interpolation start point is determined based on the motion potential vector of the last frame of the motion potential vector sequence in the idle state.

[0038] According to the video frame interpolation method, the motion latent vector corresponding to the interpolation start point is used as the starting motion latent vector, and the motion latent vector of the first frame generated based on the first audio segment is used as the ending motion latent vector. Interpolation is performed to generate a transition motion latent vector sequence from the idle state to the dialogue state.

[0039] Based on the first audio segment, generate the remaining motion potential vectors excluding the first frame, and concatenate them with the transition motion potential vector sequence to generate the first motion potential vector sequence.

[0040] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the video frame interpolation method and / or the video generation method.

[0041] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the video frame interpolation method and / or the video generation method.

[0042] By employing the above technical solutions, the present invention provides a video frame interpolation method, a video generation method, a storage medium, and an electronic device. By generating a modulation residual between the initial motion latent vector and the final motion latent vector using an interpolation network based on temporal self-attention and multi-scale convolution, and adding it to the linear interpolation vector to form a continuous and smooth intermediate motion latent vector, the present invention achieves seamless connection of video motion latent vector sequences driven by multiple audio segments. This effectively solves the problems of unnatural frame interpolation, transition jumps, and insufficient real-time performance when digital human videos switch between different states, and improves the coherence of video generation and the real-time interactive experience.

[0043] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0045] Figure 1 A flowchart illustrating one embodiment of the video frame interpolation method provided by this invention is shown.

[0046] Figure 2 The diagram shows a specific implementation of step S120 in the video frame interpolation method provided by the present invention.

[0047] Figure 3 A flowchart illustrating the training process of the interpolation network provided in an embodiment of the present invention is shown.

[0048] Figure 4 A flowchart illustrating one embodiment of the video generation method provided by this invention is shown.

[0049] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0050] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0051] With the rapid development of generative artificial intelligence technology, digital human video dialogue systems driven by single-image human figures have been widely used in various application scenarios such as virtual anchors, online customer service, and real-time assistants. These systems generate corresponding digital human videos by inputting reference images and audio provided by the user, enabling real-time dialogue and interaction between the virtual character.

[0052] However, existing technologies still have many problems in practical applications, affecting the smoothness, real-time performance, and user experience of digital human videos. Specifically: First, existing solutions generally rely on external frame interpolation models to achieve a smooth transition between idle and talk video frames. These external frame interpolation modules are often not specifically optimized for the characteristics of digital human videos, resulting in slow generation speeds and unstable transition effects, making it difficult to meet the needs of real-time interaction and reducing system response efficiency and user experience. Second, traditional methods typically employ image-level caching technology, i.e., caching the reference image itself. Each time a new video is generated, the reference image needs to be re-encoded and processed, resulting in a significant delay in the generation of the first frame, typically exceeding one second. This high first-frame delay severely impacts the real-time responsiveness of the digital human system. Third, most existing systems use a serial generation process, first generating all video frames completely, then performing transition processing, and finally streaming the output. This serial processing method significantly increases the latency for the user to see the first frame, reducing the system's real-time feel and interactive smoothness. Furthermore, when the digital human switches between different animation states, especially from Idle to Talk, it often jumps directly from the last frame of the Idle video to the first frame of the Talk video, lacking a natural transition animation. This creates a noticeable visual jump, affecting the video's coherence and realism.

[0053] Based on this, the video frame interpolation method, video generation method, storage medium, and electronic device provided in the embodiments of the present invention can obtain the latent representation suitable for frame interpolation in advance to support efficient subsequent frame interpolation operations, and achieve fast and stable frame interpolation processing at the latent level, fundamentally improving the naturalness and coherence of video transitions. At the same time, the latency of first frame generation is significantly reduced through latent caching, improving system response speed, realizing natural and imperceptible inter-frame transitions between different animation states of digital humans, and improving the user's visual experience.

[0054] like Figure 1 The diagram shows a flowchart of one embodiment of the video frame interpolation method provided by this invention. The method may include:

[0055] S100, obtain the initial motion potential vector and the final motion potential vector.

[0056] The initial motion latent vector is the latent representation of the motion state at the beginning of the interpolation interval. It is a video motion latent variable generated by the FLOAT model at a certain time point; that is, in the video sequence that needs frame interpolation, it is the motion latent vector corresponding to the beginning of the transition sequence. The initial motion latent vector can be generated from the motion encoding of the last frame of the previous state (such as the Idle state or the Talk state), defining the initial motion state of the interpolation sequence. For example, in the FLOAT model, the initial motion latent vector can be the motion latent of the last frame of the Idle state sequence, or the motion latent of the last frame of Talk A, denoted as . .

[0057] Among them, the Float model can be the FMT (Flow Matching Transformer) model. The FMT model is a generative model that combines continuous probability flow (Flow Matching) and the Transformer architecture. The FMT model completes the forward inference of the prototype reference image of the person.

[0058] The terminating motion latent vector is the latent representation of the motion state at the end of the interpolation interval. It corresponds to the video motion latent variable at the end of the interpolation in the FLOAT model, specifically the motion latent vector at the end of the transition sequence within the video sequence requiring interpolation. The terminating motion latent vector can be generated from the motion encoding of the first frame of the next state (such as the Talk or Idle state), defining the ending motion state of the interpolation sequence. For example, in the FLOAT model, the terminating motion latent vector could be the motion latent of the first frame of the upcoming Talk state or the motion latent of the first frame of the next Idle state, denoted as... .

[0059] Specifically, in this embodiment of the invention, the two endpoint vectors of the interpolation interval can be selected as the starting vectors from the continuous motion latent sequence generated by the FLOAT model. and termination vector These two vectors correspond to the motion state latent variables at the start and end points of the interpolation in the video, respectively, ensuring that the boundaries of the interpolation are accurate and derived from real motion coding.

[0060] S110. Perform linear interpolation on the initial motion potential vector and the final motion potential vector to obtain a linear interpolation vector corresponding to at least one interpolation frame time point.

[0061] The interpolation time point is a time parameter located between the start and end times in the interpolation operation. It is typically normalized to between 0 and 1, indicating the specific time position where intermediate frames need to be generated. For example, 0 represents the start point, and 1 represents the end point. To insert 10 frames between the start and end points, the interpolation time point could be... .

[0062] The linear interpolation vector is a preliminary intermediate latent vector obtained by linear interpolating the initial and final motion latent vectors at the interpolation time points. It is used to construct the basic trajectory of the interpolation result. The formula for calculating the linear interpolation vector is as follows: .

[0063] Specifically, embodiments of the present invention can be based on preset frame interpolation time points. Using linear interpolation formula Calculate the initial potential motion vector at that time point, i.e., the linear interpolation vector.

[0064] S120. Using the initial motion potential vector, the final motion potential vector, and the temporal embedding vectors of each interpolated frame time point, generate a conditional feature sequence.

[0065] The temporal embedding vector is a vector representation that encodes the interpolated frame time points. Sine wave position coding can be used to transmit time information to the interpolation network. The temporal embedding network is denoted as... As one of the input conditions of the interpolation network, it helps the interpolation network distinguish the output at different time points.

[0066] Here, the conditional feature sequence refers to the feature sequence formed by sequentially concatenating the initial motion latent vector, the final motion latent vector, and the temporal embedding vector at the corresponding interpolation time point, which serves as the input condition for the interpolation network. For example, for each interpolation time point... Its conditional characteristics All Stacking yields the conditional feature sequence. .

[0067] Specifically, embodiments of the present invention can perform interpolation at each frame time point. Temporal embedding vectors are generated using sinusoidal positional coding or other coding methods. Then , and The conditions and characteristics for splicing together All Conditional feature sequences are formed by stacking time series data. As the input to the interpolation network, it carries start and end action information and time context.

[0068] S130. Input the conditional feature sequence into the interpolation network so that the interpolation network processes the conditional feature sequence based on temporal self-attention and multi-scale one-dimensional convolution, and outputs the residual vector corresponding to each interpolation frame time point.

[0069] The interpolation network is a neural network structure used to generate residual vectors based on conditional feature sequences, improving linear interpolation trajectories and achieving more natural motion latent vector interpolation. The interpolation network provided in this embodiment takes the conditional feature sequence as input and performs parallel processing through structures such as temporal self-attention and multi-scale one-dimensional convolution, outputting a series of residual vectors. The interpolation network employs a parallel forward design to avoid recursion and achieve high-speed frame interpolation.

[0070] Temporal self-attention is a submodule in the interpolation network that uses a self-attention mechanism to perform global temporal modeling of the input sequence to capture dynamic relationships and contextual information between different time points. This invention can perform self-attention calculation on the conditional feature sequence in the temporal dimension to model the global dependencies between all interpolated frame time points, ensuring that the generated intermediate frames are temporally coherent and consistent, resulting in a globally smooth transition trajectory.

[0071] Multi-scale one-dimensional convolution is a sub-module in the interpolation network. The convolution operation uses different kernel sizes in the time dimension to capture dynamic features at different time scales, thus aiding in temporal modeling. The multi-scale one-dimensional convolution provided in this embodiment can use multiple one-dimensional convolution kernels of different sizes (such as 1, 3, and 5) to perform convolution operations on the feature sequence in the time dimension, thereby simultaneously capturing local motion patterns and dynamic changes at different time scales.

[0072] The residual vector is the compensation information output by the interpolation network, used to correct the linear interpolation vector, making the generated intermediate motion potential vector smoother and more natural. The interpolation network provided in this embodiment can predict a residual vector for each interpolation frame time point, representing the correction amount that needs to be superimposed on the linear interpolation vector at that time point, so that the final generated intermediate motion potential vector is closer to the real and complex motion trajectory.

[0073] Specifically, embodiments of the present invention can use conditional feature sequences The data is fed into an interpolation network, which first processes the conditional feature sequence through a temporal self-attention module. Perform global temporal modeling, capture the temporal dependencies during frame interpolation, and output a globally consistent temporal context representation. Subsequently Multi-scale temporal dynamic features are extracted by one-dimensional convolutional branches with different kernel sizes to obtain multi-scale representations. The multi-scale features are then fused (concatenated or summed), normalized, and mapped using a multilayer perceptron (MLP) to finally output a residual vector sequence. Compensation information corresponding to each interpolation time point .

[0074] S140. Add the linear interpolation vector corresponding to each interpolation time point to the corresponding modulated residual vector to generate at least one intermediate motion potential vector located between the starting motion potential vector and the ending motion potential vector.

[0075] The intermediate motion latent vector refers to the motion latent vector generated by adding the linear interpolation vector to the modulated residual vector. It lies between the initial and final motion latent vectors and represents the refined motion state at the interpolation time point. In this embodiment of the invention, a boundary constraint shape function can be used to modulate the residual vector, ensuring that the residual is zero at the start and end points. Intermediate motion latent vector The calculation formula can be:

[0076] ,

[0077] in, It is a linear interpolation vector; For boundary constraint shape function, This is the residual vector.

[0078] Specifically, embodiments of the present invention can utilize boundary constraint shape functions to modulate residual vectors. To avoid interpolation disrupting the consistency of start and end points, calculations are performed. This yields a smooth and natural intermediate motion potential vector, which serves as the frame interpolation result, enabling continuous transitions between action states.

[0079] This invention obtains the initial motion latent vector and the final motion latent vector, generates a preliminary linear interpolation vector using linear interpolation, and then constructs a conditional feature sequence input interpolation network by combining it with a temporal embedding vector. The interpolation network accurately captures temporal dependence and multi-scale dynamic features with the help of temporal self-attention and multi-scale one-dimensional convolution modules, and outputs a residual vector for correcting the interpolation. Finally, the linear interpolation vector is added to the modulation residual to generate a smooth and natural intermediate motion latent vector, thereby effectively ensuring the consistency between the interpolated frame sequence and the start and end frames, improving the coherence and stability of motion latent vector interpolation, and helping to achieve the beneficial effect of natural connection and smooth transition of multiple actions in digital human videos.

[0080] Optional, based on Figure 1 The method shown is as follows: Figure 2The diagram shows a specific implementation of step S120 in the video frame interpolation method provided by this invention. Step S120 may specifically include:

[0081] S200. Determine at least one interpolation time point within the specified interpolation interval.

[0082] Specifically, embodiments of the present invention can select one or more time points uniformly or non-uniformly according to the frame interpolation requirements and the frame interpolation interval [0,1]. These time points represent the locations where intermediate motion potential vectors need to be generated. For example, embodiments of the present invention can use an equal-interval sampling method to ensure smooth frame interpolation, and the sampling density can also be flexibly adjusted according to the frame interpolation length and transition requirements.

[0083] S210. Obtain the time embedding vector corresponding to each interpolated frame time point through the position encoding function.

[0084] Specifically, embodiments of the present invention can perform interpolation at each frame time point. Input to the position encoding function Such as sine-cosine coding or other continuous coding methods, to generate corresponding temporal embedding vectors. ,in, Represents the set of real numbers. The vector dimension is represented, thereby providing temporal context information for the interpolation network, helping the network understand the positional relationship of time points within the interpolation frame interval, and effectively enhancing the temporal modeling capability.

[0085] S220. Concatenate the starting motion potential vector, the ending motion potential vector, and the temporal embedding vector of each interpolated frame time point to obtain the conditional features of each interpolated frame time point.

[0086] Specifically, embodiments of the present invention can target each interpolation time point. The initial motion potential vector Termination motion potential vector With temporal embedding vector splicing into conditional features This conditional feature integrates the start and end motion states and time information, providing the interpolation network with complete and explicit interpolation semantic conditions.

[0087] S230. Obtain the condition feature sequence by utilizing the condition features of all interpolated time points.

[0088] Specifically, embodiments of the present invention can identify the conditional features corresponding to all interpolated time points. Stacked in chronological order to form a matrix This constitutes a conditional feature sequence. This conditional feature sequence serves as the input to the interpolation network, supporting parallel processing of all interpolation points. It enables the network to understand the temporal variation trend of the interpolation interval as a whole, providing temporal context and motion boundary constraints for subsequent residual prediction.

[0089] This invention combines the temporal embedding vectors of each interpolation time point, concatenating the start and end vectors with temporal information to generate conditional features. This clearly encodes motion boundaries and temporal positions, helping the interpolation network accurately capture the temporal features of motion changes. By determining the interpolation time points within a specified interpolation interval and using a positional encoding function to obtain the temporal embedding, not only is the expressive power of temporal information improved, but the network's ability to distinguish different time points is also enhanced. The resulting conditional feature sequence serves as the network input, fully integrating motion state and temporal context. This significantly improves the interpolation network's prediction accuracy and continuity for intermediate motion latent vectors, enhancing the naturalness, fluency, and temporal consistency of the interpolation effect.

[0090] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, before step S140, the method may further include:

[0091] The residual vector corresponding to each interpolation time point is multiplied by the value of a boundary constraint shape function at the interpolation time point to obtain the modulated residual vector, wherein the boundary constraint shape function has a value of zero at the start and end of the interpolation interval.

[0092] Specifically, to ensure that the interpolated frame sequence is strictly connected between the first and last frames, this embodiment of the invention provides a boundary constraint shape function. The function satisfies and That is, the values ​​at both ends of the interpolation interval [0,1] are zero, and the value in the middle is positive. The interpolation result satisfies This is to structurally prevent frame insertion from disrupting the consistency of start and end frames.

[0093] Optionally, the boundary constraint shape function is: ,in, This refers to the frame insertion time point.

[0094] Furthermore, embodiments of the present invention can use the residual vector obtained from network prediction. Shape function with boundary constraints Multiplying the values ​​at the corresponding time points yields the modulated residual vector. This allows the residual to automatically decay to zero at the start and end points, avoiding boundary jumps in the interpolation results and ensuring that the motion potential vector is completely consistent with the start and end vectors at the boundary.

[0095] Before adding the residual vector to the linear interpolation vector, this embodiment of the invention modulates the residual using a boundary constraint shape function to ensure that the residual naturally decays to zero at the start and end points of the interpolation interval. This strictly maintains consistency with the start and end motion potential vectors, effectively avoiding boundary jumps and discontinuities caused by residual superposition during the interpolation process. It improves the smoothness and coherence of the interpolation results, ensuring that the generated intermediate motion potential vectors have rich detail variations and achieve the beneficial effect of seamlessly connecting the start and end frames.

[0096] Optional, based on Figure 1 The method shown is as follows: Figure 3 The diagram shows a flowchart of the training process of the interpolation network provided in this embodiment of the invention. The training process of the interpolation network may include:

[0097] S300, Obtain the sequence of continuous motion potential vectors generated by the FLOAT model.

[0098] Specifically, in this embodiment of the invention, a continuous audio input and a reference photo can be processed by the FLOAT model to generate a corresponding motion latent sequence (i.e., continuous motion latent vector) frame by frame. The continuous motion latent vector sequence reflects the dynamic performance and lip movements of the character during the time period and records the continuous motion latent vector from the initial state to the final state, which serves as the basic data for training and inference of the frame interpolation module.

[0099] As examples, embodiments of the present invention can collect or generate a large number of diverse audio segments (such as audio containing speech, silence, and different speaking speeds) and corresponding reference images. These paired data are then batch-input into a pre-trained FLOAT model. The forward inference process of the FLOAT model is then run. Inside the model, the motion encoder generates a latent motion vector for each video frame based on the audio and identity encoding. The complete vector sequence generated for each sample is recorded. ,in, , The truth value in the middle From { Equal-interval sampling.

[0100] S310. Select the motion potential vector corresponding to the starting frame from the continuous motion potential vector sequence as the starting motion potential vector sample, and select the motion potential vector corresponding to the ending frame as the ending motion potential vector sample.

[0101] Specifically, in embodiments of the present invention, the latent position of the starting frame in a continuous motion latent vector sequence can be selected as the starting motion latent vector. The latent position of the terminating frame is used as the latent vector for terminating motion. These two points are the boundary conditions of the interpolation trajectory, ensuring that the interpolation network performs intermediate interpolation calculations based on these two states.

[0102] As examples, to increase data diversity and simulate transitions of different lengths, embodiments of the present invention may randomly select or fix a sequence length. ( This involves extracting a subsequence from the original long sequence. The selected sequence fragment... In the middle, the motion potential vector of the first frame is explicitly included. as potential vector samples of initial motion At the same time, the motion latent vector of the last frame is... As a potential vector sample for termination of motion This constitutes the input pair of a training sample. These represent the two endpoints of the interpolation interval.

[0103] S320. Select the motion potential vector of at least one intermediate frame located between the start frame and the end frame from the continuous motion potential vector sequence as a supervision sample.

[0104] Specifically, in this embodiment of the invention, one or more intermediate time points can be uniformly sampled between the start frame and the end frame, and the corresponding motion latent vector is used as the ground truth for supervision of the frame interpolation network. This is used to calculate the error between the interpolated prediction and the true intermediate state during training, which helps the network learn more accurate motion latent vector completion points.

[0105] As some examples, embodiments of the present invention can determine n interpolation time points within the interval (0,1). Based on these time points, in a continuous vector sequence Equal-interval sampling is performed (or mapped to the nearest frame based on the time point). For example, if the sequence length is L+1 frames, then the 1st frame... Interpolation points The corresponding real frame index is approximately The motion latent vector retrieved from this index is the true supervised sample corresponding to that time point. Perform this operation on all n interpolation points to obtain a set of true intermediate motion latent vectors. These serve as supervision labels for the training samples. They represent the results from... arrive A realistic and natural movement transition path.

[0106] S330. Using the initial motion latent vector samples and the final motion latent vector samples as input, and the supervision samples as ground truth labels, train the network containing the temporal self-attention module and the multi-scale one-dimensional convolution module to obtain the trained interpolation network.

[0107] Specifically, in this embodiment of the invention, the input condition features can be constructed as a concatenated sequence of starting latent vector, ending latent vector and temporal embedding, which is then input into an interpolation network consisting of a temporal self-attention module and a multi-scale one-dimensional convolution module. The interpolation network predicts the residuals of all interpolated frame time points in parallel. Through optimization using multiple loss functions such as reconstruction loss, smoothing regularization and residual energy constraints, the interpolation network can accurately compensate for the errors of linear interpolation, and finally, an interpolation network that can efficiently and coherently generate motion latent vectors is obtained.

[0108] As some examples, sample pairs The interpolation network is trained by inputting the interpolation time points. Following the inference flow S100-S140, the network captures global dependencies through a temporal self-attention module, then fuses local multi-scale features through a multi-scale one-dimensional convolution module, finally outputting the predicted residual sequence. According to the formula Synthetic prediction intermediate vector sequence Calculate the predicted sequence. With real supervision label The loss between the two is calculated using the backpropagation algorithm to compute the gradient of the loss with respect to the network parameters, and the network weights are updated using an optimizer (such as Adam). This process is iterated on a large number of samples until the network converges, ultimately resulting in a well-trained interpolation network that can predict smooth and natural intermediate motion sequences with high quality and speed based solely on the latent vectors of the first and last motions.

[0109] This invention utilizes a continuous sequence of motion latent vectors generated by the FLOAT model, selects the latent vectors corresponding to the start and end frames as boundary conditions for interpolation, and uses intermediate true latent vectors as supervision targets. This provides the interpolation network with accurate and continuous training samples. Combining a network structure with temporal self-attention and multi-scale one-dimensional convolution, it effectively captures the dynamic changes of motion latent vectors over time, enabling the interpolation network to learn to generate natural, smooth, and detailed intermediate motion latent vector sequences between the start and end latent vectors. This training process not only ensures the high quality and consistency of the interpolation results but also enhances the interpolation network's adaptability to complex dynamic changes. Ultimately, it achieves efficient, stable, and consistent intermediate frame motion latent vector prediction based on limited start and end information, significantly improving the natural transition effect of video frame interpolation.

[0110] Optionally, in the above Figure 3Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, the loss function used when training the interpolation network includes at least one of reconstruction loss, first-order smoothing loss, second-order smoothing loss and residual energy constraint loss.

[0111] The formula for calculating the reconstruction loss can be: .

[0112] The formula for calculating the first-order smoothing loss is as follows: .

[0113] The formula for calculating the second-order smoothing loss is as follows: .

[0114] The formula for calculating the residual energy constraint loss can be: .

[0115] Specifically, embodiments of the present invention can calculate the predicted sequences separately. With real supervision label The loss function includes reconstruction loss, first-order smoothing loss, second-order smoothing loss, and residual energy constraint loss, which are then summed to obtain the total loss:

[0116] ,

[0117] in, , and The loss weights are set as coefficients, and the gradient of the total loss with respect to the network parameters of the interpolation network is calculated using the backpropagation algorithm. The network weights are then updated using an optimizer, and the process is iterated until the network converges to obtain the trained interpolation network.

[0118] In training the interpolation network, this invention introduces multiple loss functions, including reconstruction loss, first-order smoothing loss, second-order smoothing loss, and residual energy constraint loss. These functions not only effectively promote the network's accurate reconstruction of intermediate motion latent vectors, ensuring that the interpolation results are highly consistent with the actual motion state, but also constrain the continuity and curvature smoothness of the interpolation trajectory through first-order and second-order smoothing terms, preventing abrupt changes and unnatural jumps during the interpolation process. At the same time, the residual energy constraint limits the residual amplitude, avoiding overfitting and abnormal oscillations. Overall, this improves the stability, smoothness, and generalization ability of the interpolation network, thereby achieving high-quality, coherent, and natural motion latent vector frame interpolation, significantly optimizing the visual smoothness and realism of connecting multiple video segments.

[0119] like Figure 4 The diagram shows a flowchart of one embodiment of the video generation method provided by this invention. The video generation method may include:

[0120] S400, obtain a prototype reference image of the character and at least two audio inputs.

[0121] In this context, the prototype reference image refers to a static image used to represent the identity and physical characteristics of a specific person. The prototype reference image serves as input to the video synthesis system, used to extract the person's identity features and ensure that the person's appearance in the generated video remains consistent with that image.

[0122] Specifically, in this embodiment of the invention, a user-provided prototype reference image of a person can be acquired to specify the identity and physical characteristics of the person in the video. Simultaneously, the user inputs at least two audio segments, which typically contain continuous or segmented speech information that the person needs to express, providing speech-driven data for subsequent video generation.

[0123] As examples, embodiments of the present invention can receive a clear, frontal or near-frontal photograph of a person uploaded by a user, or select one from a preset image library. This prototype reference image will serve as the sole basis for the visual identity of the person in all generated videos, including their appearance, hairstyle, and facial features. Simultaneously, embodiments of the present invention can receive multiple audio segments output by a Text-to-Speech (TTS) engine. These audio segments can correspond to multi-sentence responses streamed from a Large Language Model (LLM) dialogue system; for example, the first audio segment might be the first sentence of the response, and the second audio segment the second sentence. Together, they constitute the audio input sequence that drives the person to perform continuous speaking actions.

[0124] S410. Generate identity codes representing identity features based on prototype reference images of people.

[0125] Identity encoding refers to the high-dimensional vector obtained by encoding a prototype reference image of a person, representing the person's identity characteristics. Identity encoding can capture the facial structure, style, and feature details of a person, which is used to guide the generation of motion latent vectors and video synthesis, so that the generated content has the visual characteristics of a specific person.

[0126] Specifically, in this embodiment of the invention, a prototype reference image of a person can be input into an image encoder, and identity-related information in the image can be extracted through a deep feature extraction network (such as CNN or visual Transformer) to obtain a high-dimensional identity code. The identity code can effectively represent the facial features, structure, and style of a person, providing identity conditions for subsequent audio-driven generation of person motion.

[0127] As examples, embodiments of the present invention can input a prototype reference image of a person into a pre-trained identity encoder in the FLOAT model. The identity encoder performs deep feature extraction on the image and outputs one or more sets of high-dimensional vectors, such as: style encoding (sr), multi-scale identity features (sr_feats), and identity residual encoding (rs). These vectors together constitute a compact and comprehensive identity code that uniquely represents the visual identity of the person.

[0128] Optionally, embodiments of the present invention may store the identity code in memory or a cache. Throughout the entire video stream generation process, all subsequent steps will reuse this code to ensure that the appearance of the person in all generated video clips remains consistent.

[0129] S420. Based on the identity encoding and the first audio segment, the first motion potential vector sequence corresponding to the first audio segment is generated through the FLOAT model.

[0130] The first audio segment refers to the first speech or sound signal input related to the person, which is the basis for generating the first motion potential vector sequence.

[0131] Specifically, in this embodiment of the invention, the identity code and the first audio segment can be input into the FLOAT model respectively. The audio is converted into audio features by a feature extraction network (such as an audio encoder) and then fused with the identity code. The FLOAT model generates a sequence of latent motion vectors that match the content and rhythm of the first audio segment based on the fused features. The latent vector of each frame describes the dynamic state of the person at that moment.

[0132] As examples, embodiments of the present invention can input the first audio segment into the audio encoder of the FLOAT model to obtain audio-related motion codes, such as audio content coding (wa) and sentiment / prosodic coding (we). The audio codes (wa, we) are combined with the identity residual coding portion from the cached identity coding and input into the motion generator (i.e., motion latent decoder) of the FLOAT model. This motion generator then generates a sequence of motion latent vectors synchronized with the audio and conforming to the identity of the person frame by frame, denoted as... This sequence fully defines all movements of the character's head, lips, and facial expressions in the first video segment.

[0133] S430, based on identity encoding and the second audio segment, generates a second motion latent vector sequence corresponding to the second audio segment through the FLOAT model.

[0134] The second audio segment, similar to the first, refers to speech input at a different time point, serving as the basis for generating the second motion latent vector sequence. The second audio segment corresponds to the subsequent speech content of the person in the video, and may be sequentially connected to or separated from the first audio segment.

[0135] Specifically, using the same method as step S420, this embodiment of the invention can feed the identity code and the second audio segment into the FLOAT model, outputting a second motion latent vector sequence that matches the second audio segment. This allows for the acquisition of the continuous dynamic motion latent representation of the character driven by the two audio segments.

[0136] As examples, similar to step S420, embodiments of the present invention can input the second audio segment into the audio encoder to obtain a new audio code (wa', we'). Similarly, the new audio code is combined with the cached identity residual code and input into the motion generator of the FLOAT model. A second motion latent vector sequence is independently generated, denoted as... It should be noted that due to the randomness of FLOAT model frame generation, and The last frame The movement is usually discontinuous, and direct splicing will produce abrupt changes.

[0137] S440. According to the above video frame interpolation method, the last frame motion potential vector of the first motion potential vector sequence is used as the starting motion potential vector, and the first frame motion potential vector of the second motion potential vector sequence is used as the ending motion potential vector. Interpolation is performed to generate a transition motion potential vector sequence, wherein the transition motion potential vector sequence contains multiple intermediate motion potential vectors.

[0138] The first motion latent vector sequence refers to a series of motion latent vectors generated by the FLOAT model based on the person's identity code and the first audio segment. It describes the continuous motion state of the person as they move with the speech content of the first audio segment (such as facial expressions, lip movements, and head movements), and is the latent representation of the corresponding first video stream.

[0139] The second motion latent vector sequence refers to a series of motion latent vectors generated by the FLOAT model based on the person's identity code and the second audio segment. It describes the continuous motion state of the person as the second audio segment's speech content is presented, and is the latent representation of the corresponding second video stream.

[0140] The transitional motion latent vector sequence refers to a sequence of multiple intermediate motion latent vectors generated by an interpolation network between the end of the first motion latent vector sequence and the beginning of the second motion latent vector sequence. The transitional motion latent vector sequence is used to achieve a smooth transition between two motion states, allowing the video to naturally switch from the first segment to the second, eliminating discontinuities or abrupt changes caused by direct splicing.

[0141] Specifically, in this embodiment of the invention, the last frame of the first motion potential vector sequence can be extracted as the starting motion potential vector, and the first frame of the second motion potential vector sequence can be used as the ending motion potential vector. Both, along with the time required for frame interpolation, are embedded into an input frame interpolation network. The frame interpolation network uses linear interpolation combined with residual compensation to generate multiple consecutive intermediate motion potential vectors, forming a transition motion potential vector sequence, thus achieving a natural transition between two motion states.

[0142] As some examples, embodiments of the present invention can use the motion latent vector of the last frame of the first sequence. as the potential vector of initial motion The motion latent vector of the first frame of the second sequence. As the potential vector for terminating motion .Will and The input is fed into a pre-trained interpolation network. The network, based on a preset number of transition frames (e.g., 10 frames) and incorporating temporal coding, calculates the residuals for all intermediate points in parallel. According to the formula... Synthesize a complete sequence of potential vectors for transitional motion. This sequence achieves a change in motion state from... arrive Its natural and smooth evolution.

[0143] S450. Insert the transitional motion potential vector sequence between the first motion potential vector sequence and the second motion potential vector sequence to obtain a continuous target motion potential vector sequence.

[0144] The target motion latent vector sequence refers to the complete and continuous motion latent vector sequence formed by connecting the first motion latent vector sequence, the transitional motion latent vector sequence, and the second motion latent vector sequence in chronological order. The target motion latent vector sequence reflects the continuous motion state changes of the entire video from the first audio segment to the second audio segment.

[0145] Specifically, in this embodiment of the invention, a transitional motion latent vector sequence can be sequentially inserted between the end of the first motion latent vector sequence and the beginning of the second motion latent vector sequence, making the entire sequence coherent and smooth in the motion latent space. This ultimately forms a complete target motion latent vector sequence that smoothly transitions from the first audio segment to the second audio segment.

[0146] As some examples, embodiments of the present invention can be carried out in accordance with , and The three motion potential vector sequences are concatenated end-to-end in the time dimension, following a specific order. The concatenation results in a continuous and smooth sequence of target motion potential vectors. This sequence fully describes the entire continuous movement process of a character, from performing the first action to naturally transitioning to performing the second action.

[0147] S460, based on the target motion potential vector sequence and identity encoding, synthesizes a continuous video stream.

[0148] In this context, a video stream refers to a continuous sequence of video frames generated by the decoding and synthesis modules of the FLOAT model, based on the target's latent motion vector sequence and identity encoding. The video stream presents the visual effect of a person moving and speaking continuously and naturally according to the input audio content, and is the output viewed by the end user.

[0149] Specifically, embodiments of the present invention can input the complete target motion latent vector sequence and identity code into the video decoder or synthesis network of the FLOAT model, and synthesize the corresponding frame's person image based on the motion latent vector and identity features of each frame. All synthesized frames are sequentially connected to form a visually coherent, consistent, and voice-driven synchronized video stream, realizing seamless digital-driven video generation of multiple audio content segments.

[0150] As some examples, embodiments of the present invention can generate a continuous sequence of potential vectors for target motion. Along with the cached identity encoding, the data is fed into the generator of the FLOAT model, which is based on a Generative Adversarial Network (GAN) or a diffusion model. Based on the motion latent vector of each frame (controlling dynamics) and the fixed identity encoding (controlling static appearance), the generator generates corresponding character images frame by frame. All generated image frames are combined chronologically and encoded into a continuous video stream. In the final output video, the character's identity remains consistent, and a natural, smooth visual transition is achieved at the junctions of different content segments, solving the problem of disjointed and abrupt transitions between segments in the original FLOAT model output.

[0151] This invention directly acquires the motion latent representation during the inference process of the FLOAT model, eliminating the need for an external frame interpolation model. Instead, it constructs its own frame interpolation network to perform frame interpolation within the latent space, significantly improving the speed and efficiency of frame interpolation. Simultaneously, the motion latent itself contains rich pose and motion trend information, making the transition frames generated by the interpolation more natural and coherent in terms of motion expression, effectively eliminating abrupt changes and inconsistencies between multiple video segments. The frame interpolation network employs temporal self-attention combined with multi-scale convolution for feature fusion, ensuring not only the stability and consistency of the frame interpolation process but also improving the quality of the generated frame sequence. Through these improvements, this invention achieves a 2-5 times speed increase compared to traditional image domain frame interpolation schemes, with significantly enhanced stability and visual effects. It enables efficient, smooth, and natural transitions between multiple audio-driven video segments, demonstrating significant technical advantages and application value.

[0152] Optionally, in the above Figure 4 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, after step S400, the video generation method may further include:

[0153] Based on the prototype reference image of the character and the blank audio, the motion potential vector sequence of the idle state is generated by the FLOAT model.

[0154] Specifically, in this embodiment of the invention, after the user inputs a prototype reference image of a person, an identity code representing the person's identity characteristics is generated. Next, a short audio clip (such as silent or non-vocal audio) is input, and the identity code and the features of the silent audio are fused using the FLOAT model to generate a motion latent vector sequence in an "Idle state" (a state where the person is naturally idle and not speaking). This motion latent vector sequence reflects the person's natural facial expressions and micro-movements when not speaking, providing a smooth starting point for subsequent state transitions and frame interpolation.

[0155] As examples, embodiments of the present invention can generate or load a segment of blank audio (silent audio), the duration of which can be set as needed (e.g., corresponding to a few seconds of idle video). The cached identity encoding (especially the identity residual encoding) is input together with the encoding of the blank audio into the motion encoder of the FLOAT model. Based on these inputs, the motion encoder generates a sequence of latent motion vectors for the idle state. This sequence encodes the slight, natural breathing or subtle movements of a person in the absence of speech, and is continuously looped or retained from the last frame, forming the initial background state of the video and providing a smooth starting point for subsequent dialogue.

[0156] Step S420 may specifically include: when the first audio segment is a streamed audio segment, upon receiving the first audio segment, determining the interpolation start point based on the last frame motion latent vector of the motion latent vector sequence in the idle state; according to the above video frame interpolation method, using the motion latent vector corresponding to the interpolation start point as the starting motion latent vector and the first frame motion latent vector generated based on the first audio segment as the ending motion latent vector, performing interpolation to generate a transition motion latent vector sequence from the idle state to the dialogue state; generating the remaining motion latent vectors excluding the first frame based on the first audio segment and concatenating them with the transition motion latent vector sequence to generate the first motion latent vector sequence.

[0157] Specifically, in this embodiment of the invention, the last frame of the idle state motion latent vector sequence can be used as the frame interpolation start point. That is, the motion latent vector at the end of the Idle state serves as the starting point for the transition from Idle to Talk; when the first audio segment arrives streaming, preparations are immediately made to generate the transition from Idle to Talk. At this time, the last frame in the Idle motion latent vector sequence is determined as the starting motion latent vector (…). The motion latent vector of the first frame generated based on the first audio segment is used as the terminating motion latent vector. The input is a frame interpolation network. This network uses linear interpolation combined with residual compensation, along with temporal self-attention and multi-scale convolutional structures, to generate multiple intermediate transitional motion latent vectors (e.g., 10 frames), achieving a natural transition from Idle to Talk. The FLOAT model generates a corresponding sequence of motion latent vectors frame by frame based on the first audio segment. Since the first frame has already served as the interpolation termination frame for the transition, subsequent motion latent vectors (i.e., the second frame and all subsequent frames) are sequentially concatenated with the aforementioned transitional motion latent vector sequence to form a complete and coherent "first motion latent vector sequence." Thus, the final motion latent vector sequence represents a natural transition between Idle and Talk (the first audio segment).

[0158] As examples, embodiments of the present invention can continuously monitor streaming audio output from text-to-speech. When the arrival of the first valid audio segment is detected, a transition process is triggered: the computation and rendering time required for the frame interpolation process is estimated (e.g., the time required to generate 10 transition frames). Based on this duration, the latent vector sequence of the currently playing Idle motion is... In the process, tracing back, a suitable frame is determined as the starting point for the interpolation transition. The motion latent vector corresponding to the found starting point (such as...) The k-th frame from the end is determined as This refers to the starting motion latent vector for interpolation, ensuring a smooth transition from a natural state point within the Idle state, rather than starting from the potentially incomplete last frame. The first received audio segment is input into the FLOAT model to generate the corresponding Talk motion latent vector sequence. The motion latent vector of its first frame is then extracted and used as the ending motion latent vector for interpolation. The determined (From the Idle sequence) and (From the first sentence of Talk) Input the trained interpolation network. The interpolation network generates a sequence of latent motion vectors for transitions based on a preset number of transition frames (e.g., 10 frames). This sequence describes the natural and smooth evolution from the Idle state to the state of beginning to speak the first sentence. : From the complete Talk motion latent vector sequence generated based on the first audio segment, remove those already used as... The first frame preserves the remaining sequence. The three parts are concatenated in chronological order: the portion from the Idle sequence up to the interpolation start point, the transition sequence, and so on. h and Talk remaining sequences The complete sequence obtained after splicing is the first motion potential vector sequence. It fully represents the character's natural transition from the Idle state and the smooth delivery of the first paragraph, perfectly solving the visual jump problem caused by direct splicing.

[0159] This invention first generates a sequence of motion latent vectors for the idle state based on a prototype reference image of a person and blank audio, ensuring that the first frame of the Idle state can be presented instantly, significantly reducing the first frame latency. When the first audio segment arrives in the streaming reception, the motion latent vector of the last frame of the idle state is used to determine the interpolation starting point, and interpolation is performed with the motion latent vector of the first frame generated from the first audio segment to form a smooth transition motion latent vector sequence. This sequence is then spliced ​​with subsequent frames to complete a natural and smooth switch from the Idle to the dialogue state, thereby achieving seamless connection and smooth transition between dialogue animations, greatly improving the user experience, ensuring a fast response in the first frame and a coherent animation transition.

[0160] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0161] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0162] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the video frame interpolation method and / or the video generation method.

[0163] This invention provides a processor for running a program, wherein the program executes the video frame interpolation method and / or the video generation method.

[0164] like Figure 5 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned video frame interpolation method and / or the video generation method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0165] The present invention also provides a computer program product, which, when executed on an electronic device, is adapted to execute a program having the steps of initializing the video interpolation method and / or the video generation method.

[0166] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0167] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0168] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0169] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0171] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0172] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0173] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0174] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A video frame interpolation method, characterized in that, include: Obtain the initial motion potential vector and the final motion potential vector; Linear interpolation is performed on the initial motion potential vector and the final motion potential vector to obtain a linear interpolation vector corresponding to at least one interpolation frame time point; A conditional feature sequence is generated using the initial motion potential vector, the final motion potential vector, and the temporal embedding vectors at each interpolated time point; The conditional feature sequence is input into the interpolation network, so that the interpolation network processes the conditional feature sequence based on temporal self-attention and multi-scale one-dimensional convolution, and outputs the residual vector corresponding to each interpolation frame time point; The linear interpolation vector corresponding to each interpolation time point is added to the corresponding modulated residual vector to generate at least one intermediate motion potential vector located between the starting motion potential vector and the ending motion potential vector.

2. The video frame interpolation method according to claim 1, characterized in that, The step of generating a conditional feature sequence using the initial motion latent vector, the final motion latent vector, and the temporal embedding vectors of each interpolated frame time point includes: Determine at least one interpolation time point within the specified interpolation interval; The temporal embedding vector corresponding to each interpolated frame time point is obtained through a positional encoding function; The starting motion potential vector, the ending motion potential vector, and the temporal embedding vector of each interpolated time point are concatenated to obtain the conditional features of each interpolated time point. The condition feature sequence is obtained by utilizing the condition features of all interpolated time points.

3. The video frame interpolation method according to claim 2, characterized in that, Before adding the linear interpolation vector corresponding to each interpolation time point to the corresponding modulated residual vector, the method further includes: The residual vector corresponding to each interpolation time point is multiplied by the value of a boundary constraint shape function at that interpolation time point to obtain the modulated residual vector, wherein the boundary constraint shape function has a value of zero at the start and end of the interpolation interval.

4. The video frame interpolation method according to claim 3, characterized in that, The boundary constraint shape function is: ,in, This refers to the frame insertion time point.

5. The video frame interpolation method according to claim 1, characterized in that, The training process of the interpolation network includes: Obtain a continuous motion latent vector sequence generated by the FLOAT model; From the continuous motion potential vector sequence, the motion potential vector corresponding to the starting frame is selected as the starting motion potential vector sample, and the motion potential vector corresponding to the ending frame is selected as the ending motion potential vector sample. Select the motion potential vector of at least one intermediate frame located between the start frame and the end frame from the continuous motion potential vector sequence as a supervision sample; Using the initial motion latent vector sample and the final motion latent vector sample as input, and the supervision sample as the ground truth label, the network containing the temporal self-attention module and the multi-scale one-dimensional convolution module is trained to obtain the trained interpolation network.

6. The video frame interpolation method according to claim 5, characterized in that, When training the interpolation network, the loss function used includes at least one of reconstruction loss, first-order smoothing loss, second-order smoothing loss, and residual energy constraint loss.

7. A video generation method, characterized in that, include: Obtain a prototype reference image of the character and at least two audio inputs; An identity code representing identity features is generated based on the aforementioned prototype reference image; Based on the identity encoding and the first audio segment, a first motion latent vector sequence corresponding to the first audio segment is generated using the FLOAT model; Based on the identity encoding and the second audio segment, a second motion latent vector sequence corresponding to the second audio segment is generated using the FLOAT model; According to any one of claims 1 to 6, the video frame interpolation method uses the last frame motion potential vector of the first motion potential vector sequence as the starting motion potential vector and the first frame motion potential vector of the second motion potential vector sequence as the ending motion potential vector to perform interpolation to generate a transition motion potential vector sequence, wherein the transition motion potential vector sequence includes a plurality of intermediate motion potential vectors. The transitional motion potential vector sequence is inserted between the first motion potential vector sequence and the second motion potential vector sequence to obtain a continuous target motion potential vector sequence; Based on the target motion potential vector sequence and the identity encoding, a continuous video stream is synthesized.

8. The video generation method according to claim 7, characterized in that, After obtaining the prototype reference image of the figure and at least two audio inputs, the method further includes: Based on the prototype reference image of the character and the blank audio, a sequence of motion potential vectors in the idle state is generated using the FLOAT model; The step of generating a first motion latent vector sequence corresponding to the first audio segment using the FLOAT model based on the identity encoding and the first audio segment includes: When the first audio segment is a streaming audio segment, when the first audio segment is received, the interpolation start point is determined based on the motion potential vector of the last frame of the motion potential vector sequence in the idle state. According to any one of claims 1 to 6, the video frame interpolation method uses the motion potential vector corresponding to the interpolation start point as the starting motion potential vector and the motion potential vector of the first frame generated based on the first audio segment as the ending motion potential vector to perform interpolation to generate a transition motion potential vector sequence from the idle state to the dialogue state. Based on the first audio segment, generate the remaining motion potential vectors excluding the first frame, and concatenate them with the transition motion potential vector sequence to generate the first motion potential vector sequence.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the video frame interpolation method as described in any one of claims 1 to 6, and / or the video generation method as described in any one of claims 7 to 8.

10. An electronic device, characterized in that, The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the video frame interpolation method as described in any one of claims 1 to 6, and / or the video generation method as described in any one of claims 7 to 8.

Images