Video generation model construction method, video generation method, and related device

Abstract

The application relates to the technical field of virtual digital humans and discloses a video generation model construction method, a video generation method and related devices. The video generation model construction method comprises the following steps: obtaining training data and inputting a teacher model and a student model to be trained; intermediate features and attention maps of the teacher model and the student model at respectively selected target network blocks are extracted; the intermediate features and the attention maps of the student model are respectively aligned with the intermediate features and the attention maps of the teacher model; based on the aligned intermediate features and the attention maps, feature distillation loss, attention distillation loss, real output alignment loss and final output distillation loss are calculated, and a total loss is obtained by weighted summation; based on the total loss, parameters of the student model are updated until the model converges, and a trained video generation model is obtained. The application meets the low-latency requirement of real-time live streaming scenarios for video generation and realizes lightweight deployment of the video generation model.

Description

Technical Field

[0001] This invention relates to the field of virtual digital human technology, and in particular to a method for constructing a video generation model, a video generation method, and related apparatus. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially in fields such as live streaming and virtual anchors, AI digital human technology has emerged to solve problems such as high cost and limited duration in live streaming. Traditional digital human technology is usually based on image-driven or parametric models, which significantly reduces its effectiveness when dealing with complex scenarios such as occlusion and side profiles.

[0003] In recent years, video generation technology based on diffusion models has made groundbreaking progress, enabling the generation of high-quality, high-fidelity video content. Injecting audio conditions (such as Mel spectrum and phoneme embedding) into these models allows for audio-driven facial and body motion generation, with the generated videos exhibiting significantly higher synchronization between facial expressions, gestures, and speech compared to traditional methods. However, these large-scale video generation models are computationally extremely complex and time-consuming, making it difficult to meet the low latency requirements of real-time live streaming scenarios (typically requiring first-frame latency in the hundreds of milliseconds). Summary of the Invention

[0004] The main objective of this invention is to provide a video generation model construction method, a video generation method, and related apparatus, aiming to solve the technical problems of existing large-scale video generation models having extremely high computational complexity, long generation time, and difficulty in meeting the requirements of real-time live streaming scenarios.

[0005] The first aspect of this invention provides a method for constructing a video generation model, comprising: Acquire training data and input the training data into the teacher model and the student model to be trained; Extract the intermediate features and attention maps of the teacher model and the student model at their respective selected target network blocks; Align the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model respectively. Based on the aligned intermediate features and attention map, the feature distillation loss, attention distillation loss, true output alignment loss and final output distillation loss are calculated and weighted to obtain the total loss; Based on the total loss, the parameters of the student model are updated until the model converges, resulting in a trained video generation model.

[0006] Optionally, in a second implementation of the first aspect of the present invention, obtaining the training data includes: Obtain a real video dataset, and select high-quality images from the real video dataset as the first frame image, and slice the corresponding audio data from the real video data containing the first frame image; The first frame image and the audio data are input into the teacher model to generate a synthetic video dataset; The training dataset is obtained by mixing the real video dataset and the synthetic video dataset; The first frame image is associated with the corresponding audio data and video data to form training data, wherein the first frame image and the audio data are used as input conditions, and the video data is used as noise input.

[0007] Optionally, in a third implementation of the first aspect of the present invention, both the teacher model and the student model include a denoising model, wherein the number of network blocks in the denoising model of the student model is less than the number of network blocks in the denoising model of the teacher model; the network blocks in the denoising models of both the teacher model and the student model each include three stage partitions: early stage network blocks, mid-stage network blocks, and late stage network blocks, with each stage partition corresponding to a different partition ratio; the index set of the target network blocks selected in each of the teacher model and the student model is related to the partition ratio and the total number of network blocks in the teacher model and the student model.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, aligning the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model respectively includes: Determine the differences between the student model and the teacher-generated model in intermediate features and attention maps, respectively, including differences in channel dimension, spatial dimension and temporal dimension; If there are differences in the channel dimension of the intermediate features, a convolution operation is used to align the number of feature channels of the student model with that of the teacher model; if there are differences in the spatial dimension or temporal dimension of the intermediate features, adaptive pooling is used to align the size of the intermediate features of the student model or the frame sequence length with that of the teacher model. If there are differences in the channel dimension of the attention map, a convolution operation is used to align the number of attention heads of the student model with that of the teacher model; if there are differences in the spatial or temporal dimensions of the attention map, adaptive pooling is used to align the size of the attention map of the student model or the frame sequence length with that of the teacher model.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of calculating the feature distillation loss, attention distillation loss, true output alignment loss, and final output distillation loss based on the aligned intermediate features and attention map, and then weighted summing them to obtain the total loss, includes: The intermediate feature differences between the aligned student model and the teacher model at the target network block are calculated and used as the feature distillation loss. The difference in attention maps between the aligned student model and the teacher model at the target network block is calculated and used as the attention distillation loss. The difference between the final output of the student model after alignment and the real video data is calculated and used as the alignment loss of the real output. The difference between the final output of the aligned student model and the final output of the teacher model is calculated and used as the final output difference loss. The feature distillation loss, the attention distillation loss, the true output alignment loss, and the final output distillation loss are assigned weight coefficients and then summed in a weighted manner to obtain the total loss.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, updating the parameters of the student model based on the total loss until the model converges to obtain a trained video generation model includes: Based on the total loss, calculate the gradient of each parameter in the student model; The optimizer updates the parameters of the student model based on the gradient; During the training iteration, the proportion of the synthetic video dataset in the training dataset is dynamically adjusted so that the proportion decreases as the training progresses. Monitor the trend of loss changes and determine whether the model has converged; When the change in loss is less than a preset threshold or the maximum number of iterations is reached, the model is considered to have converged, training is stopped, and a well-trained video generation model is obtained.

[0011] A second aspect of the present invention provides a video generation method, comprising: When the first video segment is generated, input data is acquired, which includes a reference image and audio data of the target live streaming object; The input data is input into the video generation model to obtain the first video segment, wherein the video generation model is obtained according to any of the video generation model construction methods described above; When a non-first video segment is generated, a preset number of frame images are extracted from the end of the previous video segment generated by the video generation model, and new audio data is obtained. The extracted frame images and the acquired new audio data are used as input data for the next round of video generation model and input into the video generation model to obtain the next video segment; Repeat the video generation process to output continuous live video of digital humans.

[0012] A third aspect of the present invention provides a video generation model construction apparatus, the video generation model construction apparatus comprising: An input module is used to acquire training data and input the training data into the teacher model and the student model to be trained. The extraction module is used to extract the intermediate features and attention maps of the teacher model and the student model at their respective selected target network blocks; The alignment module is used to align the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model, respectively. The loss module is used to calculate the feature distillation loss, attention distillation loss, true output alignment loss, and final output distillation loss based on the aligned intermediate features and attention map, and then sum them up to obtain the total loss. The update module is used to update the parameters of the student model based on the total loss until the model converges, thus obtaining a trained video generation model. A fourth aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the computer device to perform the video generation model construction method and / or the video generation method described above.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the video generation model construction method and / or the video generation method described above.

[0014] The video generation model construction method provided in this embodiment first acquires training data, then inputs the training data into a teacher model and a student model to be trained for video generation processing. Both the teacher model and the student model generate intermediate features and attention maps. In this embodiment, to ensure that the student model can learn more from the experience of the teacher model, intermediate features and attention maps are extracted from their respective selected target network blocks. Then, the intermediate features and attention maps of the student model are aligned with those of the teacher model. This alignment of intermediate features and attention maps ensures the accuracy and efficiency of knowledge distillation. Finally, based on the aligned intermediate features and attention maps, feature distillation loss, attention distillation loss, true output alignment loss, and final output distillation loss are calculated and weighted to obtain the total loss. Based on the total loss, the parameters of the student model are updated until the model converges, resulting in a trained video generation model. Since the video generation model used in this embodiment is obtained by training the student model based on the teacher model, the video generation model is lightweight, thereby reducing computational complexity and video generation time. The embodiments of the present invention employ a lightweight video generation model, which meets the low latency requirements for video generation in real-time live streaming scenarios and achieves lightweight deployment of the video generation model. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of one embodiment of the video generation method in this invention; Figure 2 This is a schematic diagram of one embodiment of the video generation model training method in this invention; Figure 3 This is a schematic diagram of one embodiment of the video generation model construction device in this invention; Figure 4 This is a schematic diagram of one embodiment of the computer device in this invention. Detailed Implementation

[0016] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the video generation method in this invention includes: 101. When the first video segment is generated, the input data is obtained, which includes the reference image and audio data of the target live streaming object; In this embodiment, before generating the first video clip, it is necessary to obtain the original materials that drive the generation of the digital human, that is, the input data of the model, including the reference image and audio data of the target live streaming object.

[0018] (1) Reference image of the target live streaming object Reference images are static images used to determine the physical features of a digital human, such as facial contours, facial details, skin texture, and clothing style. In live streaming scenarios, these reference images are typically uploaded or selected by the user before the live stream begins.

[0019] Before inputting the reference image into the model, a series of preprocessing operations can be performed on the reference image to ensure the quality of the video generation. For example, a face detection algorithm can be used to locate the face region in the image. Secondly, based on the detected facial key points (such as the corners of the eyes, the tip of the nose, and the corners of the mouth), an affine transformation is performed on the image to align the face to a standard coordinate system, ensuring consistency in the angle of the frontal or side view. Simultaneously, the aligned image is cropped and scaled to the input size required by the model (e.g., 512x512). Finally, the image pixel values ​​are normalized from [0, 255] to a floating-point range of [-1, 1] and converted to tensor format for input into the neural network.

[0020] (2) Audio data Audio data is the driving force behind digital humans' speech, gestures, and facial expressions. It can originate from real-time microphone input or from pre-recorded audio files. In live streaming scenarios, it typically comes from real-time microphone input or real-time audio streams generated by text-to-speech engines.

[0021] Because raw waveform data contains a lot of redundant information and is difficult to align directly with video features, systems typically do not directly input the waveform into the model, but instead convert it into acoustic features. Common processing methods include extracting Mel spectra or using pre-trained audio encoders to extract high-dimensional semantic features. Furthermore, to support streaming generation, the continuous audio stream needs to be cut into fixed-length segments. An overlapping slicing strategy is usually used, where adjacent segments overlap (e.g., 50%) to eliminate jitter at segment seams.

[0022] 102. Input the input data into the video generation model to obtain the current video segment, wherein the video generation model is obtained according to the video generation model construction method; This embodiment specifically utilizes a lightweight student model for video generation. This student model is trained based on the teacher model and features a small number of parameters and fast inference speed.

[0023] The video generation model in this embodiment is typically based on a diffusion model architecture, which mainly consists of three parts: an audio encoder, a variational autoencoder, and a denoising model.

[0024] Audio encoder: Responsible for encoding the acquired audio data into audio conditional embeddings aligned with the video latent space.

[0025] Variational autoencoders (VAEs) consist of an encoder and a decoder. During the inference phase, the decoder of the VAE is primarily used. The role of a VAE is to transform between the pixel space and the latent space. Typically, the image in the pixel space is compressed by 8 to 16 times into the latent space, significantly reducing computational cost.

[0026] The denoising model, also known as the denoising network, is the core of the student model and is typically a lightweight U-Net or Transformer structure. Its role is to progressively remove noise from the latent space to generate a clear latent representation of the video.

[0027] In one embodiment, the process of generating video clips based on a video generation model is as follows: (1) Initial latent state: For diffusion-based generation, it usually starts with a pure Gaussian noise tensor. For example, the noise tensor (1,4,16,32,32) represents the latent features of 1 sample, 4 channels, 16 frames, and a resolution of 32x32.

[0028] (2) Conditional injection: The reference image is encoded into latent image features by the encoder of the variational autoencoder (if it is generated from the first frame, it is used directly; if it is generated continuously, the previous frame is used as input). At the same time, the audio features are injected into the denoising model through the cross-attention mechanism.

[0029] (3) Denoising iteration: The denoising model receives the current noise latent state, time step, audio conditions, image conditions, and predicted noise at time t. Then, it updates the latent state according to the diffusion formula.

[0030] (4) Decoding output: After multiple denoising steps, the final clear latent features are obtained. The decoder of the input variational autoencoder is decoded back into pixel space to obtain the generated video frame sequence.

[0031] 103. When a non-first video segment is generated, a preset number of frame images are extracted from the end of the previous video segment generated by the video generation model, and new audio data is obtained. To ensure the continuity of the video stream and avoid jumps between video segments generated in each round, the results generated in the previous round need to be used as the input context for the next round.

[0032] Suppose the video clip generated in the previous round contained N frames (e.g., 16 frames). Instead of discarding all N frames, the system retains the last K frames. K is a hyperparameter, typically set based on the model's temporal receptive field. For example, if the model's temporal receptive field is 8 frames, then K should be at least 8 frames to ensure the model sees enough historical information to predict the motion trend of the next frame. K is usually less than N, for example, N=16, K=4 or K=8.

[0033] The frame images extracted in this embodiment can be either original RGB images in pixel space or latent features encoded by a variational autoencoder. To accelerate the next round of inference, it is preferable to extract and save the latent features, so that the variational autoencoder encoding process for these frames can be skipped during the next round of generation, and they can be used directly as input conditions.

[0034] This step essentially creates a sliding window. By preserving the tail frame, the task of generating a long video is transformed into generating shorter, overlapping video clips. The overlapping frames (i.e., the extracted tail frames) act as anchors in the next generation round, constraining the model's starting state to remain consistent with the ending state of the previous round, thus visually eliminating stitching artifacts.

[0035] 104. Use the extracted frame images and the acquired new audio data as input data for the next round of video generation model and input them into the video generation model to obtain the next video segment; This step is used to construct the complete input conditions required for the next round of inference. The system obtains the audio data for the next duration from the audio stream. For example, if the previous round processed the audio from 0.0 to 0.64 seconds, this round obtains the audio from 0.64 to 1.28 seconds. Similarly, feature extraction is performed on this audio segment to obtain new audio features. In this embodiment, the tail frame image extracted in the previous step is used as the image condition input, while the new audio features are used as the driving signal.

[0036] In addition, it is necessary to ensure that the new audio data and the extracted frame images are logically continuous in time. For example, if the audio time corresponding to the last extracted frame is t, then the starting time of the new audio data should be t+Δt (Δt is the time interval of one frame) to ensure strict synchronization between lip movements and sound.

[0037] 105. Repeat the video generation process to output continuous live video of the digital human.

[0038] This step is an iterative process: generate segments -> extract the tail frame to obtain new audio -> combine inputs -> generate the next segment, thus outputting continuous live video of the digital human until the live stream ends or a stop command is received. In this way, short video segments are connected to form a continuous video stream. In this embodiment, the repeated execution of the video generation process corresponds to the loop execution of steps 103-105 until a stopping condition is met, such as reaching the total video generation duration.

[0039] Furthermore, in one embodiment, to ensure a good viewing experience for viewers, a display buffer is typically set up instead of streaming each frame generated (because the generation speed may fluctuate). That is, in each round of N frames generated, the last K frames reserved for the next round's context are removed, and the remaining first NK frames are the truly "new" valid frames for that round. The system pushes these valid frames into a buffer queue. Then, the streaming server or player reads frame data from the queue at a fixed frame rate (e.g., 25fps) for display. As long as the generation speed is slightly faster than or equal to the playback speed (i.e., the time to generate N frames ≤ the time to play N frames), the buffer will not run out, and the live stream can proceed smoothly.

[0040] The student model used in this embodiment is smaller and has a faster inference speed, which significantly reduces the time required to generate a frame (or a video segment). For example, the teacher model may take 500ms to generate a frame, while the student model only takes 40ms. Under the 25fps live streaming standard, each frame is spaced 40ms apart. This means that the student model can generate frames in real time, or even slightly faster than real time, thereby greatly reducing first-frame latency and end-to-end latency.

[0041] For example, in the first round, the model generates frames 1-16, outputs frames 1-12, and retains frames 13-16; in the second round, it generates frames 17-32 based on frames 13-16, outputs frames 13-24, and retains frames 25-32, and so on. Since the student model only needs 300 milliseconds to generate 16 frames, while playing 16 frames takes 640 milliseconds, the generation speed is more than twice the playback speed. Therefore, viewers see a very smooth, lag-free virtual digital human speaking continuously. This embodiment achieves efficient and continuous digital human live video generation through a lightweight model and a sliding window mechanism.

[0042] Please see Figure 2 , Figure 2This is one embodiment of the video generation model training method in this invention. Both the teacher and student models used in this embodiment include denoising models, and the number of network blocks in the student model is less than the number of network blocks in the teacher model.

[0043] The teacher model is a large, pre-trained model with excellent performance but a large number of parameters on a large dataset. During distillation, the parameters of the teacher model are frozen and do not participate in gradient updates. The student model, on the other hand, is a smaller model with a simpler structure and fewer parameters. For example, the teacher model's U-Net has a depth of 32 layers and 640 channels; the student model's U-Net might only have a depth of 16 layers and 320 channels. The student model's parameters are randomly initialized and need to be updated through training.

[0044] In this embodiment, the video generation model is obtained through the following training steps: 201. Obtain training data and input the training data into the teacher model and the student model to be trained; The quality and diversity of training data directly determine the distillation effect. The training data in this embodiment not only includes high-quality audio and video pairs collected from the real world, but also synthetic data generated by the teacher model.

[0045] Real-world video datasets refer to large collections of video clips of people speaking, gathered from the internet or professional libraries. These videos encompass diverse races, ages, genders, lighting conditions, postures, and backgrounds.

[0046] Synthetic video datasets: These refer to datasets where pseudo-labeled data is generated by the teacher model based on a portion of real data, in order to enable the student model to quickly learn the distribution of the teacher model.

[0047] In an optional embodiment, step 201 above further includes: 2011. Obtain a real video dataset, and select high-quality images from the real video dataset as the first frame image and slice the corresponding audio data from the real video data containing the first frame image; In this embodiment, the real data can be collected from publicly available datasets on the Internet or from a privately owned copyrighted material library, consisting of large-scale video data of people speaking. The dataset needs to cover a wide range of scenarios, including different skin tones, ages, genders, lighting conditions (e.g., soft indoor light, strong outdoor light), shooting angles (e.g., front view, side view, upward view, downward view), and background complexity (e.g., solid color background, dynamic background, cluttered background).

[0048] (1) The first frame image specifically refers to the first frame image in the video segment to be generated, which determines the identity and initial posture of the person in the generated video. The first frame image can be selected based on factors such as image clarity and facial features. To ensure the high quality of the first frame image, image enhancement processing can also be performed on the selected first frame. For example, a super-resolution model can be used to enlarge the low-resolution first frame to high definition; a face restoration model can be used to remove noise or blemishes; and the contrast and brightness can be adjusted to conform to the standard distribution, etc.

[0049] (2) Audio data needs to be extracted from real video data containing the first frame image, and also needs to be sliced ​​to ensure that the length of the audio slice strictly matches the duration of the video slice. For example, if the video is 25 frames (25fps), then the corresponding number of audio sampling points is 1×16000 (assuming a sampling rate of 16kHz). In addition, in one embodiment, to accelerate training and reduce computation, the original waveform is usually not used directly, but acoustic features are extracted. The most commonly used is the Mel spectrum. The audio waveform is converted into a time-frequency graph through a short-time Fourier transform, and then a Mel filter bank is applied to obtain the Mel spectrum.

[0050] For example, suppose a real video file is acquired. The system first decodes it. The total video duration is 60 minutes. The system divides it into 1200 3-second segments. For segment number 45, V1, the system determines that the image is clear and the face is facing the camera, so it retains it. The system extracts the first frame of this segment, performs super-resolution processing to obtain a clear frontal photo, which is used as I1 for this sample. The system also extracts the 3 seconds of audio, performs Mel transform to obtain an 80×300 feature matrix A1.

[0051] 2012. Input the first frame image and the audio data into the teacher model to generate a synthetic video dataset; This embodiment leverages the powerful video generation capabilities of the teacher model to construct pseudo-realistic data, which guides the student model to quickly learn the distribution of the teacher model. Therefore, the selected first frame image I1 and its corresponding audio feature A1 need to be batched and input into the teacher model whose parameters have been frozen.

[0052] To balance quality and efficiency when generating synthetic video data, fewer sampling steps than standard inference are typically used (e.g., reduced from 50 steps to 20 steps), or a specified sampling strategy is employed to ensure reproducibility. The teacher model, based on audio conditions, denoises in the latent space, generating corresponding video latent features, which are then decoded back to pixel space by a variational autoencoder decoder to obtain the synthetic video frame sequence V2. Although the synthetic video data may not be as clear as real video data in details (such as finger textures and background text), and may even contain subtle flaws unique to the teacher model (such as slight hand tremors), its overall motion patterns, lip-sync accuracy, and latent manifold distribution are highly consistent with the teacher model.

[0053] 2013. The real video dataset and the synthetic video dataset are mixed to obtain the training dataset; This embodiment requires mixing real video datasets and synthetic video datasets. The mixing ratio is not fixed but dynamically changes, depending on the training progress.

[0054] The data structure of the real video dataset includes: the first frame image I1, the corresponding audio waveform A1, and the real video frame sequence V1, while the data structure of the synthetic video dataset includes: the first frame image I1, the corresponding audio waveform A1, and the synthetic video frame sequence V2 generated by the professor model.

[0055] 2014. The first frame image is associated with the corresponding audio data and video data to form training data, wherein the first frame image and the audio data are used as input conditions, and the video data is used as noise input.

[0056] In this embodiment, after obtaining the training dataset, each first frame image needs to be further associated with the corresponding audio and video data to form training data. For example, a real data sample in the real video dataset is {I1,A1,V1}. A pseudo-real data sample in the synthetic video dataset is {I1,A1,V2}. The first frame image and audio data serve as input conditions, while the video data serves as the input for adding noise. The video data (whether V1 or V2) is not directly input into the denoising model. Instead, it is first encoded by a variational autoencoder to obtain latent features, and then Gaussian noise is added according to the diffusion process to obtain the noise data, which is then used as the input to the denoising model.

[0057] 202. Extract the intermediate features and attention maps of the teacher model and the student model at their respective selected target network blocks; To enable the student model to learn the internal decision-making logic of the teacher model, we cannot simply look at the final output; we must delve into the network's internal structure for supervision. Since the teacher and student models have different structures (different numbers of network blocks), it is necessary to define target network blocks.

[0058] In one optional embodiment, the target network block is determined in the following manner: (1) Divide the network blocks in the denoising models of the teacher model and the student model into three stage partitions: early stage network block, middle stage network block and late stage network block, and set the partition ratio corresponding to each stage partition; (2) Multiply each partition ratio by the total number of network blocks in the teacher model and the student model and round down to obtain the index set of the target network blocks selected in the teacher model and the student model respectively.

[0059] Existing video generation denoising models typically consist of dozens of stacked network blocks. These network blocks have varying semantic representation capabilities at different depths. This embodiment divides the depth of the denoising model into three stages: early, middle, and late stages, proportionally.

[0060] The early network blocks are located in the shallow layers of the denoising model (close to the input) and are mainly responsible for low-level feature extraction, such as edges, textures, and preliminary contours. The mid-level network blocks are located in the middle layers of the denoising model and are responsible for combining low-level features into mid-level semantic components, such as the shape of the eyes, the three-dimensionality of the nose, and the opening and closing state of the lips. The late network blocks are located in the deep layers of the denoising model (close to the output) and are responsible for generating high-level details and the final realistic textures, such as skin pores, eye highlights, and hair details.

[0061] To achieve semantic alignment between teachers and students, absolute layer numbers (e.g., layer 10) cannot be selected. For example, the teacher model might have 30 layers, while the student model might only have 15 layers. Therefore, a relative depth must be chosen. This embodiment introduces three partitioning ratio parameters: R1, R2, and R3, with values ​​ranging from (0, 1).

[0062] For example, we can set R1=0.3 (at 30% depth), R2=0.6 (at 60% depth), and R3=0.9 (at 90% depth). This choice of relative positions ensures that the teacher model and the student model are compared with layers that have the same semantic stage, rather than layers with the same number of physical layers.

[0063] In this embodiment, the index of the target network block is calculated as follows: Assuming the total number of blocks in the teacher model is N1 and the total number of blocks in the student model is N2, and the index set calculation function is f(R,N)=round(R×N), where round represents rounding to the nearest integer, the target block index set is represented as follows: (1) The teacher model target block index set Idx1={f(R1,N1),f(R2,N1),f(R3,N1)}.

[0064] (2) Student model target block index set Idx2={f(R1,N2),f(R2,N2),f(R3,N2)}.

[0065] For example, suppose the teacher model is a deep 3D Unet model with 32 network blocks, i.e., N1=32. The student model is a lightweight Unet model, with half the layers for speed, resulting in 16 network blocks, i.e., N2=16. The partition ratio is set as: R1=0.3, R2=0.6, R3=0.9.

[0066] The target block index of the teacher model is calculated as follows: Early network block index: round(0.3×32)=round(9.6)=10; Intermediate network block index: round(0.6×32)=round(19.2)=19; Later network block index: round(0.9×32)=round(28.8)=29.

[0067] Therefore, the target block index set Idx1={10,19,29} of the teacher model indicates that the output of the 10th, 19th, and 29th network blocks will be intercepted during the forward propagation of the teacher model.

[0068] The student model target block index is calculated as follows: Early network block index: round(0.3×16)=round(4.8)=5; Intermediate network block index: round(0.6×16)=round(9.6)=10; Later network block index: round(0.9×16)=round(14.4)=14.

[0069] Therefore, the target block index set Idx2={5,10,14} of the student model indicates that the output of the 5th, 10th, and 14th network blocks will be captured during the forward propagation of the student model.

[0070] Through the calculation of the target block index above, a network block mapping relationship between the teacher model and the student model is established, for example: The fifth layer (early stage) of the student model corresponds to the tenth layer (early stage) of the teacher model. The 10th level (mid-term) of the student model corresponds to the 19th level (mid-term) of the teacher model. The 14th layer (later stage) of the student model corresponds to the 29th layer (later stage) of the teacher model.

[0071] This network layer mapping ensures supervision at similar stages of feature evolution, even when models vary in depth. For example, in later stages, the output features of the student model's 14th layer (which contains high-level texture information) are forced to align with the output features of the teacher model's 29th layer.

[0072] During the forward propagation, when the network reaches the index position of the above calculation, it captures and saves the output of that layer: intermediate features and attention map.

[0073] Intermediate features typically refer to the activation values ​​after convolutional or linear layers, with a shape of (B, C, T, H, W), where B represents the batch size; C represents the dimension of the feature vector extracted by this layer, i.e., the number of feature channels; T represents the number of time frames, i.e., the number of frames contained in the video segment on the time axis, i.e., the length of the sequence; H and W represent the resolution of the feature map in the vertical and horizontal directions, i.e., the image height and width, respectively.

[0074] An attention map specifically refers to the attention weight matrix in a cross-attention module. In cross-attention, the query comes from video features, and the key / value pairs come from audio features. The attention map shows which spatial location in the video focuses on which part of the audio. In multi-head attention mechanisms, the shape of the attention map is typically (B, H, N). Q N K B represents the batch size; H represents the number of attention heads, which is somewhat equivalent to the channel dimension and represents the model's ability to focus on features in different subspaces; N Q Indicates the sequence length, corresponding to the number of tokens in the video, typically equal to the number of time frames × spatial height × spatial width; N K This indicates the length of the conditional sequence and the number of tokens in the corresponding audio.

[0075] 203. Align the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model, respectively. Since intermediate features and attention maps may differ in the teacher and student models, the loss cannot be directly calculated. Therefore, a dynamic alignment scheme for intermediate features and attention maps is introduced.

[0076] In an optional embodiment, step 203 above further includes: 2031. Determine the differences between the student model and the teacher-generated model in intermediate features and attention maps, respectively, including differences in channel dimension, spatial dimension and temporal dimension; (1) Difference in channel dimension: In order to pursue high quality, the teacher model usually uses a wider network (i.e., a larger number of feature channels per layer). For example, the teacher model has 640 channels in the middle block, while the student model may use 320 channels to save memory and computation. Without alignment processing, loss calculation cannot be performed.

[0077] (2) Spatial dimension difference: This is due to the different compression rates of the variational autoencoder or the different downsampling rates of the model. For example, the latent space of the teacher model is 60×60, while the latent space of the student model is compressed to 30×30 for the ultimate speed. This results in inconsistencies in the height and width of the feature maps.

[0078] (3) Temporal dimension differences: Although the number of input frames is usually the same, the number of time frames in the intermediate layers may be different due to different temporal downsampling strategies in the network (e.g., the student model may perform temporal downsampling more aggressively). Or, in the attention map, the flattened sequence length may be different due to different spatial resolutions.

[0079] For example, before training begins, the system automatically checks the intermediate features and attention map findings of the student and teacher models: Assuming the intermediate feature shape output by student model network block 5 is [B,320,16,32,32], while the intermediate feature shape output by teacher model network block 10 is [B,640,16,64,64], the difference determination results for the intermediate features are as follows: (1) Batch size: consistent (2) Number of time frames: consistent (16 frames) (3) Number of feature channels: Inconsistent (320 vs 640) -> Alignment required (4) Feature map height: inconsistent (32 vs 64) -> alignment required (5) Feature map width: inconsistent (32 vs 64) -> alignment required Assuming the attention map output by student model network block 5 has the shape [B,8,1024,100] and spans 8 frames on the time axis, while the attention map output by teacher model network block 10 has the shape [B,32,4096,100] and spans 16 frames on the time axis, the difference in attention maps is determined as follows: (1) Batch size: consistent (2) Number of attention heads: Inconsistent (8 vs 32) -> Needs alignment (3) Spatial resolution: Inconsistent (1024 vs 4096) -> Alignment required (4) Frame sequence length: inconsistent (8 vs 16) -> alignment required 2032. If there are differences in the channel dimension of the intermediate features, a convolution operation is used to align the number of feature channels of the student model with that of the teacher model; if there are differences in the spatial dimension or temporal dimension of the intermediate features, adaptive pooling is used to align the size of the intermediate features of the student model or the frame sequence length with that of the teacher model. In this embodiment, if there are differences in the channel dimensions of intermediate features, the convolution alignment principle is used to align the number of feature channels. If the number of channels C1 of the student feature is less than the number of channels C2 of the teacher feature, a 1×1 convolutional layer can be used to increase the dimensionality of the student feature. A 1×1 convolution can linearly combine feature channels without changing the spatial and temporal dimensions, thereby mapping the feature channels from C1 to C2, making them consistent in channel dimension.

[0080] In this embodiment, if the feature map size of the student model differs from that of the teacher model, adaptive pooling is used for alignment. Adaptive pooling can adjust input features of any size to a specified target size while preserving the main information of the features.

[0081] 2033. If there is a difference in the channel dimension of the attention map, a convolution operation is used to align the number of attention heads of the student model with that of the teacher model; if there is a difference in the spatial dimension or temporal dimension of the attention map, adaptive pooling is used to align the size of the attention map of the student model or the frame sequence length with that of the teacher model.

[0082] In lightweight design, student models often reduce the number of attention heads to minimize computational overhead. For example, a teacher model might use 32 attention heads to capture complex semantic relationships, while a student model might only use 8. In this embodiment, if there are differences in the channel dimensions of the attention maps, a convolution operation is used to expand the number of attention heads in the student model to the same number as the teacher model.

[0083] This embodiment employs adaptive pooling to align the attention map in both spatial and temporal dimensions, specifically including: (1) Spatial dimension alignment Assuming the student model's attention map is spatially based, adaptive pooling can be used for alignment, with the output size specified to be the same as the teacher model's. Adaptive pooling automatically calculates the stride and kernel size, smoothly transforming the 32×32 map to 64×64. This is essentially an upsampling process (implemented through interpolation), mapping the sparse attention distribution onto a dense grid.

[0084] (2) Time-series alignment If the student model's attention map is already a flattened two-dimensional matrix, adaptive pooling can be used for alignment. The length of the student model's output frame sequence is set to the length of the teacher model's frame sequence. This way, the attention weights of the student model at each position are assigned to the corresponding finer-grained positions in the teacher model.

[0085] 204. Based on the aligned intermediate features and attention map, calculate the feature distillation loss, attention distillation loss, true output alignment loss, and final output distillation loss, and sum them by weight to obtain the total loss; This step constructs multi-task learning objectives, thereby constraining the student model from different levels and improving the model's learning accuracy.

[0086] In an optional embodiment, step 204 above further includes: 2041. Calculate the intermediate feature difference between the aligned student model and the teacher model at the target network block, and use it as the feature distillation loss; Calculate the difference between aligned student median features and teacher median features. Commonly used formulas are mean squared error or smoothed L1 Loss function.

[0087] Characteristic distillation loss (L) feat This is used to force the student model to extract feature distributions in the intermediate layers as closely as possible to the teacher model. This helps the student model learn the teacher model's semantic understanding of image content.

[0088] 2042. Calculate the difference in attention maps between the aligned student model and the teacher model at the target network block, and use it as the attention distillation loss; Calculate the difference between the aligned student attention map and the teacher attention map. Since attention maps are usually normalized and exhibit probability distribution characteristics, KL divergence is often used.

[0089] Attention distillation loss (L attn This is used to transfer the focus of knowledge learned by the model, enabling the student model to learn the cross-modal alignment strategy of the teacher model. Even with a small student model capacity, it can learn to focus on the correct region at the right time. For example, when a closed-mouth sound appears in the audio, the teacher model's attention map highlights the mouth area. Through attention distillation loss, the student model is also forced to learn to focus on the mouth at this moment. This directly solves the problems of audio-visual synchronization and motion naturalness.

[0090] 2043. Calculate the difference between the final output of the student model after alignment and the real video data, and use it as the alignment loss of the real output; Calculate the difference between the video predicted by the student model and the real video data.reel The most basic supervisory signal is to ensure that the results generated by the model are real and to avoid the illusions or errors that may be produced by over-imitating the teacher model.

[0091] 2044. Calculate the difference between the final output of the student model and the final output of the teacher model after alignment, and use it as the final output difference loss; Calculate the difference between the final output of the student model and the final output of the teacher model. Final output difference loss (L...) final The goal is to make the final result of the student model as close as possible to the teacher model. This is output layer distillation, which complements feature and attention distillation.

[0092] 2045. Assign weight coefficients to the feature distillation loss, the attention distillation loss, the true output alignment loss, and the final output distillation loss, and then perform a weighted summation to obtain the total loss.

[0093] Total loss Ltotal = λ1 * L feat +λ2*L attn +λ3*L real +λ4*L final .

[0094] Here, λ is the weighting coefficient, which needs to be adjusted based on the validation set results. Typically, L... real The weight of L is relatively large to ensure authenticity, while L attn The weight is initially small and then gradually increased for fine-tuning.

[0095] 205. Based on the total loss, update the parameters of the student model until the model converges, and obtain the trained video generation model.

[0096] This step is a closed-loop execution step in model optimization. By continuously updating the parameters of the student model, the student model learns the distribution of the teacher model until the model converges, and then the trained video generation model can be obtained.

[0097] In an optional embodiment, step 205 further includes: 2051. Based on the total loss, calculate the gradient of each parameter in the student model; 2052. The optimizer is used to update the parameters of the student model based on the gradient; 2053. During the training iteration process, the proportion of the synthetic video dataset in the training dataset is dynamically adjusted so that the proportion decreases as the training progresses; 2054. Monitor the trend of loss changes and determine whether the model has converged; 2055. When the change in loss is less than the preset threshold or the maximum number of iterations is reached, the model is determined to have converged, training is stopped, and the trained video generation model is obtained.

[0098] (1) Gradient calculation and backpropagation: Based on the calculated total loss, the gradient of the total loss with respect to each learnable parameter (weight and bias) in the student model is calculated using the chain rule automatic differentiation mechanism. The parameters of the teacher model are all frozen, therefore no gradient-based updates are required.

[0099] (2) Parameter update: Use an optimizer (such as Adam or AdamW) to update the parameters based on the calculated gradients.

[0100] (3) Dynamically adjust the proportion of data In the initial training phase: Student models are randomly initialized, resulting in a significant difference in distribution from real data. Directly learning from real data is difficult to converge and prone to getting stuck in local optima. At this point, the proportion of synthetic data (teacher model output) is increased (e.g., 90% synthetic, 10% real). Because synthetic data has a smooth distribution and closely resembles the teacher's data, student models can quickly get started and learn basic video generation capabilities and the distribution of the teacher model.

[0101] In the later stages of training: As the student model gradually learns the distribution of the teacher model, continuing to use a large amount of synthetic data may amplify the teacher's flaws and prevent the model from learning the details of real data. At this point, linearly reduce the proportion of synthetic data until it is very low (e.g., 10% synthetic, 90% real). Allow the student model to be fine-tuned towards real data in the final stage to correct the errors of the teacher model and improve accuracy.

[0102] (4) Monitoring and convergence judgment: In each training cycle, the model's performance metrics are evaluated on the validation set at regular intervals, while the loss curve on the training set is monitored. Training stops when the loss (total loss and each sub-loss) no longer decreases significantly, or when the validation set metrics no longer improve or even begin to decrease (overfitting), or when the preset maximum number of iterations (e.g., 100,000 steps) is reached. Finally, the parameters of the best-performing model on the validation set are saved as the final trained student model, i.e., the lightweight video generation model.

[0103] Through the detailed student model training process described above, this embodiment distills a large video generation model that is too large to run in real time into a lightweight, high-quality student model, laying the foundation for subsequent real-time live streaming applications.

[0104] The video generation model construction method in the embodiments of the present invention has been described above. The video generation model construction apparatus in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 3 One embodiment of the video generation model construction device in this invention includes: Input module 301 is used to acquire training data and input the training data into the teacher model and the student model to be trained; Extraction module 302 is used to extract intermediate features and attention maps of the teacher model and the student model at their respective selected target network blocks; Alignment module 303 is used to align the intermediate features and attention map of the student model with the intermediate features and attention map of the teacher model, respectively. The loss module 304 is used to calculate the feature distillation loss, attention distillation loss, true output alignment loss and final output distillation loss based on the aligned intermediate features and attention map, and then sum them up to obtain the total loss. The update module 305 is used to update the parameters of the student model based on the total loss until the model converges, thus obtaining a trained video generation model.

[0105] In an optional embodiment, the input module 301 is further specifically used for: Obtain a real video dataset, and select high-quality images from the real video dataset as the first frame image, and slice the corresponding audio data from the real video data containing the first frame image; The first frame image and the audio data are input into the teacher model to generate a synthetic video dataset; The training dataset is obtained by mixing the real video dataset and the synthetic video dataset; The first frame image is associated with the corresponding audio data and video data to form training data, wherein the first frame image and the audio data are used as input conditions, and the video data is used as noise input.

[0106] In an optional embodiment, both the teacher model and the student model include a denoising model, wherein the number of network blocks in the denoising model of the student model is less than the number of network blocks in the denoising model of the teacher model; the network blocks in the denoising models of both the teacher model and the student model each include three stage partitions: early stage network blocks, mid-stage network blocks, and late stage network blocks, with each stage partition corresponding to a different partition ratio; the index set of the target network blocks selected in each of the teacher model and the student model is related to the partition ratio and the total number of network blocks in the teacher model and the student model.

[0107] In an optional embodiment, the alignment module 303 is specifically used for: Determine the differences between the student model and the teacher-generated model in intermediate features and attention maps, respectively, including differences in channel dimension, spatial dimension and temporal dimension; If there are differences in the channel dimension of the intermediate features, a convolution operation is used to align the number of feature channels of the student model with that of the teacher model; if there are differences in the spatial dimension or temporal dimension of the intermediate features, adaptive pooling is used to align the size of the intermediate features of the student model or the frame sequence length with that of the teacher model. If there are differences in the channel dimension of the attention map, a convolution operation is used to align the number of attention heads of the student model with that of the teacher model; if there are differences in the spatial or temporal dimensions of the attention map, adaptive pooling is used to align the size of the attention map of the student model or the frame sequence length with that of the teacher model.

[0108] In an optional embodiment, the loss module 304 is specifically used for: The intermediate feature differences between the aligned student model and the teacher model at the target network block are calculated and used as the feature distillation loss. The difference in attention maps between the aligned student model and the teacher model at the target network block is calculated and used as the attention distillation loss. The difference between the final output of the student model after alignment and the real video data is calculated and used as the alignment loss of the real output. The difference between the final output of the aligned student model and the final output of the teacher model is calculated and used as the final output difference loss. The feature distillation loss, the attention distillation loss, the true output alignment loss, and the final output distillation loss are assigned weight coefficients and then summed in a weighted manner to obtain the total loss.

[0109] In an optional embodiment, the update module 305 is specifically used for: Based on the total loss, calculate the gradient of each parameter in the student model; The optimizer updates the parameters of the student model based on the gradient; During the training iteration, the proportion of the synthetic video dataset in the training dataset is dynamically adjusted so that the proportion decreases as the training progresses. Monitor the trend of loss changes and determine whether the model has converged; When the change in loss is less than a preset threshold or the maximum number of iterations is reached, the model is considered to have converged, training is stopped, and a well-trained video generation model is obtained.

[0110] Since the embodiments of the device part correspond to the embodiments of the above method, the description of the video generation model construction device provided by the present invention should refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above video generation model construction method.

[0111] above Figure 3 The video generation model construction device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The computer device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0112] Figure 4 This is a schematic diagram of the structure of a computer device 500 provided in an embodiment of the present invention. The computer device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the computer device 500.

[0113] Computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0114] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the video generation model construction method and / or video generation method in the above embodiments. The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, the computer-readable storage medium storing instructions, which, when executed on a computer, cause the computer to perform the steps of the video generation model construction method and / or video generation method.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a video generation model, characterized in that, The video generation model construction method includes: Acquire training data and input the training data into the teacher model and the student model to be trained; Extract the intermediate features and attention maps of the teacher model and the student model at their respective selected target network blocks; Align the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model respectively. Based on the aligned intermediate features and attention map, the feature distillation loss, attention distillation loss, true output alignment loss and final output distillation loss are calculated and weighted to obtain the total loss; Based on the total loss, the parameters of the student model are updated until the model converges, resulting in a trained video generation model.

2. The video generation model construction method according to claim 1, characterized in that, The acquisition of training data includes: Obtain a real video dataset, and select high-quality images from the real video dataset as the first frame image, and slice the corresponding audio data from the real video data containing the first frame image; The first frame image and the audio data are input into the teacher model to generate a synthetic video dataset; The training dataset is obtained by mixing the real video dataset and the synthetic video dataset; The first frame image is associated with the corresponding audio data and video data to form training data, wherein the first frame image and the audio data are used as input conditions, and the video data is used as noise input.

3. The video generation model construction method according to claim 1, characterized in that, Both the teacher model and the student model include a denoising model, with the number of network blocks in the denoising model of the student model being less than the number of network blocks in the denoising model of the teacher model. The network blocks in the denoising models of both the teacher model and the student model each include three stage partitions: early stage network blocks, mid-stage network blocks, and late stage network blocks, with each stage partition corresponding to a different partition ratio. The index set of the target network blocks selected in each of the teacher model and the student model is related to the partition ratio and the total number of network blocks in the teacher model and the student model.

4. The video generation model construction method according to any one of claims 1-3, characterized in that, The step of aligning the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model includes: Determine the differences between the student model and the teacher-generated model in intermediate features and attention maps, respectively, including differences in channel dimension, spatial dimension and temporal dimension; If there are differences in the channel dimension of the intermediate features, a convolution operation is used to align the number of feature channels of the student model with that of the teacher model; if there are differences in the spatial dimension or temporal dimension of the intermediate features, adaptive pooling is used to align the size of the intermediate features of the student model or the frame sequence length with that of the teacher model. If there are differences in the channel dimension of the attention map, a convolution operation is used to align the number of attention heads of the student model with that of the teacher model; if there are differences in the spatial or temporal dimensions of the attention map, adaptive pooling is used to align the size of the attention map of the student model or the frame sequence length with that of the teacher model.

5. The video generation model construction method according to claim 1, characterized in that, The process involves calculating feature distillation loss, attention distillation loss, true output alignment loss, and final output distillation loss based on the aligned intermediate features and attention map, and then weighted summing these to obtain the total loss, which includes: The intermediate feature differences between the aligned student model and the teacher model at the target network block are calculated and used as the feature distillation loss. The difference in attention maps between the aligned student model and the teacher model at the target network block is calculated and used as the attention distillation loss. The difference between the final output of the student model after alignment and the real video data is calculated and used as the alignment loss of the real output. The difference between the final output of the aligned student model and the final output of the teacher model is calculated and used as the final output difference loss. The feature distillation loss, the attention distillation loss, the true output alignment loss, and the final output distillation loss are assigned weight coefficients and then summed in a weighted manner to obtain the total loss.

6. The video generation model construction method according to claim 1, characterized in that, The step of updating the parameters of the student model based on the total loss until the model converges to obtain the trained video generation model includes: Based on the total loss, calculate the gradient of each parameter in the student model; The optimizer updates the parameters of the student model based on the gradient; During the training iteration, the proportion of the synthetic video dataset in the training dataset is dynamically adjusted so that the proportion decreases as the training progresses. Monitor the trend of loss changes and determine whether the model has converged; When the change in loss is less than a preset threshold or the maximum number of iterations is reached, the model is considered to have converged, training is stopped, and a well-trained video generation model is obtained.

7. A video generation method, characterized in that, The video generation method includes: When the first video segment is generated, input data is acquired, which includes a reference image and audio data of the target live streaming object; The input data is input into the video generation model to obtain the first video segment, wherein the video generation model is obtained by the video generation model construction method according to any one of claims 1-6; When a non-first video segment is generated, a preset number of frame images are extracted from the end of the previous video segment generated by the video generation model, and new audio data is obtained. The extracted frame images and the acquired new audio data are used as input data for the next round of video generation model and input into the video generation model to obtain the next video segment; Repeat the video generation process to output continuous live video of digital humans.

8. A video generation model construction device, characterized in that, The video generation model construction device includes: An input module is used to acquire training data and input the training data into the teacher model and the student model to be trained. The extraction module is used to extract the intermediate features and attention maps of the teacher model and the student model at their respective selected target network blocks; The alignment module is used to align the intermediate features and attention maps of the student model with the intermediate features and attention maps of the teacher model, respectively. The loss module is used to calculate the feature distillation loss, attention distillation loss, true output alignment loss, and final output distillation loss based on the aligned intermediate features and attention map, and then sum them up to obtain the total loss. An update module is used to update the parameters of the student model based on the total loss until the model converges, thus obtaining a trained video generation model.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the computer device to perform the video generation model construction method as described in any one of claims 1-6, and / or perform the video generation method as described in claim 7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the video generation model construction method as described in any one of claims 1-6, and / or the video generation method as described in claim 7.

Images