DIT-based multi-modal video generation method, apparatus and device, and medium

Abstract

The invention provides a DIT-based multi-modal video generation method and device, equipment and a medium, and relates to the technical field of multi-modal video generation. The method comprises the following steps: extracting audio features of an input audio signal through a Whisper model and a 3D convolutional layer, and fusing the audio features with hidden layer features added with noise to generate audio embedded features; mouth position mask information and first frame reference mask information are obtained, and mask embedding features and first frame embedding features of a mouth area are extracted respectively; performing addition fusion on the hidden layer feature, the audio embedding feature and the mask embedding feature to obtain a first fusion feature, and fusing the first frame embedding feature with an input first frame reference image to obtain a second fusion feature; and inputting the first fusion feature, the second fusion feature and an input prompt text into a pre-trained DIT mouth shape alignment model to generate a video. According to the method, high-quality video generation and precise mouth shape alignment can be realized, and the problems of incoherence and insufficient detail control of video generation of an existing GAN model are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning-based video generation, and in particular to a DIT-based multimodal video generation method, apparatus, device, and medium. Background Art

[0002] With the rapid development of artificial intelligence and computer vision, deep learning-based video generation technology has been widely applied in various fields. In particular, cutting-edge technologies based on the fusion of diffusion models and the Transformer architecture (DIT, Diffusion and Transformer) have demonstrated outstanding performance in both text-based and image-based video tasks. These large-scale models not only generate videos with good continuity, but also exhibit rich detail, vibrant colors, and cinematic quality.

[0003] However, existing technologies still have significant shortcomings in the specific task of lip alignment. Traditional lip alignment models based on generative adversarial networks (GANs) rely primarily on single-frame synthesis and lack the ability to model temporal information. This results in incoherence between generated video frames, and the overall video quality is far inferior to that of large DIT-based models. Furthermore, videos generated by GAN models are highly dependent on the original video content. They cannot directly generate a video from a simple image, nor can they flexibly control the generated video content through conditional information such as prompt words. These limitations significantly restrict the adaptability and controllability of GAN models in practical applications.

[0004] Therefore, how to achieve high-quality lip alignment using a large DIT-based model has become a technical problem that needs to be solved urgently. In view of this, this application is specially proposed. Summary of the Invention

[0005] The present invention aims to provide a DIT-based multimodal video generation method, device, equipment and medium to address the shortcomings of existing lip alignment models in video generation, such as the lack of timing information leading to discontinuity between generated video frames, and the high dependence on the original video content, making it impossible to directly generate high-quality videos through a single image or prompt word.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions: A multimodal video generation method based on DIT, comprising: S1, obtains the input audio signal and extracts audio features through the Whisper model and 3D convolution layer; S2, fusing the audio feature with the latent feature to which noise is added, to generate an audio embedding feature having the same dimension as the latent feature; S3, obtain the mouth position mask information and the first frame reference mask information, and extract the mask embedding features and first frame embedding features of the mouth area respectively; S4, additively fusing the latent layer feature, the audio embedding feature, and the mask embedding feature to obtain a first fused feature, and fusing the first frame embedding feature with the input first frame reference image to obtain a second fused feature; S5: Input the first fusion feature, the second fusion feature, and the input prompt text into a pre-trained DIT lip alignment model to generate a video.

[0007] Preferably, the S2 is specifically: The audio features are fused with the latent features with added noise by element-by-element addition to generate audio embedding features, which are expressed as: ; ; in, embedding features for the audio; is the audio feature; is the latent feature with noise added; is the input audio signal; is the Whisper model function; It is a 3D convolution layer operation.

[0008] Preferably, the S3 is specifically: According to the mouth position mask information, obtain the mouth position mask sequence : ; Among them, single frame mask ,0 represents the non-mouth area, and 1 represents the mouth area; is the number of frames, For high, For width; Mask sequence for mouth position Feature extraction is performed through the 2D convolution layer, and the mask embedding feature of the mouth area is output. : ; in, It is a 2D convolution layer operation; is a real number; is the batch number; is the number of feature channels; Will The spatial position information of is encoded as an embedded feature that matches the dimension of the latent layer feature; According to the first frame reference mask information, obtain the first frame reference mask , to mark the mouth position of the first frame, and extract the first frame embedding features through the 2D convolution layer : ; in, , represents the first frame feature.

[0009] Preferably, the first fusion feature is fused by additive fusion to associate the audio information with the mouth area mask, and the constraint model focuses on the correspondence between the mouth area and the audio, which is expressed as: ; in, is the first fusion feature; is the latent feature with noise added; embedding features for the audio; Embed features for masks.

[0010] Preferably, the process of acquiring the second fusion feature is: The first frame reference image is used to extract image embedding features through a 2D convolutional layer; After expanding the first frame embedding feature to the same dimension as the number of frames of the generated target video, it is added element by element to the image embedding feature to obtain the second fusion feature.

[0011] Preferably, the pre-trained DIT lip alignment model is a diffusion model based on an attention mechanism, and is trained using a multi-scale progressive training method, specifically: In the first training phase, the object detection model extracts the coordinates of the rectangular frame of the mouth area in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames, using a fixed step length. The smoothed mouth area is cropped and unified to the target size before being synthesized into a new video. This video is then fed into the model for training to strengthen the association between the mouth area and the audio. The new video only contains the mouth area. Second stage training: The object detection model extracts the coordinates of the rectangular frame of the face area in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames with a fixed step length. The smoothed face area is cropped and unified to the target size before being synthesized into a second new video. This video is then fed into the model for training and fine-tuning. The second new video only contains the face area. The third training phase: The object detection model extracts the coordinates of the rectangular outer frames of the human body region in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames with a fixed length as the step size. The smoothed human body region is cropped and unified to the target size before being synthesized into a third new video. This video is then input into the model for training and fine-tuning. The third new video only contains the human body region. The fourth stage of training: the original video is resized and fed into the model trained in the three stages to continue training, in order to optimize global consistency and enable the model to take into account both local lip alignment and overall video quality; When the training reaches the evaluation standard, the trained DIT lip alignment model is obtained.

[0012] Preferably, the coordinate smoothing process is performed once for each adjacent rectangular frame with a fixed length as the step size, and the mean smoothing method is used for coordinate smoothing, and the coordinates of the first and last frames are retained or the symmetrical filling smoothing method is used to obtain the first and last frames of the video to reduce jitter.

[0013] The present invention also provides a DIT-based multimodal video generation device, comprising: The audio feature extraction unit is used to obtain the input audio signal and extract audio features through the Whisper model and 3D convolution layer; an audio embedding unit, configured to fuse the audio features with the latent features to which noise is added, so as to generate audio embedding features having the same dimension as the latent features; The mask unit is used to obtain the mouth position mask information and the first frame reference mask information, and respectively extract the mask embedding features and the first frame embedding features of the mouth area; a fusion unit, configured to perform additive fusion on the latent layer feature, the audio embedding feature, and the mask embedding feature to obtain a first fused feature, and to fuse the first frame embedding feature with the acquired first frame reference image to obtain a second fused feature; A generating unit is used to input the first fusion feature, the second fusion feature and the input prompt text into a pre-trained DIT lip alignment model to generate a video.

[0014] The present invention also provides a DIT-based multimodal video generation device, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the DIT-based multimodal video generation method as described above.

[0015] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a DIT-based multimodal video generation method as described above is implemented.

[0016] In summary, compared with the prior art, the present invention has the following beneficial effects: The present invention solves the problems of video frame incoherence and insufficient lip alignment accuracy in the existing technology through audio injection and multi-scale training strategies, thereby improving the performance of image-based video models in lip alignment tasks.

[0017] This paper uses audio injection, mask encoding, and training strategies to achieve precise alignment of the character's lip movements in the video with the input audio: (1) Audio injection method: The audio features, mask features and latent features are fused through addition operations to enhance the correlation between audio and video content.

[0018] (2) Mask encoding: The mouth position mask feature and the first frame reference mask feature provide the model with clear mouth position information, enabling the model to accurately locate the mouth area and establish a connection with the input audio.

[0019] (3) Training strategy: A multi-scale progressive training strategy is adopted to gradually enhance the model's learning ability for the mouth area.

[0020] In summary, the video generated by the present invention has high-quality image quality and precise lip alignment effect, which solves the discontinuity and low-quality generation problems existing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a DIT-based multimodal video generation method provided in Example 1.

[0023] Figure 2 This is a diagram of the overall model structure of a DIT-based multimodal video generation method provided in Example 1.

[0024] Figure 3 A structural diagram of a DIT-based multimodal video generation device provided in Example 2.

[0025] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] Example 1 Embodiment 1 of the present invention provides a DIT-based multimodal video generation method, which can be implemented by a DIT-based multimodal video generation device (hereinafter referred to as a generation device), and in particular, executed by one or more processors in the generation device.

[0028] In this embodiment, the generating device may be an electronic device equipped with a processor, which carries a computer program of the DIT-based multimodal video generation method and can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0029] like Figure 1 As shown, a DIT-based multimodal video generation method includes steps S1 to S5.

[0030] S1 obtains the input audio signal and extracts audio features through the Whisper model and 3D convolution layer.

[0031] In this step, we obtain the input audio signal and feed it into the Whisper open-source model. The Whisper model uses an encoder-decoder architecture to perform time-frequency analysis and semantic encoding on the audio, outputting initial audio features.

[0032] Design a 3D convolutional layer. For example, assume that the convolutional layer parameters are: kernel size k=(3,3,3), stride s=(1,1,1), and padding p=(1,1,1). Input the initial audio features into this layer, perform channel expansion (for example, processing into 5120 channels) and align the dimensions, which can be expressed as: ; in, is the audio feature; is the latent feature with noise added; is the input audio signal; is the Whisper model function; It is a 3D convolution layer operation.

[0033] S2: Fusing the audio features with the latent features to which noise is added to generate audio embedding features having the same dimension as the latent features.

[0034] The audio features are fused with the latent features with added noise by element-by-element addition to achieve the fusion of audio information and latent features and generate audio embedding features, whose expression is: ; in, is the audio embedding feature.

[0035] Latent features The added noise may be Gaussian noise, such as Gaussian noise with a mean of 0 and a standard deviation of 0.1.

[0036] S3, obtain the mouth position mask information and the first frame reference mask information, and extract the mask embedding features and first frame embedding features of the mouth area respectively.

[0037] According to the mouth position mask information, obtain the mouth position mask sequence : ; Among them, single frame mask ,0 represents the non-mouth area, and 1 represents the mouth area; is the number of frames, For high, For width; Mask sequence for mouth position Feature extraction is performed through the 2D convolution layer, and the mask embedding feature of the mouth area is output. : ; in, It is a 2D convolution layer operation; is a real number; is the batch number; is the number of feature channels; Will The spatial position information of is encoded as an embedded feature that matches the dimension of the latent layer features.

[0038] According to the first frame reference mask information, obtain the first frame reference mask , to mark the mouth position of the first frame, and extract the first frame embedding features through the 2D convolution layer : ; in, , represents the first frame feature.

[0039] The first frame embedding feature can be expanded later dimensions to fit the video sequence.

[0040] S4, additively fusing the latent layer feature, the audio embedding feature and the mask embedding feature to obtain a first fused feature, and fusing the first frame embedding feature with the input first frame reference image to obtain a second fused feature.

[0041] The first fusion feature is fused by addition to associate the audio information with the mouth area mask, constraining the model to focus on the correspondence between the mouth area and the audio. Its expression is: ; in, is the first fusion feature; is the latent feature with noise added; embedding features for the audio; Embed features for masks.

[0042] The process of obtaining the second fusion feature is as follows: The first frame reference image Extract image embedding features through 2D convolutional layers; After the first frame embedding feature is expanded to the same dimension as the number of frames of the generated target video, it is added element by element to the image embedding feature to ensure that the first frame feature is compatible with the video sequence dimension to obtain the second fusion feature : ; Expands the operation for the dimension.

[0043] S5: Input the first fusion feature, the second fusion feature, and the input prompt text into a pre-trained DIT lip alignment model to generate a video.

[0044] In this step, the first fusion feature , the second fusion feature The pre-trained DIT lip alignment model is fed with the input prompt text (such as "virtual anchor explains technology news"). The model generates the final video output through a multimodal attention mechanism and diffusion deformation transformation.

[0045] The pre-trained DIT lip alignment model is trained using a multi-scale progressive training method, specifically: First stage training: Use an object detection model (such as YOLO) to extract the coordinates of the rectangular frame of the mouth area in each frame of the original video. Smooth the coordinates of every adjacent rectangular frame (e.g., 5) using a fixed step length (e.g., a fixed length of 5). The smoothed mouth area is cropped and unified to the target size (e.g., 512*512), then synthesized into a first new video. This is input into the model for training to strengthen the association between the mouth area and the audio. The first new video only contains the mouth area. Second stage training: The object detection model extracts the coordinates of the rectangular frame of the face area in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames with a fixed step length. The smoothed face area is cropped and unified to the target size before being synthesized into a second new video. This video is then fed into the model for training and fine-tuning. The second new video only contains the face area. The third training phase: The object detection model extracts the coordinates of the rectangular outer frames of the human body region in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames with a fixed length as the step size. The smoothed human body region is cropped and unified to the target size before being synthesized into a third new video. This video is then input into the model for training and fine-tuning. The third new video only contains the human body region. Fourth stage training: The original video is resized (e.g., to a resolution of 512*512) and then fed into the model trained in the three stages to continue training, in order to optimize global consistency and enable the model to take into account both local lip alignment and overall video quality. When the training reaches the evaluation standard, the trained DIT lip alignment model is obtained.

[0046] With a fixed length as the step size, coordinate smoothing is performed once for every several adjacent rectangular frames. The mean smoothing method can be used for coordinate smoothing, and the coordinates of the first and last frames can be retained or the symmetrical filling smoothing method can be used to obtain the first and last frames of the video to reduce jitter.

[0047] When training at each stage, the length of the step and the number of adjacent rectangular boxes can be the same or different, depending on user needs and are not limited here.

[0048] This multi-stage training strategy ensures that the model can gradually learn the details of the mouth area from local to global, solving the discontinuity problem when using the GAN network to generate lip shapes.

[0049] like Figure 2As shown, in practical applications, users can use the present invention to generate a high-quality video by inputting a first-frame reference image (image_embedding) and an audio signal (audio). For example, if a user inputs a frontal photo of a person and a voice recording, the model first extracts audio features using the Whisper model and the audio encoding module (audio_embedding) and processes them into the same dimension as the latent feature X. Mouth position mask information (masks) and first-frame reference mask information (first_mask) are annotated with mouth position information using the mouth position mask embedding module (masks_embedding) and the first-frame mask embedding module (first_masks_embedding), respectively. Subsequently, the audio features, mouth position mask embedding features, and latent features are fused through an addition operation and fed into the DIT model for processing along with the first-frame embedding features, the input first-frame reference image (image_embedding), and the prompt text (prompt_embedding). The resulting video (video) is precisely aligned with the audio and lip movements. The resulting video not only has movie-quality image quality but also maintains coherence between video frames and supports controlling video content through prompt word information.

[0050] The audio encoding module (audio_embedding) consists of 3D convolutions to process the corresponding whisper features into the same dimensionality as the latent features. The mouth position mask embedding features extracted by the mouth position mask embedding module (masks_embedding) are added to the latent features X to provide a working area for the audio encoding module.

[0051] In summary, compared with the prior art, the present invention has the following beneficial effects: First, this invention uses audio injection to fuse audio features, mask features, and latent features through an additive operation, enhancing the relevance of audio and video content. Second, through mask embedding and first-frame embedding features, the model is provided with clear mouth position information, enabling it to accurately locate the mouth region and establish a connection with the input audio. Finally, an innovative training strategy employs a multi-scale progressive training strategy to gradually enhance the model's ability to learn the mouth region.

[0052] The video generated by the present invention has high-quality image quality and precise lip alignment effect, solving the problems of discontinuity and low-quality generation in the prior art.

[0053] Example 2 like Figure 3 As shown, the second embodiment of the present invention further provides a multimodal video generation device based on DIT, including: The audio feature extraction unit is used to obtain the input audio signal and extract audio features through the Whisper model and 3D convolution layer; an audio embedding unit, configured to fuse the audio features with the latent features to which noise is added, so as to generate audio embedding features having the same dimension as the latent features; The mask unit is used to obtain the mouth position mask information and the first frame reference mask information, and respectively extract the mask embedding features and the first frame embedding features of the mouth area; a fusion unit, configured to perform additive fusion on the latent layer feature, the audio embedding feature, and the mask embedding feature to obtain a first fused feature, and to fuse the first frame embedding feature with the acquired first frame reference image to obtain a second fused feature; A generating unit is used to input the first fusion feature, the second fusion feature and the input prompt text into a pre-trained DIT lip alignment model to generate a video.

[0054] Example 3 The third embodiment of the present invention also provides a DIT-based multimodal video generation device, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the DIT-based multimodal video generation method as described above.

[0055] Example 4 The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the DIT-based multimodal video generation method as described above is implemented.

[0056] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of a code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0057] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0058] If the functions are implemented in the form of software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0059] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0060] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0061] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0062] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0063] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multimodal video generation method based on DIT, characterized in that: include: S1, obtains the input audio signal and extracts audio features through the Whisper model and 3D convolution layer; S2, fusing the audio feature with the latent feature to which noise is added, to generate an audio embedding feature having the same dimension as the latent feature; S3, obtain the mouth position mask information and the first frame reference mask information, and extract the mask embedding features and first frame embedding features of the mouth area respectively; S4, additively fusing the latent layer feature, the audio embedding feature, and the mask embedding feature to obtain a first fused feature, and fusing the first frame embedding feature with the input first frame reference image to obtain a second fused feature; S5: Input the first fusion feature, the second fusion feature, and the input prompt text into a pre-trained DIT lip alignment model to generate a video.

2. The DIT-based multimodal video generation method according to claim 1, characterized in that , the S2 is specifically: The audio features are fused with the latent features with added noise by element-by-element addition to generate audio embedding features, which are expressed as: ； ； in, embedding features for the audio; is the audio feature; is the latent feature with noise added; is the input audio signal; is the Whisper model function; It is a 3D convolution layer operation.

3. The DIT-based multimodal video generation method according to claim 1, characterized in that , the S3 is specifically: According to the mouth position mask information, obtain the mouth position mask sequence : ； Among them, single frame mask ,0 represents the non-mouth area, and 1 represents the mouth area; is the number of frames, For high, For width; Mouth position mask sequence Feature extraction is performed through the 2D convolution layer, and the mask embedding feature of the mouth area is output. : ； in, It is a 2D convolution layer operation; is a real number; is the batch number; is the number of feature channels; Will The spatial position information of is encoded as an embedded feature that matches the dimension of the latent layer feature; According to the first frame reference mask information, obtain the first frame reference mask , to mark the mouth position of the first frame, and extract the first frame embedding features through the 2D convolution layer : ； in, , represents the first frame feature.

4. The DIT-based multimodal video generation method according to claim 1, characterized in that The first fusion feature is fused by addition to associate the audio information with the mouth area mask, constraining the model to focus on the correspondence between the mouth area and the audio. Its expression is: ； in, is the first fusion feature; is the latent feature with noise added; embedding features for the audio; Embed features for masks.

5. The DIT-based multimodal video generation method according to claim 1, characterized in that ,The acquisition process of the second fusion feature is as follows: The first frame reference image is used to extract image embedding features through a 2D convolutional layer; After expanding the first frame embedding feature to the same dimension as the number of frames of the generated target video, it is added element by element to the image embedding feature to obtain the second fusion feature.

6. The DIT-based multimodal video generation method according to claim 1, characterized in that The pre-trained DIT lip alignment model is a diffusion model based on the attention mechanism, which is trained using a multi-scale progressive training method, specifically: In the first training phase, the object detection model extracts the coordinates of the rectangular frame of the mouth area in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames, using a fixed step length. The smoothed mouth area is cropped and unified to the target size before being synthesized into a new video. This video is then fed into the model for training to strengthen the association between the mouth area and the audio. The new video only contains the mouth area. Second stage training: The object detection model extracts the coordinates of the rectangular frame of the face area in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames with a fixed step length. The smoothed face area is cropped and unified to the target size before being synthesized into a second new video. This video is then fed into the model for training and fine-tuning. The second new video only contains the face area. The third training phase: The object detection model extracts the coordinates of the rectangular outer frames of the human body region in each frame of the original video. The coordinates are smoothed every several adjacent rectangular frames with a fixed length as the step size. The smoothed human body region is cropped and unified to the target size before being synthesized into a third new video. This video is then input into the model for training and fine-tuning. The third new video only contains the human body region. The fourth stage of training: the original video is resized and fed into the model trained in the three stages to continue training, in order to optimize global consistency and enable the model to take into account both local lip alignment and overall video quality; When the training reaches the evaluation standard, the trained DIT lip alignment model is obtained.

7. The DIT-based multimodal video generation method according to claim 6, characterized in that ,With a fixed length as the step size, every adjacent rectangular frame is smoothed once, and the mean smoothing method is used for coordinate smoothing, and the coordinates of the first and last frames are retained or the symmetrical filling smoothing method is used to obtain the first and last frames of the video to reduce jitter.

8. A multimodal video generation device based on DIT, characterized in that: include: The audio feature extraction unit is used to obtain the input audio signal and extract audio features through the Whisper model and 3D convolution layer; an audio embedding unit, configured to fuse the audio features with the latent features to which noise is added, so as to generate audio embedding features having the same dimension as the latent features; The mask unit is used to obtain the mouth position mask information and the first frame reference mask information, and respectively extract the mask embedding features and the first frame embedding features of the mouth area; a fusion unit, configured to perform additive fusion on the latent layer feature, the audio embedding feature, and the mask embedding feature to obtain a first fused feature, and to fuse the first frame embedding feature with the acquired first frame reference image to obtain a second fused feature; A generating unit is used to input the first fusion feature, the second fusion feature and the input prompt text into a pre-trained DIT lip alignment model to generate a video.

9. A multimodal video generation device based on DIT, characterized in that: The system comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement a DIT-based multimodal video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor of the device where the computer-readable storage medium is located, a DIT-based multimodal video generation method according to any one of claims 1 to 7 is implemented.

Images