Method and apparatus for generating video from images

Abstract

This disclosure provides a method and related equipment for generating image-to-video, relating to the fields of computer vision and image processing technology. The method includes acquiring an original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, driving audio information, and mask image; generating initial noisy video latent variables; fusing the video constraint information, initial noisy video latent variables, and mask image to generate a contextual condition vector; and generating a target video based on the original image and contextual condition vector. This disclosure provides an image-to-video solution that requires no model training and no additional acquisition or construction of training data, effectively reducing data costs and computational overhead, enabling direct generation of target videos from input original images and driving audio information, improving ease of use and deployment efficiency.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and image processing technology, and in particular to a method for generating video from an image, an apparatus for generating video from an image, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Generative artificial intelligence has made significant breakthroughs in the field of visual content creation. It can generate multi-character dialogue videos by driving images with audio. Image-generated video technology can generate coherent and realistic dialogue videos based on input static images and single-channel audio.

[0003] In related technologies, multi-role dialogue videos can be generated through diffusion-based transformer architectures (DiT) or multimodal diffusion self-attention mechanisms. However, these graph-based video generation methods heavily rely on large-scale labeled datasets, resulting in problems such as long dataset construction cycles, high costs, limited coverage of scenarios, poor model versatility, low deployment and iteration efficiency, and poor generalization and scalability.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a method and related device for generating image-to-video, which at least to some extent overcomes the problems of long dataset construction cycles, poor model versatility, low deployment and iteration efficiency, and poor generalization and scalability in image-to-video generation methods in related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a method for generating video from an image is provided, comprising: acquiring an original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, the driving audio information, and the mask image; generating initial noisy video latent variables; fusing the video constraint information, the initial noisy video latent variables, and the mask image to generate a contextual condition vector; and generating a target video based on the original image and the contextual condition vector.

[0008] In some embodiments of this disclosure, generating video constraint information based on the original image, the driving audio information, and the mask image includes: extracting features from the driving audio information and the original image to obtain audio-image fusion features, wherein the audio-image fusion features are used to characterize the correspondence between audio embedding features and image global embedding features; and generating the video constraint information based on the audio-image fusion features, the original image, and the mask image.

[0009] In some embodiments of this disclosure, the step of extracting features from the driving audio information and the original image to obtain audio-image fusion features includes: extracting features from the driving audio information using a multimodal parsing unit to obtain audio embedding features; encoding visual features of the original image using the multimodal parsing unit to obtain image global embedding features; and performing cross-modal alignment between the audio embedding features and the image global embedding features to obtain the audio-image fusion features.

[0010] In some embodiments of this disclosure, the video constraint information includes structured prompt text; wherein, generating the video constraint information based on the audio-image fusion features, the original image, and the mask image includes: using a prompt engineering unit to perform fusion processing on the original image, audio intent, and the mask image according to a preset prompt template to generate the structured prompt text.

[0011] In some embodiments of this disclosure, generating the initial noisy video latent variable includes: generating the initial noisy video latent variable based on the number of frames of the target video, the size of the original image, and the number of channels, wherein the initial noisy video latent variable conforms to a Gaussian distribution.

[0012] In some embodiments of this disclosure, fusing the video constraint information, the initial noisy video latent variables, and the mask image to generate a context condition vector includes: using a cross-attention mechanism to map the video constraint information, the initial noisy video latent variables, and the mask image to a target latent space, and performing splicing and fusion processing to obtain the context condition vector.

[0013] In some embodiments of this disclosure, the mask image includes a first region and a second region; wherein, generating a target video based on the original image and the context condition vector includes: mapping the original image to the target latent space using a visual diffusion self-attention architecture, superimposing it with the initial noisy video latent variables to obtain a video to be driven; calculating the attention weights of each layer in the video to be driven; and generating the target video based on the attention weights and the context condition vector.

[0014] In some embodiments of this disclosure, generating the target video based on the attention weight and the context condition vector includes: for a first target region located within the first region and whose attention weight is greater than a preset weight threshold, performing feature updates and dynamic driving on the first target region based on the context condition vector; for a second target region located in the second region or located in the first region and whose attention weight is less than the preset weight threshold, not performing feature updates and dynamic driving on the second target region, retaining the features of the original image to generate the target video.

[0015] In some embodiments of this disclosure, calculating the attention weight of each layer in the video to be driven includes: calculating the attention weight of the first region; and setting the attention weight of the second region to 0.

[0016] According to another aspect of this disclosure, an image-to-video apparatus is also provided, comprising: a data acquisition module for acquiring an original image and driving audio information; a mask generation module for generating a mask image based on the original image; a constraint generation module for generating video constraint information based on the original image, the driving audio information, and the mask image; a noise generation module for generating initial noise video latent variables; a context generation module for fusing the video constraint information, the initial noise video latent variables, and the mask image to generate a context condition vector; and a video generation module for generating a target video based on the original image and the context condition vector.

[0017] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described image-to-video method by executing the executable instructions.

[0018] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the above-described image-to-video method.

[0019] According to another aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program or computer instructions, the computer program or computer instructions being loaded and executed by a processor to enable a computer to implement the above-described image-to-video method.

[0020] In this embodiment, the following steps are taken: acquiring the original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, driving audio information, and mask image; generating initial noisy video latent variables; fusing the video constraint information, initial noisy video latent variables, and mask image to generate a contextual condition vector; and generating the target video based on the original image and contextual condition vector. The image-to-video solution provided in this disclosure eliminates the need for model training and additional data acquisition or construction, effectively reducing the cost of data acquisition and computing power deployment, simplifying the implementation process of image-to-video, and enabling the direct generation of the target video by inputting only the original image and driving audio information. This significantly improves the usability and practical deployment efficiency of the image-to-video solution, providing strong support for the large-scale application of image-to-video technology.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1 The diagram illustrates a flowchart of a video generation method provided in an embodiment of this disclosure.

[0024] Figure 2 The diagram shows a flowchart of a video constraint information generation method provided in an embodiment of this disclosure.

[0025] Figure 3 The flowchart illustrates an audio-image fusion feature determination method provided by an embodiment of this disclosure.

[0026] Figure 4 This diagram illustrates another video constraint information generation method provided by an embodiment of the present disclosure.

[0027] Figure 5 This illustration shows a flowchart of another image-generated video method provided by an embodiment of this disclosure.

[0028] Figure 6 This illustration shows a flowchart of another image-to-video method provided in an embodiment of the present disclosure.

[0029] Figure 7 A flowchart of a target video generation method provided in an embodiment of this disclosure is shown.

[0030] Figure 8 This diagram illustrates the structure of a video-generating device according to an embodiment of the present disclosure.

[0031] Figure 9 A structural block diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0033] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the following detailed description is provided in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0035] Generative artificial intelligence has made significant breakthroughs in the field of visual content creation. It can generate multi-character dialogue videos by driving images with audio. Image-generated video technology can generate coherent and realistic dialogue videos based on input static images and single-channel audio. Image-generated video technology can use single or multiple reference images of people as the base reference image for video generation, and use multiple dialogue audio as the driving signal to generate dialogue videos with multiple characters in the same frame, lip movements, expressions, and postures that are precisely synchronized with the audio, and natural interactions between characters. It is widely used in scenarios such as virtual live streaming and online meetings.

[0036] In related technologies, multi-role dialogue videos can be generated through a Diffusion Transformer (DiT) architecture based on diffusion modules or a multimodal diffusion self-attention mechanism. DiT proposes L-ROPE label rotation position encoding and adaptive character localization to achieve precise binding of multi-stream audio to characters, supporting the generation of multi-person dialogue videos from reference images, multiple audio streams, and text commands. Alternatively, an audio-face cross-attention and multi-stream loop processing structure can be used to support arbitrary number of characters driven by different roles, constructing pseudo-multi-person samples for training with single-person data, thus reducing data costs. Based on the multimodal diffusion Transformer, a face-aware audio adapter and latent spatial face mask are proposed to achieve independent audio injection for multiple characters, balancing identity consistency and fine-grained emotional control.

[0037] However, the above-mentioned image-generated video method has the following drawbacks: 1. The above-mentioned image-to-video generation methods all adopt a data-driven and model training technical path, which requires the pre-collection or construction of paired datasets containing multiple characters, multiple dialogue scenes, multiple speakers' audio, and corresponding face / pose / expression / temporal video. Some generation methods also require additional annotations such as character masking, character localization, and dialogue temporal labels, resulting in long dataset construction cycles and high costs. The datasets can only cover a limited number of characters, limited scenes, and limited dialogue styles, and cannot support any new characters, any new scenes, or any combination of dialogues. If there are new characters, changes in scenes, or changes in the layout of multiple people, the dataset must be re-collected or reconstructed, resulting in poor versatility. 2. The aforementioned image-to-video methods are all based on deep learning models, such as speech head generation networks, video diffusion models, and multi-character animation generation models. Deep learning models undergo processes such as model training, fine-tuning or transfer learning for multi-person dialogue scenarios, and mismatch optimization for specific tasks or styles before they can be used. However, training deep learning models relies on computing power, is time-consuming and resource-intensive, and is not user-friendly for ordinary users or non-R&D scenarios. They cannot be used immediately for generation. Once trained, deep learning models have weak generalization ability for scenarios without visible characters, audio, or composition, easily leading to problems such as lip-sync asynchrony, identity distortion, misalignment of multiple characters, and stiff facial expressions. This results in low deployment and iteration efficiency for deep learning models. 3. The above-mentioned image-to-video method relies on the distribution of training data and the learned pattern. The generation process is limited to the distribution covered by the training set. It cannot be directly driven by a single image of a person that has never been seen before, nor can it directly input any number of dialogue audios to achieve plug-and-play multi-person simultaneous dialogue. Multi-role binding, temporal alignment, interactive actions, etc. all rely on the priors learned in the training stage, rather than being completed by direct reasoning. This results in poor robustness and poor generalization ability of the deep learning model in actual use, making it difficult to support open multi-person dialogue with any reference image and any number of audios. 4. The aforementioned image-to-video generation methods require model training and datasets. Deep learning models have complex structures and poor scalability, making them unsuitable for lightweight, edge-based, and rapidly iterative content creation needs. These methods struggle to meet the demands for quickly changing characters, dialogue content, and generating multiple video versions. They also fail to meet the requirements of a simplified workflow for non-professional users and are incompatible with the current needs for digital human content creation and rapid virtual dialogue video generation.

[0038] In summary, the above-mentioned image-to-video methods rely heavily on large-scale labeled datasets, resulting in problems such as long dataset construction cycles and high costs, limited coverage of only certain scenarios, poor model versatility, low deployment and iteration efficiency, and poor generalization and scalability.

[0039] In one scenario, the image-to-video method disclosed herein can be applied to use a single audio dialogue as the driving signal, and one or more reference images with a driving object as the base reference image for video generation, to generate dialogue-based videos with precise synchronization between the lip movements, facial expressions, and postures of a single character and the audio. The driving object can be a person, animal, plant, etc.

[0040] In another scenario, the image-generated video method disclosed herein can also be applied to use multiple dialogue audios as driving signals, and use a single or multiple reference images with multiple driving objects as the reference image for video generation, to generate dialogue videos with multiple characters in the same frame, lip movements, facial expressions, postures and audio precisely synchronized, and natural interactions between characters.

[0041] To at least partially address the aforementioned technical problems, the image-to-video method provided in this disclosure includes: acquiring an original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, driving audio information, and mask image; generating initial noisy video latent variables; fusing the video constraint information, initial noisy video latent variables, and mask image to generate a contextual condition vector; and generating a target video based on the original image and contextual condition vector. This disclosure provides a lightweight, universal, and plug-and-play image-to-video solution that requires no model training or additional data collection or construction. It effectively reduces the cost of data acquisition and computing power deployment, simplifies the image-to-video implementation process, and enables direct generation of target videos using only the original image and driving audio information. This significantly improves the usability and practical deployment efficiency of the image-to-video solution, providing strong support for the large-scale application of image-to-video technology.

[0042] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.

[0043] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0044] First, this disclosure provides a method for generating video from an image, which can be executed by any electronic device with computing capabilities. The image-to-video method provided in this disclosure can also be executed by a server.

[0045] Figure 1 This diagram illustrates a flowchart of a video generation method provided in an embodiment of this disclosure, such as... Figure 1 As shown, the image-to-video method provided in this embodiment mainly includes the following steps: S102, Obtain the original image and driving audio information.

[0046] In some embodiments, the original image may be an image captured by a user through a terminal device and uploaded to a server, or it may be an image obtained from a database. The original image may include at least one driving object, which may include people, animals, plants, etc. In this disclosure, multiple people are used as driving objects for illustration.

[0047] The original image serves as the starting frame for video generation. The format is uniformly RGB three-channel image, i.e., three-channel color image mode. The resolution is normalized to a fixed size according to the system configuration, such as 512×512 or 768×768.

[0048] The driving audio information may include multiple dialogue audio segments, each corresponding to a character. The number of dialogue audio segments may be equal to the number of characters in the original image, or greater than the number of characters in the original image. This disclosure does not impose any specific limitations on this.

[0049] In some embodiments, when a user uploads the driving audio information to the server, they can select the speaking order of the characters in the original image and the correspondence between the characters and the dialogue audio. For example, the original image includes character A and character B, and the dialogue audio includes audio A1, A2, A3 and audio B1, B2, B3. The relationship between the characters and the dialogue audio can be: character A corresponds to audio A1, A2, A3, character B corresponds to audio B1, B2, B3, and character A speaks before character B.

[0050] S104. Generate a mask image based on the original image.

[0051] A mask image, also known as a mask image, is a single-channel binary image used to determine the target region in the original image that drives the audio information, thereby achieving precise region control.

[0052] In one specific implementation, in a single-channel binary image, the main subject region is the driving region and is marked as 1; the non-subject region is the non-driving region and is marked as 0.

[0053] In some embodiments, a video subject detection model can be used to detect and locate video subjects or people in the original image, identify subject region features such as faces, bodies, or other key targets in the image, and output subject location boxes and subject masks to obtain a mask image, thereby improving generation stability. Subject region features are used to strengthen the focus on target regions during target video generation and avoid unreasonable changes in non-target regions, which is key to realizing multi-person dialogue.

[0054] Video subject detection models can be deep learning models, such as Gaussian mixture models, codebook models, etc.

[0055] In some embodiments, the mask image may also be set manually by the user.

[0056] S106. Generate video constraint information based on the original image, driving audio information, and mask image.

[0057] In some embodiments, video constraint information is used to limit constraints such as content, action, region, and style of video generation.

[0058] In some embodiments, video constraint information can be generated by a prompt engineering (PE) unit based on the original image, driving audio information, and mask image.

[0059] Video constraint information can include structured prompt text, such as "Based on the original input image, generate natural, synchronous, and realistic dynamic changes according to the audio rhythm." The structured prompt text does not rely on manual writing but is automatically generated by the system based on the image content and simple prompt words entered by the user.

[0060] S108. Generate initial noisy video latent variables.

[0061] In some embodiments, the initial noise video latent variable refers to the initial input random variable in the image-to-video generation process. As the iterative starting point for the generation of the target video, it can be a set of random noise vectors or tensors that conform to a specific distribution (such as Gaussian distribution) to provide initial, iteratively optimized basic materials for the generation of the target video.

[0062] S110. The video constraint information, initial noisy video latent variables, and mask image are fused to generate a context condition vector.

[0063] In some embodiments, the above fusion refers to mapping features of different modalities and dimensions to the same latent space, concatenating them, and obtaining a context condition vector of a unified dimension.

[0064] The context condition vector is an intermediate feature vector in the graph-generated video generation process. It is fused from video constraint information, initial noisy video latent variables, and mask image. It can integrate all the key information required to generate the target video, including the video constraint logic corresponding to the original image and driving audio information, the diversity provided by the initial noisy video latent variables, and the main region features corresponding to the mask image. As a key input for the generation of the target video, it works in conjunction with the original image to guide the generation of the target video, thereby ensuring the temporal consistency, content accuracy, and detail richness of the video.

[0065] S112. Generate the target video based on the original image and the contextual condition vector.

[0066] In some embodiments, the original image and contextual conditional vector can be processed based on a visual diffusion self-attention architecture to generate a target video. The target video is the final generated image-generated video.

[0067] The visual diffusion self-attention architecture takes the original image and combines it with the mask image, driving audio information and initial noise video latent variables in the context condition vector. It performs audio-driven video generation operations within the driving region of the mask image, gradually generating video frames that match the driving audio information and conform to the region constraints defined by the mask image, and finally outputs the target video.

[0068] In this embodiment, the following steps are taken: acquiring the original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, driving audio information, and mask image; generating initial noisy video latent variables; fusing the video constraint information, initial noisy video latent variables, and mask image to generate a contextual condition vector; and generating the target video based on the original image and contextual condition vector. The image-to-video solution provided in this disclosure eliminates the need for model training and additional data acquisition or construction, effectively reducing the cost of data acquisition and computing power deployment, simplifying the implementation process of image-to-video, and enabling the direct generation of the target video by inputting only the original image and driving audio information. This significantly improves the usability and practical deployment efficiency of the image-to-video solution, providing strong support for the large-scale application of image-to-video technology.

[0069] Figure 2 This diagram illustrates a flowchart of a video constraint information generation method provided by an embodiment of this disclosure. Figure 2 As shown, in some embodiments, the above-described S106 generates video constraint information based on the original image, driving audio information, and mask image, including: S202. Extract features from the driving audio information and the original image to obtain audio-image fusion features. The audio-image fusion features are used to characterize the correspondence between audio embedding features and global image embedding features. S204. Generate video constraint information based on audio-image fusion features, the original image, and the mask image.

[0070] In some embodiments, audio features can be extracted from the driving audio information, such as frequency division, spectral feature extraction, semantic recognition, etc., and audio embedding features can be obtained through an audio editor. Visual feature encoding can be performed on the original image to obtain global image embedding features, and audio-image fusion features can be obtained based on the audio embedding features and the global image embedding features.

[0071] Audio embedding features can include audio features and deep feature information, where deep feature information can include audio intent, rhythm, semantics, etc. Image global embedding features are used to represent image content. Furthermore, audio-image fusion features are used to represent the correspondence between audio intent and image content.

[0072] In some embodiments, such as Figure 3 As shown, S202 above performs feature extraction on the driving audio information and the original image to obtain audio-image fusion features, including: S302. Use a Multimodal Large Language Model (MLLM) to extract features from the driving audio information to obtain audio embedding features; S304. Use a multimodal parsing unit to encode the visual features of the original image to obtain the global embedding features of the image; S306. Perform cross-modal alignment on the audio embedding features and the global image embedding features to obtain the audio-image fusion features.

[0073] The multimodal parsing unit is equipped with a multimodal parsing model and a multimodal large language model, which can simultaneously process, understand and fuse multiple types of input data (such as raw images, driving audio information, etc.) to achieve cross-modal semantic alignment and collaborative reasoning.

[0074] Feature embedding, or feature embedding, refers to the process of transforming raw data (such as raw images or driving audio information) into high-dimensional feature vectors through an encoder. The transformed feature vectors can be used for subsequent calculations and fusion in MLLM, and are the foundation for cross-modal information interaction.

[0075] Cross-modal alignment refers to mapping data from different modalities (such as audio embedding features and image global embedding features) to the same feature space, enabling MLLM to understand the correspondence between different modal data, such as the correspondence between audio rhythm and changes in a person's lip shape.

[0076] In some embodiments, MLLM can perform semantic parsing and feature extraction on the input driving audio information to obtain audio features, or it can mine deep feature information such as audio intent, rhythm, and semantics in the driving audio information and fuse the audio features and deep feature information into audio embedding features.

[0077] MLLM is used to process the original image, extracting local textures, contours and other visual features. An attention mechanism is combined to strengthen the weight of the features of the main body region of the image, and finally a fixed-dimensional feature vector is output to complete the visual feature encoding and obtain the global embedding features of the image.

[0078] Global image embedding features are a condensed feature representation of the original image. They are a set of fixed-dimensional numerical vectors that comprehensively capture the global semantic information and key local features of the original image. These features serve as the core visual reference for subsequent mask image generation and preset video generation, enabling accurate transfer and reuse of the original image features. Global semantic information can include subject category, overall style, color analysis, etc., while key local features can include subject outline, detailed texture, etc.

[0079] In this embodiment, a cross-modal attention alignment mechanism can be used to align audio embedding features and image global embedding features. By calculating attention weights, the correlation between audio embedding features and image global embedding features can be explored, such as the correspondence between audio rhythm and image subject, and the matching between audio emotion and image style. This strengthens the weights of related features and suppresses interference from irrelevant features. Through feature splicing and fusion, the aligned audio embedding features and image global embedding features are fused to output an audio-image fusion feature with fixed dimensions and close semantic association. This audio-image fusion feature can simultaneously carry the visual information of the original image and the temporal and emotional information that drives the audio information, providing more comprehensive multimodal features for the subsequent generation of visual constraint information.

[0080] like Figure 4 As shown, in some embodiments, the video constraint information includes structured prompt text; the above-described S204 generates video constraint information based on audio-image fusion features, the original image, and the mask image, including: S2042. Using the prompting engineering unit, the original image, audio intent, and mask image are fused according to the preset prompting template to generate structured prompting text.

[0081] Pre-configured prompt templates can be configured in the prompt engineering unit. These templates characterize the generation of structured prompt text with fixed sentence structures, word order, and information layout from the user-input original image, extracted audio-image fusion features, and mask images. This ensures the final structured prompt text has a regular format and standardized expression, avoiding semantic clutter and disordered word order, and guaranteeing standardized output of prompt information. Pre-configured prompt templates can include fields such as visual content, action, audio intent, region, style, and mask constraints to correspond to different content extraction and fusion processes.

[0082] For example, the structured prompt text could be "Based on the input original image, generate natural, synchronous, and realistic dynamic changes according to the audio rhythm." This prompt does not rely on manual writing but is generated by the system based on the content of the original image and simple prompt words input by the user, providing clear constraints.

[0083] Figure 5 A flowchart of another image-to-video method provided in this disclosure is shown. Figure 5 As shown, in some embodiments, the above-described S108 generates initial noisy video latent variables, including: S1082. Based on the number of frames in the target video, the size of the original image, and the number of channels, generate initial noisy video latent variables. The initial noisy video latent variables conform to a Gaussian distribution.

[0084] In some embodiments, based on the number of frames N of the target video, the height H, width W, and number of channels C of the original image, the dimension of the initial noisy video latent variable is determined to be [N, H, W, C]. Based on this dimension, a four-dimensional tensor matching the spatiotemporal dimension of the target video is generated through a random initialization function. Gaussian distribution sampling is performed on all elements in the four-dimensional tensor so that the four-dimensional tensor follows a standard Gaussian distribution with a mean of 0 and a variance of 1, thus obtaining the initial noisy video latent variable, which serves as the initial iterative basis for the target video generation process.

[0085] Figure 6 This diagram illustrates a flowchart of yet another image-to-video method provided in an embodiment of this disclosure. Figure 6 As shown, in some embodiments, the above-described S110 fuses video constraint information, initial noisy video latent variables, and mask image to generate a context condition vector, including: S1102. Using the cross-attention mechanism, the video constraint information, the initial noisy video latent variables, and the mask image are mapped to the target latent space, and then spliced ​​and fused to obtain the context condition vector.

[0086] In deep learning, latent space can be used to represent the deep features of data in a high-dimensional space.

[0087] In practice, a unified target latent space is constructed, and the video constraint information, initial video latent variables and mask image are adapted for dimension and projected for feature respectively, so as to map the three to the same target latent space, thereby eliminating the dimensional differences and feature domain differences between different data.

[0088] A cross-attention mechanism is introduced to calculate the cross-attention weights among the three types of features, thereby mining the correlation between video constraint information, initial video latent variables, and mask images. In other words, it mines the correlation between video constraint semantics, temporal features, and spatial features. Video constraint semantics are determined by video constraint information, temporal features are determined by initial video latent variables, and spatial features are determined by mask images. Based on the cross-attention weights, effective correlated features are strengthened and irrelevant redundant features are weakened, thus completing the information interaction and semantic alignment of multiple types of features.

[0089] After completing the attention-weighted optimization, the three types of features that have been mapped and weighted are sequentially concatenated according to the preset channel dimensions. Deep information fusion is achieved through a multi-layer feature fusion layer. The concatenated features are then subjected to dimensional compression and semantic integration, and finally, a context condition vector is output. The context condition vector has semantic uniformity, complete spatiotemporal information, and combines constraint rules and spatial region attributes.

[0090] In this embodiment, multi-source information dimension normalization is achieved through target latent space mapping, eliminating modal heterogeneity and dimensional differences among the three features. The cross-attention mechanism can strengthen effective features, filter out invalid interference features, improve the accuracy of contextual condition vectors in the target video generation process, and reduce the problem of image distortion.

[0091] Figure 7 This diagram illustrates a flowchart of a target video generation method provided by an embodiment of the present disclosure. Figure 7 As shown, in some embodiments, the mask image includes a first region and a second region; the above-described S112 generates the target video based on the original image and the context condition vector, including: S702. Using a visual diffusion self-attention architecture (Diffusion Transformer, DiT), the original image is mapped to the target latent space and superimposed with the latent variables of the initial noisy video to obtain the video to be driven. S704. Calculate the attention weights of each layer in the video to be driven; S706. Generate the target video based on the attention weights and contextual condition vectors.

[0092] DiT is a video-level diffusion generation architecture that supports temporal frame sequence modeling. It maps the static original image to a target latent space with the same dimension as the initial noisy video latent variables. The superposition of the two has a frame-time dimension. The original image can provide visual style, subject content, etc., while the multi-frame Gaussian distributed initial noisy video latent variables provide the degrees of freedom for temporal changes in the target video. The superimposed video to be driven forms a video-level latent feature with spatiotemporal dimensions.

[0093] In each layer of the DiT network, the video to be driven is split into an ordered sequence of video frames and a sequence of spatial feature blocks along the spatiotemporal dimension, constructing a temporal-spatial joint feature sequence, which includes multiple ordered temporal-spatial joint features. In each self-attention module of DiT, the temporal-spatial joint features are mapped to query vectors, key vectors, and value vectors, respectively. The spatiotemporal feature similarity is calculated by matrix multiplication of query vectors and key vectors to obtain the original attention score. The original attention score is then subjected to Softmax normalization to generate the attention weight of the video to be driven in the current layer. This attention weight can characterize the correlation strength and dependency between spatial regions and temporal frames in the video to be driven.

[0094] In some embodiments, when calculating the attention weights of each layer in the video to be driven, the attention weights of the positions marked as the first region in the mask image can be calculated only, and the original image in the driving region can be updated according to the audio and cue conditions; the attention weights of the positions marked as the second region in the mask image can be not calculated, or the attention weights of the positions in the second region can be set to 0, thereby forcibly preserving the features of the original image, and the non-driving regions do not participate in the dynamic update.

[0095] In other embodiments, S706 above generates the target video based on attention weights and contextual condition vectors, including: For the first target region located within the first region and whose attention weight is greater than the preset weight threshold, feature updates and dynamic driving are performed on the first target region based on the context condition vector; For the second target region that is in the second region or in the first region and whose attention weight is less than the preset weight threshold, no feature update or dynamic driving is performed on the second target region. The features of the original image are preserved to generate the target video.

[0096] The aforementioned preset weight threshold can be pre-configured in DiT. The value of the preset weight threshold can be determined as needed, for example, the preset weight threshold is 0.5, 0.7, etc., and this disclosure does not make specific limitations in this regard.

[0097] The first region refers to the driving region in the masked image, which may include the driving subject, such as a person; the second region refers to the non-driving region in the masked image, which usually includes the background.

[0098] The first target region is the driving region of the original image determined jointly by the first region and the attention weight. The second target region includes the second region and the portion of the first region where the attention weight is less than a preset weight threshold.

[0099] In this embodiment of the disclosure, during the attention weight calculation process of each layer in the video to be driven, a preset weight threshold is set, and a first target region and a second target region are determined by combining the mask region. Feature updates and dynamic driving are performed on the image in the first target region, but no feature updates and dynamic driving are performed on the second target region. This achieves precise control that the audio signal drives only the specified region and the non-specified region remains unchanged, ensuring the integrity of the background and non-driven regions of the generated target video and avoiding unexpected dynamic changes.

[0100] Based on the same inventive concept, this disclosure also provides a video image generation device, as shown in the following embodiments. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0101] Figure 8 This diagram illustrates the structure of a video-generating device according to an embodiment of this disclosure. Figure 8 As shown, the image-to-video apparatus of this disclosure includes a data acquisition module 810, a mask generation module 820, a constraint generation module 830, a noise generation module 840, a context generation module 850, and a video generation module 860.

[0102] Among them, the data acquisition module 810 is used to acquire the original image and driving audio information; The mask generation module 820 is used to generate a mask image based on the original image; The constraint generation module 830 is used to generate video constraint information based on the original image, driving audio information, and mask image; Noise generation module 840 is used to generate initial noise video latent variables; The context generation module 850 is used to fuse video constraint information, initial noisy video latent variables and mask image to generate a context condition vector; The video generation module 860 is used to generate the target video based on the original image and the contextual condition vector.

[0103] It should be noted that the data acquisition module 810, mask generation module 820, constraint generation module 830, noise generation module 840, context generation module 850, and video generation module 860 correspond to S102 to S112 in the method embodiment. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiment. It should also be noted that these modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0104] In some embodiments, the constraint generation module 830 is used to extract features from the driving audio information and the original image to obtain audio-image fusion features, which are used to characterize the correspondence between audio embedding features and global image embedding features; and to generate video constraint information based on the audio-image fusion features, the original image and the mask image.

[0105] In some embodiments, the constraint generation module 830 is used to extract features from the driving audio information using a multimodal parsing unit to obtain audio embedding features; to encode visual features of the original image using a multimodal parsing unit to obtain image global embedding features; and to perform cross-modal alignment of the audio embedding features and the image global embedding features to obtain audio-image fusion features.

[0106] In some embodiments, the video constraint information includes structured prompt text; the constraint generation module 830 is used to use the prompt engineering unit to perform fusion processing on the original image, audio intent and mask image according to a preset prompt template to generate structured prompt text.

[0107] In some embodiments, the noise generation module 840 is used to generate an initial noise video latent variable based on the number of frames of the target video, the size of the original image, and the number of channels. The initial noise video latent variable conforms to a Gaussian distribution.

[0108] In some embodiments, the context generation module 850 is used to utilize a cross-attention mechanism to map video constraint information, initial noisy video latent variables, and mask image to the target latent space, and perform splicing and fusion processing to obtain a context condition vector.

[0109] In some embodiments, the mask image includes a first region and a second region; the video generation module 860 is used to map the original image to the target latent space using a visual diffusion self-attention architecture, and superimpose it with the initial noisy video latent variables to obtain the video to be driven; calculate the attention weight of each layer in the video to be driven; and generate the target video according to the attention weight and the context condition vector.

[0110] In some embodiments, the video generation module 860 is used to update the features and dynamically drive the first target region based on the context condition vector for the first target region located in the first region and whose attention weight is greater than a preset weight threshold; for the second target region located in the second region or the second target region located in the first region and whose attention weight is less than the preset weight threshold, the second target region is not updated or dynamically driven, and the features of the original image are retained to generate the target video.

[0111] In this embodiment, the following steps are taken: acquiring the original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, driving audio information, and mask image; generating initial noisy video latent variables; fusing the video constraint information, initial noisy video latent variables, and mask image to generate a contextual condition vector; and generating the target video based on the original image and contextual condition vector. The image-to-video solution provided in this disclosure eliminates the need for model training and additional data acquisition or construction, effectively reducing the cost of data acquisition and computing power deployment, simplifying the implementation process of image-to-video, and enabling the direct generation of the target video by inputting only the original image and driving audio information. This significantly improves the usability and practical deployment efficiency of the image-to-video solution, providing strong support for the large-scale application of image-to-video technology.

[0112] In a specific implementation, the image-to-video device may include a data acquisition module 810, a constraint generation module 830, and a content generation module. The constraint generation module 830 includes an MLLM unit and a PE unit, and the content generation module includes a mask generation module 820, a noise generation module 840, a context generation module 850, and a video generation module 860.

[0113] The data acquisition module 810 serves as the data entry point for the device, used to receive externally input raw images and driving audio signals. The raw images serve as the base frame for video generation, and the driving audio signals are used to provide characteristic information for audio driving.

[0114] The mask generation module 820 is connected to the data acquisition module 810 and is used to detect and locate the visual subject of the original image, extract the features of the subject region, and obtain the mask image.

[0115] The MLLM unit is connected to the data acquisition module 810 and is used to extract features from the input driving audio information to obtain audio embedding features; to encode visual features of the original image to obtain image global embedding features; and to perform cross-modal alignment of the audio embedding features and the image global embedding features to obtain audio-image fusion features, thereby mining deep feature information such as audio intent, rhythm, and semantics in the driving audio signal.

[0116] The PE unit is connected to the MLLM unit and is used to receive the audio-image fusion features, mask image, original image and audio intent output by the MLLM unit. Through preset prompt template and engineered prompt strategy, the audio intent, original image and mask image are fused to generate a structured prompt text Prompt to guide the video generation module 860. The structured prompt text limits the content, action, region and style of the video generation. The PE unit outputs the structured prompt text to the content generation module.

[0117] The noise generation module 840 is used to generate initial noise video latent variables, providing latent variable support for the generation process of the video generation module 860.

[0118] The context generation module 850 is connected to the data acquisition module 810, the noise generation module 840, and the mask generation module 820. It is used to receive the structured prompt text output by the PE unit, the initial noise video latent variables output by the noise generation module 840, the mask image output by the mask generation module 820, and the original image output by the data acquisition module 810. It performs fusion processing on the structured prompt text, the initial noise video latent variables, the mask image, and the original image to generate a context condition vector. The context condition vector contains audio driving features, mask region constraint features, noise latent variable information, etc.

[0119] The video generation module 860 is connected to the data acquisition module 810 and the context generation module 850. It is used to receive the original image output by the data acquisition module 810 and the context condition vector output by the context generation module 850. Based on the original image, combined with the mask region constraint features, audio driving features and noise latent variable information in the context condition vector, it performs audio-driven video generation operation in the driving region of the mask image, gradually generating video frames that match the audio signal and conform to the region constraints, and finally outputs the target video, thereby realizing high-precision, controllable, and specified region audio-driven video generation.

[0120] This disclosure is the first to automatically divide the mask region through the mask generation module 820, and combine the mask region to realize the precise driving of the audio signal to the specified region in the original image. It breaks through the technical defects of the existing technology, which has the problem of multi-dimensional full-domain driving of audio driving, which cannot achieve precise region control, or requires manual marking of the mask region and is cumbersome.

[0121] This disclosure uses a mask generation module 820 to perform subject recognition and localization on the original image, automatically detecting the core subjects in the original image, such as faces, bodies, and specific objects. Based on the detection results, a mask image aligned with the size of the original image is generated. The mask image is a single-channel binary image. In the mask image, the subject region is the driving region and is marked as 1; the non-subject region is the non-driving region and is marked as 0. The mask image is used as the core constraint condition and passed to the context generation module 850 and the video generation module 860 to provide a basis for audio-driven region limitation. This realizes a closed-loop process of automatic subject region recognition, automatic mask region division, and precise driving of specified regions without the need for manual intervention in labeling the driving region, thus improving the ease of operation and driving accuracy.

[0122] This disclosure eliminates the need for separate training, fine-tuning, or adaptation for specific audio-driven video generation models. It can be directly applied to existing audio-driven video generation models, significantly improving versatility and practicality, and reducing the cost of technology implementation and promotion.

[0123] The region constraints disclosed herein are independent of the video generation model. A standardized input system of region constraints and audio features is constructed through a parsing module and a contextual condition fusion module. The mask region constraints and audio-driven features are transformed into contextual condition vectors that can be universally recognized by the model. Without modifying the network structure of the model itself or performing additional training iterations, it can directly connect to various audio-driven generation models to achieve audio-driven functions for specified regions. This solves the problems of poor universality and high adaptation costs of existing region-driven technologies.

[0124] By setting a threshold filtering mechanism in the Self-attention module of the video generation module, the driving range of the audio signal can be precisely controlled, ensuring that the audio drive only acts on the driving area specified in the mask image, while the non-specified non-driving areas retain the original image state, thereby further improving the accuracy of region driving and the stability of the generated video.

[0125] The audio-driven features output by the context generation module 850 and the region features of the mask image output by the mask generation module are input into the video generation module 860. During the Self-attention calculation process, a preset weight threshold (which can be adjusted according to actual generation requirements) is set to filter the input feature map. Only the region features within the first region with a feature weight greater than the threshold are retained. The audio-driven signal only updates and dynamically drives the features of the filtered region. For the region within the second region, or the region outside the first region with a feature weight less than or equal to the threshold, the feature information of the original image is forcibly retained, and no audio-driven feature updates are performed. This achieves precise control that the audio signal only drives the specified region, while the non-specified region remains unchanged. This ensures the integrity of the background and non-driven regions of the generated video and avoids unexpected dynamic changes.

[0126] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0127] The following reference Figure 9 To describe an electronic device 900 according to this embodiment of the present invention. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0128] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including storage unit 920 and processing unit 910).

[0129] The storage unit stores program code, which can be executed by the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 910 can perform actions such as... Figure 1 The process shown involves: acquiring the original image and driving audio information; generating a mask image based on the original image; generating video constraint information based on the original image, driving audio information, and mask image; generating initial noisy video latent variables; fusing the video constraint information, initial noisy video latent variables, and mask image to generate a contextual condition vector; and generating the target video based on the original image and contextual condition vector.

[0130] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0131] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0132] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0133] Electronic device 900 can also communicate with one or more external devices 940 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with the system, and / or with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, the system can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. Figure 9 As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although... Figure 9 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, magnetic stripe drives, and data backup storage systems.

[0134] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0135] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, performs the steps described in the foregoing "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0136] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0138] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0139] In practice, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0141] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0142] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A graph-based video method, characterized by, include: Acquire raw image and driving audio information; Generate a mask image based on the original image; Based on the original image, the driving audio information, and the mask image, video constraint information is generated; Generate initial noisy video latent variables; The video constraint information, the initial noisy video latent variables, and the mask image are fused to generate a context condition vector; The target video is generated based on the original image and the contextual condition vector.

2. The graph-based video method of claim 1, wherein, The step of generating video constraint information based on the original image, the driving audio information, and the mask image includes: Feature extraction is performed on the driving audio information and the original image to obtain audio-image fusion features, which are used to characterize the correspondence between audio embedding features and image global embedding features; The video constraint information is generated based on the audio-image fusion features, the original image, and the mask image.

3. The graph-based video method of claim 2, wherein, The step of extracting features from the driving audio information and the original image to obtain audio-image fusion features includes: The driving audio information is used to extract features using a multimodal parsing unit to obtain audio embedding features; The original image is visually feature-encoded using the multimodal parsing unit to obtain global embedding features of the image; The audio embedding features and the image global embedding features are aligned across modalities to obtain the audio-image fusion features.

4. The graph-based video method of claim 2, wherein, The video constraint information includes structured prompt text; The step of generating the video constraint information based on the audio-image fusion features, the original image, and the mask image includes: Using the prompting engineering unit, the original image, audio intent, and mask image are fused according to a preset prompt template to generate the structured prompt text.

5. The graph-based video method of claim 1, wherein, The generation of initial noisy video latent variables includes: The initial noisy video latent variable is generated based on the frame number of the target video, the size of the original image, and the number of channels. The initial noisy video latent variable conforms to a Gaussian distribution.

6. The image-to-video method according to any one of claims 1-5, characterized in that, The process of fusing the video constraint information, the initial noisy video latent variables, and the mask image to generate a context condition vector includes: Using a cross-attention mechanism, the video constraint information, the initial noisy video latent variables, and the mask image are mapped to the target latent space, and then spliced ​​and fused to obtain the context condition vector.

7. The image-to-video method according to claim 6, characterized in that, The mask image includes a first region and a second region; The step of generating the target video based on the original image and the contextual condition vector includes: Using a visual diffusion self-attention architecture, the original image is mapped to the target latent space and superimposed with the latent variables of the initial noisy video to obtain the video to be driven; Calculate the attention weights of each layer in the video to be driven; The target video is generated based on the attention weights and the contextual condition vector.

8. The image-to-video method according to claim 7, characterized in that, The step of generating the target video based on the attention weights and the contextual condition vector includes: For a first target region located within the first region and whose attention weight is greater than a preset weight threshold, feature updates and dynamic driving are performed on the first target region based on the context condition vector; For a second target region that is located in the second region or in the first region and whose attention weight is less than the preset weight threshold, no feature update or dynamic driving is performed on the second target region. Instead, the features of the original image are preserved to generate the target video.

9. The graph-based video method of claim 7, wherein, The calculation of the attention weights for each layer in the video to be driven includes: Calculate the attention weights for the first region; Set the attention weight of the second region to 0.

10. A graph-based video device, comprising: include: The data acquisition module is used to acquire raw image and driving audio information; A mask generation module is used to generate a mask image based on the original image; The constraint generation module is used to generate video constraint information based on the original image, the driving audio information, and the mask image; The noise generation module is used to generate initial noisy video latent variables; The context generation module is used to fuse the video constraint information, the initial noisy video latent variables, and the mask image to generate a context condition vector; The video generation module is used to generate a target video based on the original image and the context condition vector.

11. An electronic device, comprising: include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image-generated video method as claimed in any one of claims 1-9 by executing the executable instructions.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the image-generated video method as described in any one of claims 1-9.

13. A computer program product, characterised in that, The computer program product includes a computer program or computer instructions, which are loaded and executed by a processor to enable the computer to implement the image-to-video method as described in any one of claims 1-9.

Images