Video generation method and device, intelligent agent, electronic equipment and storage medium

Abstract

The invention discloses a video generation method, belongs to the technical field of artificial intelligence, particularly relates to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as digital humans and the like. The method comprises the following steps: extracting a target audio feature of a target audio and a virtual image feature of a virtual image in a reference image; based on an attention mechanism, processing the target audio feature and the initial video feature to obtain a target video feature, the initial video feature being formed by splicing a virtual image feature and random noise; and decoding the target video feature to obtain a target video, the target video comprising a video frame of the virtual image expressed based on the target audio speech. The invention further provides a video generation device, an intelligent agent, electronic equipment and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to scenarios such as digital humans. Specifically, it relates to a video generation method, device, intelligent agent, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, audio-driven video generation technology has been widely used in various fields such as virtual anchors, online education, game animation, movie previews, and digital endorsements. However, the audio-generated videos in these examples still have many issues that need to be optimized. Summary of the Invention

[0003] This disclosure provides a video generation method, apparatus, intelligent agent, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a video generation method is provided, comprising: extracting target audio features of a target audio source and virtual image features of a virtual image in a reference image; processing the target audio features and initial video features based on an attention mechanism to obtain target video features, wherein the initial video features are formed by concatenating virtual image features with random noise; and decoding the target video features to obtain a target video, wherein the target video includes video frames of virtual image speech expressions based on the target audio source.

[0005] According to another aspect of this disclosure, a video generation apparatus is provided, comprising: an extraction module for extracting target audio features of a target audio source and virtual image features of a virtual image in a reference image; a processing module for processing the target audio features and initial video features based on an attention mechanism to obtain target video features, wherein the initial video features are obtained by processing the initial video features formed by splicing virtual image features with random noise; and a decoding module for decoding the target video features to obtain a target video, wherein the target video includes video frames of virtual image speech expressions based on the target audio source.

[0006] According to another aspect of this disclosure, an artificial intelligence agent is provided, configured to perform the above-described method.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 The illustration schematically depicts an exemplary system architecture to which video generation methods, apparatuses, smart agents, electronic devices, and storage media can be applied according to embodiments of the present disclosure;

[0013] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically;

[0014] Figure 3 A schematic diagram illustrating the determination of initial video features is shown;

[0015] Figure 4 The illustration shows a schematic diagram of a method for generating target video features according to an embodiment of the present disclosure;

[0016] Figure 5 The illustration shows a schematic diagram of splicing historical video features and initial video features to form a first spliced ​​video feature according to an embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram illustrating the generation of target video features according to an embodiment of the present disclosure is shown.

[0018] Figure 7 A schematic diagram illustrating the determination of compressed historical video features according to an embodiment of the present disclosure is shown.

[0019] Figure 8 The schematic diagram illustrates the architecture of a video generation model according to an example of this disclosure;

[0020] Figure 9 A schematic diagram of the structure of the dit module according to an embodiment of the present disclosure is shown;

[0021] Figure 10A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0022] Figure 11 A schematic diagram illustrating the structure of an intelligent agent of artificial intelligence according to embodiments of the present disclosure; and

[0023] Figure 12 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0026] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0027] The audio-based single-image digital human driving technologies in the relevant examples mainly include the following categories:

[0028] Rule-based lip-mapping scheme: This scheme drives the lip movement of a single-image digital human through a predefined "phoneme-lip shape" correspondence table (such as the binding of phonemes to lip keyframes), while simultaneously overlaying simple head-movement templates (such as nodding and turning the head) to generate video. For example, relying on manually annotated correspondences between speech syllables and lip shapes, inter-frame transitions are achieved through interpolation algorithms.

[0029] A 3D Morphable Model (3DMM)-based driving scheme: This scheme parameterizes facial expressions and head poses using 3DMM, mapping speech features to 3DMM motion coefficients (such as lip opening degree and head rotation angle), and then generates video through a renderer. For example, the SadTalker model learns head movement styles through PoseVAE and combines them with ExpNet to predict expression coefficients, achieving audio-driven 3D facial animation.

[0030] A two-stage deep learning approach: The first stage uses convolutional neural networks or recurrent neural networks to extract temporal features of speech (such as Mel spectrum and Mel frequency cepstral coefficients). The second stage uses generative adversarial networks (GANs) or Transformer models to generate video from single-image and speech features as input. For example, some methods combine Mel-GAN to generate lip regions and then use a 3D convolutional neural network to complete the inter-frame motion information. Mel-GAN is a specific GAN architecture designed for the inverse transformation of the Mel spectrum (i.e., generating the original audio waveform from the Mel spectrum).

[0031] However, the technical solutions in the above-mentioned examples have the following technical problems:

[0032] First, the semantic matching between actions and speech is insufficient. Rule-based lip-syncing schemes can only achieve basic lip-sync and struggle to adapt to the personalized rhythms of speech (such as speech rate and pauses), easily leading to "lip-syncing and speech misalignment." While two-stage deep learning schemes can capture temporal features, they are insufficiently responsive to emotional changes in speech (such as subtle body movements when angry), resulting in a disconnect between emotional actions and audio semantics. For example, the digital human in the generated video maintains a stiff expression even when expressing cheerful speech.

[0033] Secondly, the consistency of identity in a single image is poor. The solutions in the relevant examples are prone to problems such as "character deformation" and "background flickering" when generating long time sequences (such as exceeding 5 seconds) because the model lacks continuous anchoring of single image features; although the 3DMM solution can maintain facial structure, it is not accurate enough in restoring appearance details such as hairstyle and skin color, and it is easy to lose identity features, especially when the face is turned to the side or when the head moves at a large angle.

[0034] In addition, the scene adaptability is limited: it is designed for the "single person frontal static background" scene and has poor compatibility with complex backgrounds (such as dynamic background elements) or multilingual scenes (such as audio input in a minority language).

[0035] Based on this, this disclosure provides a video generation method, including: extracting target audio features from target audio and virtual image features from a virtual image in a reference image; processing the target audio features and initial video features based on an attention mechanism to obtain target video features, wherein the initial video features are formed by concatenating virtual image features with random noise; and decoding the target video features to obtain a target video, wherein the target video includes video frames of virtual image speech based on target audio.

[0036] This disclosure adopts a three-tier architecture of "single-image feature anchoring – audio-driven feature fusion – video frame generation optimization" and takes "precise lip alignment" and "natural and smooth movements" as the core objectives to build an end-to-end link of "multimodal input processing → deep feature fusion → efficient video generation".

[0037] This architecture takes "single audio segment" and "reference image" as input. It extracts temporal features from the audio using a finely designed feature extraction and alignment module, obtains latent image features from the reference image through a 3D Variational Autoencoder (3D VAE), and then achieves deep cross-modal feature fusion and alignment through the core Diffusion Transformer module (dit module). Finally, it uses 3D VAE decoding to generate the video. The overall process follows the logic of "low-dimensional feature interaction → high-dimensional video reconstruction," without relying on additional video templates or complex motion templates. While ensuring generation efficiency, it also ensures high matching between lip movements and audio, and naturalness of motion, supporting long video generation scenarios (with better duration coverage to meet richer scene requirements).

[0038] Figure 1 The illustration schematically depicts an exemplary system architecture to which video generation methods, apparatuses, smart agents, electronic devices, and storage media can be applied according to embodiments of the present disclosure.

[0039] It is important to note that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture applicable to video generation methods, apparatuses, intelligent agents, electronic devices, and storage media may include a terminal device. However, the terminal device may implement the video generation methods, apparatuses, intelligent agents, electronic devices, and storage media provided in the embodiments of this disclosure without interacting with a server.

[0040] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0041] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0043] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0044] It should be noted that the video generation method provided in this embodiment can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the video generation processing apparatus provided in this embodiment can also be disposed in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0045] Alternatively, the video generation method provided in this embodiment can generally be executed by server 105. Correspondingly, the video generation apparatus provided in this embodiment can generally be located in server 105. The video generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the video generation apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0046] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0047] The video generation method can be applied to content creation, such as assisting self-media and short video creation by quickly generating narrated videos using real-person avatars and voice narration, eliminating the need for repeated filming of real people. It can also be used in the field of digital assets, such as generating digital avatars of individuals or businesses, and enabling the mass production of videos for brand promotion and knowledge dissemination through audio-driven processes.

[0048] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically.

[0049] like Figure 2 As shown, the video generation method of this embodiment includes operations S210~S230.

[0050] In operation S210, the target audio features of the target audio and the virtual image features of the virtual image in the reference image are extracted.

[0051] The target audio can be any audio that drives the virtual character to speak; it can be a voice message, a song, or something else.

[0052] Target audio features are high-dimensional information extracted from target audio that can precisely drive lip movements. For example, target audio features may include acoustic features such as pitch, loudness, speech rate, and spectral characteristics; they may also include content features such as emotional tone and semantic information in speech, or rhythm and melody features in music; and they may include temporal information. By extracting target audio features, the audio signal can be transformed into quantifiable and analyzable feature data for subsequent processing.

[0053] Extracting target audio features can include: using a pre-trained audio network to process the waveform of the target audio, converting the audio into a high-dimensional, dense sequence of feature vectors, thereby obtaining the target audio features.

[0054] For example, extracting target audio features from target audio may include: using the Wav2Vec speech model to extract core acoustic features such as temporal spectrum from the target audio, generating a structured data packet called Audio Pack. This Audio Pack can be the target audio features, which may contain a sequence of high-dimensional feature vectors corresponding to each audio segment (e.g., every 20 milliseconds).

[0055] In some examples, after generating the "Audio Pack", the audio segment features of each audio segment in the Audio Pack can be made aware of the feature information of the surrounding time period (such as the first 200 milliseconds and the last 200 milliseconds) based on the context mechanism, so as to supplement the context information of each audio segment, thereby improving the temporal coherence of subsequent lip-sync generation and providing high-dimensional and accurate audio feature basis for the precise driving of lip-sync and movement.

[0056] The reference image can be a static image containing a virtual avatar, such as a cartoon character, an anime character, or a digital human. For example, the reference image could be a real-life image provided by the user.

[0057] Virtual avatar features can be latent feature representations encoded from a reference image, representing the invariant attributes of the virtual avatar's identity. These virtual avatar features can include the virtual avatar's identity information, such as face shape, hairstyle, and the basic structure of facial features.

[0058] Extracting virtual avatar features from a reference image can include encoding the reference image using 3D VAE. The three-dimensional structure of 3D VAE allows the model to better understand the spatial volume and depth information of the virtual avatar's face, thus obtaining the essential features of the image.

[0059] In operation S220, based on the attention mechanism, the target audio features and the initial video features are processed to obtain the target video features. The initial video features are formed by splicing virtual image features with random noise.

[0060] It should be noted that since the target audio features are temporal (i.e., a sequence), while the virtual avatar features of a reference image are single features, the virtual avatar features of the reference image need to be expanded into a sequence whose length perfectly matches the temporal length of the audio features. In other words, each moment of audio corresponds to a virtual avatar feature.

[0061] For example, expanding the virtual image features of a reference image into a sequence may include: copying and expanding the virtual image features into a sequence according to the sequence length of the target audio features, so that its length completely matches the temporal length of the target audio features, to obtain the expanded virtual image features.

[0062] For example, expanding the virtual image features of the reference image into a sequence may also include: expanding the virtual image features using a preset number, such as 0, to form a sequence whose length perfectly matches the temporal length of the target audio features.

[0063] Random noise can be a sequence randomly sampled from a standard Gaussian distribution, and its length can be the same as the length of an audio segment.

[0064] In the diffusion model, generation begins with pure random noise. This random noise represents the initial video frame, which at this point has no meaning.

[0065] Concatenating virtual avatar features with random noise to form initial video features can be achieved by concatenating the expanded virtual avatar features with random noise to form initial video features.

[0066] Figure 3 A schematic diagram illustrating the determination of initial video features is shown.

[0067] like Figure 3 As shown, the method 300 for determining initial video features in this embodiment may include: performing temporal extension on virtual avatar features 310 to obtain extended virtual avatar features 320; and then concatenating the extended virtual avatar features 320 with random noise 330 in the channel dimension to obtain initial video features 340.

[0068] Attention mechanisms are algorithms that mimic human attention allocation, allowing models to focus on key parts when processing information. In this scenario, they are used to guide the model to focus on the most correlated parts of the audio features and the initial video features, achieving accurate matching between the two.

[0069] Based on the attention mechanism, processing the target audio features and the initial video features may include: using a cross-attention mechanism to calculate the correlation weight between each moment of the target audio (e.g., each phoneme) and each part of the initial video features to obtain the attention weight, weighting the target audio features according to the attention weight, and adjusting the initial video features using the weighted audio features to obtain the target video features.

[0070] Before using the cross-attention mechanism to calculate the correlation weights between the target audio and the initial video features, a self-attention mechanism can also be used to process the initial video features. In this case, the initial video features themselves are used as a QKV matrix for regular matrix calculation.

[0071] By utilizing an attention mechanism, the temporal information of the target audio is precisely integrated into the video generation process, driving the virtual avatar to make lip movements and facial expressions that match the sound.

[0072] In operation S230, the target video features are decoded to obtain the target video, which includes video frames of the virtual image based on the target audio speech expression.

[0073] Verbal expression can include: speaking, giving speeches, reciting, singing, etc.

[0074] After multiple iterations of denoising using the S220, the target video features are a clean feature representation within the neural network, incorporating the virtual avatar's identity and audio information. This feature representation needs to be restored to a viewable and understandable final video product. At this point, a video decoder can be used to decode the target video features.

[0075] For example, decoding the target video features may include: inputting the target video features into a 3DVAE decoder, performing decoding operations on the target video features, and outputting a high-fidelity video frame sequence to obtain a target video that highly matches the target audio and exhibits natural and smooth motion. The virtual avatar in the target video will lip-sync and speak according to the input target audio, and may also be accompanied by natural facial expressions and micro-head movements.

[0076] By first extracting the target audio features from the target audio and the virtual avatar features from the reference image, and then using an attention mechanism, the initial video features formed by splicing the virtual avatar features with random noise and the target audio features are deeply fused together. This accurately captures the correlation between audio information and the dynamics of the virtual avatar. Finally, by decoding the fused target video features, a video frame that clearly presents the virtual avatar's speech expression based on the target audio is generated. This achieves a natural dynamic video generation effect that uses a single reference image as the avatar benchmark and is driven by audio to synchronize the virtual avatar's lip movements and actions with the sound. It effectively constructs a complete technical link from audio and image input to virtual avatar speech expression video output, improving the accuracy and realism of virtual avatar video generation.

[0077] The following is for reference. Figures 4-9 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0078] According to embodiments of this disclosure, target audio features and initial video features are processed based on an attention mechanism to obtain target video features, including: performing cross-attention processing on target audio features and initial video features under the constraint of temporal relationship information to obtain target video features, wherein the temporal relationship information indicates the temporal correspondence between the initial video features and the target audio features.

[0079] Temporal relationship information can be represented as an attention mask matrix, with a shape of [initial video feature length, target audio feature length]. The values ​​in the attention mask matrix are typically 0 and 1 (or 0 and negative infinity). 1 indicates that video frame i and audio segment j are temporally corresponding and should be considered when calculating attention. 0 indicates that video frame i and audio segment j are temporally independent and should be ignored when calculating attention.

[0080] Cross-attention processing is a technique that allows features from two different modalities (such as audio and video) to pay attention to each other. Specifically, video features pay attention to the parts of audio features that are associated with them, while audio features pay attention to the parts of video features that need to be responded to, thus achieving accurate fusion of cross-modal information.

[0081] For example, under the constraint of temporal relationship information, cross-attention processing of target audio features and initial video features can include: inputting the target audio features and initial video features into an attention model; the model calculates the correlation strength between the target audio features and the initial video features, i.e., the attention weight, and determines the audio parts that the initial video features need to focus on, and the video parts that the target audio features need to drive. Simultaneously, the temporal relationship information limits the calculation range of this correlation strength (e.g., video features at a certain moment can only have a strong correlation with audio features in adjacent time periods), avoiding temporally mismatched correlations. Through cross-attention processing, key video-related information (such as speech rhythm and stress) from the audio features is integrated into the initial video features, ultimately resulting in target video features that incorporate temporal synchronization information.

[0082] Under the constraint of temporal relationship information, namely the temporal correspondence between the initial video features and the target audio features, cross-attention processing of the target audio features and the initial video features is carried out by relying on the attention mechanism. This can accurately associate the matching relationship between audio and video features in the time dimension, effectively avoid the misalignment of the two in the temporal dimension, make feature fusion more accurate, and help improve the temporal consistency and synergy between the virtual image actions and audio content in the final video.

[0083] According to embodiments of this disclosure, under the constraint of temporal relationship information, cross-attention processing of target audio features and initial video features may include: using temporal relationship information to correct the attention weights obtained by cross-attention processing to obtain corrected attention weights; and performing feature fusion based on the initial video features, target audio features, and corrected attention weights to obtain target video features.

[0084] Attention weights are the core output of cross-attention processing. They are a set of values ​​that quantify the degree of correlation between the initial video features and the target audio features. The higher the value, the stronger the correlation between the corresponding audio and video features, and the more likely the model will prioritize fusing this information in subsequent iterations.

[0085] The revised attention weights are weights that are adjusted based on the attention weights and combined with temporal relationship information.

[0086] Using temporal relationship information to correct the attention weights obtained through cross-attention processing can include: reducing the weights of attention weights that do not conform to the time correspondence rules specified by the temporal relationship information, and keeping the weights that conform to the temporal rules unchanged, thus obtaining the corrected attention weights.

[0087] For example, if the correlation weight between the video features at time t in the initial video features and the target audio features at time t+30 is too high, then the weight representing that position does not conform to the requirements of temporal relationship information, and the value of that weight needs to be reduced.

[0088] In some embodiments, when the temporal relationship information is an attention mask, modifying the attention weights obtained through cross-attention processing using the temporal relationship information may include multiplying the attention mask by the attention weights to obtain the modified attention weights.

[0089] By using the temporal relationship information to correct the attention weights, deviations in the attention weights that do not conform to the time correspondence rules can be eliminated, making the weight allocation more in line with the actual temporal logic.

[0090] Feature fusion based on initial video features, target audio features, and corrected attention weights may include: weighting the target audio features using the corrected attention weights to obtain weighted audio features; and fusing the initial video features with the weighted audio features.

[0091] Feature fusion processing of initial video features and weighted audio features can include: incorporating audio-driven information that conforms to temporal rules into the initial video features through vector addition, element-wise multiplication, etc., and finally obtaining target video features that can be used for subsequent video generation.

[0092] By employing a temporal correspondence between initial video features and target audio features, the attention weights generated by cross-attention processing are specifically modified to obtain more temporally logical modified attention weights. Then, using the modified attention weights as the core, a technique for precise feature fusion combining initial video features and target audio features can improve the matching accuracy of audio and video features in the temporal dimension, avoid temporal misalignment caused by attention weight allocation deviations, and help improve the temporal synchronization of lip movements, actions, and audio content in subsequent virtual character speech videos generated based on target video feature decoding, thereby further enhancing the naturalness and accuracy of video generation.

[0093] The following is combined with Figure 4 The method for generating target video features under the constraint of temporal relationship information is further explained.

[0094] Figure 4 The diagram illustrates a method for generating target video features according to an embodiment of the present disclosure.

[0095] like Figure 4As shown, the target video feature generation method 400 of this embodiment may include: determining an attention weight 430 for the target audio feature 420 based on the similarity between the initial video feature 419 and the target audio feature 420; then correcting the attention weight 430 using temporal relationship information 440 to obtain a corrected attention weight 450; then processing the target audio feature 420 using the corrected attention weight 450 to obtain a weighted audio feature 460; and performing feature fusion processing on the initial video feature 410 and the weighted audio feature 460 to obtain the target video feature 470.

[0096] This technical solution employs a two-pronged approach: first, it preliminarily determines the attention weights for the target audio features based on the similarity between initial video features and target audio features, ensuring the basic rationality of feature association; then, it corrects these attention weights using temporal relationship information, eliminating weight deviations that do not conform to temporal logic, resulting in a precise and corrected attention weight. This achieves dual optimization of similarity-based weighting and temporal correction, ensuring that the attention weights match the relevance of feature content while strictly adhering to the temporal correspondence. This enhances the coupling and temporal synchronization of the fused target video features between audio and video information, providing a high-quality feature foundation for subsequent decoding and generation of virtual avatar speech videos. It effectively reduces lip-syncing, motion, and audio misalignment issues, enhancing the naturalness and realism of the video footage.

[0097] In some embodiments, when the target audio is long, it is often segmented to avoid poor audio-visual accuracy caused by the difficulty of long-term feature matching, while reducing the computational complexity of feature processing and improving model running efficiency. The following will combine... Figures 5-7 The method for generating long-time sequence videos will be further explained.

[0098] According to embodiments of this disclosure, the method further includes: segmenting the target audio into multiple audio segments and extracting audio segment features from the audio segments.

[0099] Dividing target audio into multiple audio segments can include: dividing a complete target audio (such as a complete speech audio) into multiple consecutive and non-overlapping audio segments along the time dimension. For example, dividing a 10-second audio segment into 20 independent audio segments of 0.5 seconds each, with each segment corresponding to the content of the audio within a specific short time period.

[0100] As mentioned above, the target audio features can contain a sequence of high-dimensional feature vectors corresponding to each audio segment (e.g., every 20 milliseconds). Audio segment features are feature information extracted separately for each audio segment, that is, a sequence of high-dimensional feature vectors corresponding to one audio segment in the target audio features. Audio segment features focus on key audio information within a single short time period (such as the spectral features, speech rhythm, tone changes, etc. of the segment), and also contain contextual information.

[0101] After segmenting the target audio into multiple audio segments, each audio segment is used as a processing unit to generate a corresponding video segment. In other words, the task of generating target video features from the target audio is broken down into multiple subtasks, each processing one audio segment to obtain a video segment feature; then, the video segment features are concatenated to form the target video feature.

[0102] Based on this, the target audio features and initial video features are processed using an attention mechanism to obtain the target video features, including: for any audio segment feature, processing the target audio segment feature and initial video features using an attention mechanism to obtain the video segment feature for any audio segment feature; and concatenating the video segment features of multiple audio segment features according to the time dimension to obtain the target video features.

[0103] It should be noted that the processing method for any audio segment feature and initial video feature based on the attention mechanism is the same as the processing method for target audio feature and initial video feature based on the attention mechanism described above. The difference is that the input features are audio segment features and initial video features.

[0104] For example, for any audio segment feature, processing the audio segment feature and the initial video feature based on an attention mechanism to obtain a video segment feature for that audio segment feature may include: taking a single audio segment feature and the initial video feature as input, calculating the correlation strength between them through an attention mechanism; fusing the audio segment feature and the initial video feature based on the correlation strength to generate a video segment feature that corresponds only to the current audio segment; repeating this operation to generate corresponding video segment features for all audio segment features, ensuring that each audio segment feature has a matching video segment feature.

[0105] After generating corresponding video segment features for each audio segment feature, the corresponding video segment features are sequentially concatenated according to the temporal order of the audio segments in the target audio (e.g., the 1st audio segment → the 2nd audio segment → ... → the Nth audio segment). The resulting complete feature sequence is the target video feature that covers the entire audio duration and is temporally coherent.

[0106] By segmenting the target audio, the attention mechanism can focus on the precise association between short audio segments and the initial video features, avoiding temporal misalignment and loss of details when processing long audio directly. At the same time, splicing according to the time dimension ensures the continuity of the target video features, laying the foundation for subsequent decoding and generation of a video of a virtual avatar speaking based on the target audio. This helps improve the temporal synchronization and dynamic naturalness of the virtual avatar's actions, lip movements, and audio content in the video, while also reducing the computational complexity of long temporal feature processing and improving overall processing efficiency.

[0107] To ensure the consistency and stability of the virtual avatar's appearance, when generating the video segment features for the current clip, it's necessary to consider not only the current audio clip and the initial avatar, but also previously generated video clips. For example, historical video features can be incorporated into the generation of current video segment features during the audio clip feature generation process.

[0108] According to embodiments of this disclosure, processing any audio segment feature and an initial video feature to obtain a video segment feature for any audio segment feature includes: determining historical video features for any audio segment feature based on the temporal information of any audio segment feature, wherein the historical video features include video segment features determined based on audio segment features located before the temporal sequence of any audio segment feature; and processing any audio segment feature and a first spliced ​​video feature obtained by splicing the historical video feature and the initial video feature to obtain a video segment feature for any audio segment feature.

[0109] The temporal information of audio segment features can be the temporal position information of a single audio segment within the overall target audio, used to clarify the sequential order between segments. For example, the third audio segment may be after the second audio segment and before the fourth audio segment.

[0110] Historical video features are the video segment features that correspond to the audio segments preceding the current audio segment in time. For example, when processing the 5th audio segment, its historical video features can include the video segment features corresponding to the 1st to 4th audio segments, which record the dynamic state of the virtual character before the current video segment.

[0111] The first spliced ​​video feature can be a comprehensive feature formed by splicing historical video features with initial video features. It contains the original appearance information of the virtual image and integrates its past dynamic change information.

[0112] For example, if you need to generate a 30-second video, you can divide the 30-second video into 10 segments. When generating the first video segment, the only reference information available is the reference image information (i.e., the initial video features), and the historical video features are empty at this time. When generating subsequent video segments, such as the nth video segment, you can use the information from the (n-1)th generated video segment as the historical video features.

[0113] Figure 5 The illustration shows a schematic diagram of splicing historical video features and initial video features to form a first spliced ​​video feature according to an embodiment of the present disclosure.

[0114] like Figure 5 As shown, historical video features 510 and prompting video features 520, which include virtual image features and random noise, are spliced ​​together in the time dimension to obtain a first spliced ​​video feature 530, which includes virtual image features, random noise, and historical video features.

[0115] After generating the first spliced ​​video features for the current audio segment features, the current audio segment features and the first spliced ​​video features are processed to obtain video segment features for the current audio segment features. For example, processing the current audio segment features and the first spliced ​​video features may include performing cross-attention processing on the current audio segment features and the first spliced ​​video features.

[0116] The process of performing cross-attention processing on the current audio segment features and the first spliced ​​video features is the same as the process of performing cross-attention processing on the target audio features and the initial video features, and will not be repeated here.

[0117] By introducing historical video features, each newly generated video segment is not only driven by the current audio segment but also constrained by past dynamic states. This ensures that the virtual character's movements (such as lip movements and head rotations) are continuous and smooth in the time dimension (for example, when transitioning from the lip shape of the "a" sound to the lip shape of the "i" sound, the natural transition process in between is preserved), avoiding abrupt jumps where the mouth is open in the previous frame and suddenly closed in the next frame, significantly improving the smoothness and realism of the generated video.

[0118] The following is combined with Figure 6 This diagram illustrates the generation of target video features after the target audio is segmented into multiple audio segments.

[0119] Figure 6 A schematic diagram illustrating the generation of target video features according to an embodiment of the present disclosure is shown.

[0120] like Figure 6As shown, this embodiment includes two audio segments, namely a first audio segment and a second audio segment, with the first audio segment preceding the second audio segment in time sequence; similarly, the target audio features include features of the first audio segment and features of the second audio segment. The generation of target video features in this embodiment includes two generation tasks: a first generation task 610 for the first audio segment and a second generation task 620 for the second audio segment.

[0121] First, perform the first generation task 610: perform self-attention calculation on the initial video features 611 based on the self-attention mechanism to obtain the intermediate video features 612; then perform cross-attention calculation on the intermediate video features 612 and the first audio segment features 613 to obtain the first video segment features 614 for the first audio segment features 613.

[0122] Then, the second generation task 620 is executed: the first video segment feature 614 is concatenated with the initial video feature 611 to obtain the first concatenated video feature 622; then, self-attention calculation is performed on the first concatenated video feature 622 to obtain the intermediate concatenated video feature 623; then, cross-attention calculation is performed on the intermediate concatenated video feature 623 and the second audio segment feature 624 to obtain the second video segment feature 625 for the second audio segment feature 624. Finally, the first video segment feature 614 and the second video segment feature 625 are concatenated in chronological order to obtain the target video feature 630.

[0123] By strengthening the internal correlation of video features through self-attention computation and achieving accurate matching of audio and video features through cross-attention computation, and by allowing the features of the next video segment to inherit the dynamic information of the features of the previous video segment, it not only ensures the temporal synchronization of each audio segment and the corresponding video segment features, but also ensures the dynamic coherence between video segment features. This effectively improves the naturalness, fluency, and temporal accuracy of the subsequently generated virtual avatar speaking videos. At the same time, the segmented processing mode also provides a scalable and efficient path for multi-segment audio-driven video generation.

[0124] When processing long videos, the accumulated amount of historical video features is substantial, consuming significant memory and increasing computational burden. Therefore, historical information can be compressed using preset compression rules, and the compressed historical video features can then be used in calculations, thus achieving a balance between efficiency and effectiveness.

[0125] According to embodiments of this disclosure, the above method further includes: compressing historical video features according to preset compression rules to obtain compressed historical video features;

[0126] The preset rules can be pre-defined rules or algorithms for simplifying historical video features (such as feature dimensionality reduction, keyframe extraction, information aggregation, etc.). The purpose is to remove redundant information while retaining the core dynamic information (such as key poses and lip-syncing trends) that is valuable for generating features of the current video segment.

[0127] In some embodiments, compressing historical video features according to preset compression rules may include: compressing historical video features in the temporal dimension.

[0128] For example, compressing historical video features in the time dimension can include: simplifying the time series length of historical video features, such as reducing the features of 10 consecutive historical video frames to the features of 3 frames through aggregation, sampling, etc. The core is to reduce the number of historical video features in the time dimension and reduce the data scale.

[0129] Specifically, video features that are far removed from the current time in the historical video features are removed, and only video features that are closer to the current time are retained. Specifically, the compressed historical video features can be those generated in the previous stage before the current generation stage. For example, if the current generation stage is stage n, then the compressed historical video features for stage n can be those generated in stage (n-1).

[0130] Based on the compression of historical video features, any audio segment feature, as well as the first spliced ​​video feature obtained by splicing historical video features and initial video features, are processed to obtain video segment features for any audio segment feature. This can include: processing any audio segment feature, as well as the second spliced ​​video feature obtained by splicing compressed historical video features and initial video features, to obtain video segment features for any audio segment feature.

[0131] Compression effectively reduces the amount of data in historical video features, reducing computational resource consumption during subsequent processing and improving processing efficiency while preserving key dynamic information. Simultaneously, generating video segment features based on a second spliced ​​video feature, formed by combining compressed historical video features with initial video features, ensures the continuity of the virtual character's dynamic changes and allows for precise matching of audio segment features with video features. This improves processing efficiency while guaranteeing the naturalness and temporal synchronization of subsequently generated videos.

[0132] After compressing historical video features along the temporal dimension to obtain first-level compressed historical video features, since the first-level compressed historical video still includes frame features of at least one historical video frame, further compression can be performed on this basis.

[0133] In other embodiments, further compression of the first-level compressed historical video may include: compressing the frame features of at least one historical video frame based on the time distance between the time point of the historical video frame and the time point represented by the time sequence information, to obtain compressed historical video features.

[0134] By initially compressing the temporal dimension, the temporal length of historical video features is reduced, thus decreasing the data volume. Subsequent secondary compression, combined with temporal distance, can more accurately retain the features of historical video frames closely related to the current time sequence, while eliminating redundant information with weak correlation. This significantly reduces the scale of historical video feature data and improves the efficiency of subsequent feature processing, while maximizing the retention of historical dynamic information valuable for the generation of current video segment features. This ensures the continuity of dynamic changes such as virtual character actions, making the subsequently generated video both efficient and natural and smooth.

[0135] The following is combined with Figure 7 The process of compressing historical videos multiple times to obtain compressed historical video features will be further explained.

[0136] Figure 7 A schematic diagram illustrating the determination of compressed historical video features according to an embodiment of the present disclosure is shown.

[0137] like Figure 7 As shown, the method 700 for determining compressed historical video features in this embodiment includes: compressing historical video features 710 in the temporal dimension to obtain first-level compressed historical video features 720, wherein the first-level compressed historical video 720 may include frame features of at least one historical video frame; for the frame features of any historical video frame, determining a compression weight 760 for any historical video frame based on the distance weight mapping relationship 750 and the time distance 740 between the time point of any historical video frame and the time point represented by the temporal information 730; then, processing the frame features of at least one historical video frame in the first-level compressed historical video features 720 using the compression weights 760 of each of the at least one historical video frame to obtain compressed historical video features 770.

[0138] Timing information is the time point information corresponding to the currently processed audio segment (such as the relative time position of the 5th audio segment, which is the reference time point for judging the correlation of historical video frames).

[0139] Temporal distance is the interval between the time point of each historical video frame and the time point represented by the temporal information. For example, if the historical video corresponds to the 2nd audio segment, and the current processing is of the 5th audio segment, the temporal distance is the time difference between the two audio segments. The closer the temporal distance, the higher the reference value of that historical video frame for the generation of the current video.

[0140] The distance-weight mapping relationship is a pre-defined rule that corresponds time distance to compression weight; the closer the distance, the higher the weight. For example, a compression weight of 1.0 corresponds to a time distance within 0.5 seconds, a compression weight of 0.5 corresponds to a time distance between 1 and 2 seconds, and a compression weight of 0.1 corresponds to a time distance of more than 2 seconds.

[0141] For example, the current video segment to be generated is: t~t+80; a first compression weight of 1.0 is applied to the video frames in t-4~t-1; a second compression weight of 0.5 is applied to the video frames in t-12~t-51; a third compression weight of 0.3 is applied to the video frames in t-72~t-13; and a fourth compression weight of 0.1 is applied to the video frames in t-144~t-72.

[0142] Based on the initial reduction of historical video feature data through temporal compression, the correlation between historical video frames and the current processing task is accurately quantified by time distance. Frame feature information is retained differentially based on compression weights. While retaining historical dynamic information that is valuable for generating features of the current video segment, the data scale of historical video features is further reduced, improving the efficiency of subsequent operations such as feature stitching and attention calculation.

[0143] In addition to weighting the features of historical videos, weighting can also be applied to the features of virtual avatars.

[0144] According to embodiments of this disclosure, the method further includes: weighting the virtual avatar features according to reference weights for the reference image to obtain weighted virtual avatar features; and concatenating the weighted virtual avatar features with random noise to obtain weighted initial video features.

[0145] By using reference weights to strengthen key features of the virtual avatar (such as lip movements and facial expressions), the subsequent fusion with target audio features can more accurately focus on the visual dynamics required for audio-driven operation, reduce the interference of non-key features on the core matching process, and improve the accuracy of audio-visual synchronization.

[0146] When weighting the virtual avatar features, processing any audio segment feature and a second spliced ​​video feature (composed of compressed historical video features and weighted initial video features) to obtain a video segment feature for any audio segment feature can include: processing any audio segment feature and a third spliced ​​video feature (composed of compressed historical video features and weighted initial video features) to obtain a video segment feature for any audio segment feature.

[0147] In some embodiments, the video generation process may utilize a model with a latent diffusion model as its core architecture to process target audio features and initial video features.

[0148] Diffusion model: A representative model in the field of image generation, it encodes multiple types of information and inputs them into the diffusion model, which then decodes them to generate smooth and realistic images. Latent diffusion model: A variation of the diffusion model, it applies the transformer architecture to the diffusion model.

[0149] The following is combined with Figure 8 This paper provides a detailed explanation of the video generation process using a video generation model with a latent diffusion model as its core architecture.

[0150] Figure 8 The schematic diagram illustrates the architecture of a video generation model based on an example of this disclosure.

[0151] like Figure 8 As shown, the video generation model of this embodiment includes a speech encoding module 810, an image encoding module 820, a feature processing module 830, and a decoding module 840.

[0152] The speech coding module 810 can be a Wav2Vec model, used to encode the target speech 811 to obtain the target speech features 812.

[0153] The image encoding module 820 can be a 3D VAE encoder, used to encode the reference image 821 to obtain the virtual image features 822.

[0154] After obtaining the target speech features 812, the target audio features 812 are transformed by projection and other operations, and then input into the feature processing module 830.

[0155] Meanwhile, after obtaining the virtual avatar feature 822, the virtual avatar feature 822 is temporally extended to obtain the extended virtual avatar feature 823; then the extended virtual avatar feature 823 is concatenated with random noise 824 to obtain the initial video feature, and the initial video feature is input into the feature extraction module 830.

[0156] The feature processing module 830 can be a dit module, which is used to perform attention calculation on the input target audio features 812 and initial video features, and perform N iterations of denoising to generate target video features 831.

[0157] The decoding module 840 can be a 3D VAE decoder, used to decode the target video features 831, reconstruct video frames, and obtain the target video 841.

[0158] The dit module can include a series of Transformer blocks.

[0159] Figure 9A schematic diagram of the structure of the dit module according to an embodiment of the present disclosure is shown.

[0160] like Figure 9 As shown, the dit module in this embodiment includes, in sequence, a normalization layer 910, a self-attention layer 920, a normalization layer 930, a cross-attention layer 940, a normalization layer 950, and a feedforward neural network layer 960.

[0161] Normalization layer 910 is used to normalize the initial video features of the input, mapping the initial video features to a uniform numerical range (such as 0-1 or a standard normal distribution), eliminating the differences in dimensionality or numerical distribution between different feature dimensions, and enabling subsequent layers to process the data more stably and efficiently.

[0162] Self-attention layer 920: Focuses on the correlations within the features themselves, used to calculate attention weights between different parts of the initial video features, strengthening the representation of key information within the features. For example, when processing initial video frame features, it highlights the correlations between action changes between adjacent frames.

[0163] Normalization layer 930: Normalizes the features processed by the self-attention layer 920, stabilizes the data distribution, and provides a more standardized input for the next layer of processing.

[0164] Cross-attention layer 950: It receives the initial video features output by the normalization layer 930 and the target video features processed by projection, and calculates the attention weights between the features of these two different modalities, explores the correlation between audio and video features, and realizes the fusion and alignment of cross-modal features.

[0165] Normalization layer 950: Used to normalize the cross-modal fusion features output by the cross-attention layer 950 again to ensure data stability.

[0166] Feedforward neural network layer 960: Used to perform further nonlinear transformation and information extraction on the features fused by the cross-attention layer 950, enhance the expressive power of the features, and extract more abstract and discriminative feature representations.

[0167] The features output by the feedforward neural network layer 960 are the denoised video features. Then, the denoised video features are input into the normalization layer 910 for denoising again. This process is repeated N times to obtain the completely denoised video features, i.e., the target video features.

[0168] By accurately extracting audio temporal features using Wav2Vec, combining the subtle interaction between audio and image features through self-attention and cross-attention in Transformer, and enhancing intermodal correlations using LoRA, high-precision alignment between lip movements and audio is achieved, with excellent lip-sync performance.

[0169] In addition, the introduction of reference images provides richer image and posture references for motion generation. The Transformer, combined with LoRA's cross-modal fusion mechanism, can deeply capture information such as rhythm and emotion in the audio and map it into natural limb and head movements, making the generated motion smooth and in line with the audio expression.

[0170] In addition, by performing feature interaction and reconstruction in low-dimensional space through 3D VAE, the inter-frame feature drift is effectively reduced; a long-distance strong dependency of single-image character features in temporal generation is established, ensuring the consistency of character appearance over long time series, supporting stable generation of long videos, and meeting the needs of more long content creation scenarios.

[0171] According to embodiments of this disclosure, before executing the video generation method described above, it is necessary to train the video generation model.

[0172] First, obtain the training dataset. The training samples in the training dataset can include training videos, sample audio corresponding to the training videos, and sample reference images.

[0173] The sample audio features for the sample audio and the sample virtual image features for the sample reference image are input into the video generation model to output the sample video. The parameters of the video generation model are adjusted by using the difference between the sample video and the training video until the iteration stopping condition is met.

[0174] It should be noted that a low-rank adaptation (LoRA) matrix is ​​introduced into the dit module of the video generation model to improve the training speed of the model.

[0175] Adjusting the parameters of the video generation model can be achieved by adjusting the LoRA matrix.

[0176] According to embodiments of this disclosure, before obtaining the training dataset, data filtering is also included to obtain a batch of high-quality human-centered training datasets.

[0177] Data filtering includes basic quality filtering and advanced filtering to adapt to single-map solutions.

[0178] Basic quality screening can include: using person detection algorithms to remove videos from the database that are not centered on people, as well as videos featuring multiple people. Using image quality detection and blurring algorithms to remove low-quality or blurry videos from the database. Using audio-visual synchronization detection algorithms to remove videos from the database that are out of sync with the audio. Further removing low-resolution data or data with small proportions of people from the database by setting targets such as resolution / screen aspect ratio.

[0179] Advanced screening for single-image adaptation solutions can include: using head pose estimation algorithms to remove data from the database showing large head turns, ensuring that the training data features people with forward-facing postures. Using optical flow motion detection algorithms to remove data showing fast-moving people from the database, making model learning easier.

[0180] After the above data filtering, a high-quality video dataset with millions of records can be obtained, which can be used as the training dataset for this public single-graph driven scheme.

[0181] It should be noted that the person detection algorithm (e.g., RetinaNet), image quality detection & blur algorithm (e.g., Laplacian Variance), audio-visual synchronization detection algorithm (e.g., SyncNet), head pose estimation algorithm (e.g., HopeNet), and optical flow motion detection algorithm (Farneback) all use existing algorithms, and this disclosure will not elaborate on them.

[0182] Based on the above video generation method, this disclosure also provides a video generation apparatus. The following will be combined with... Figure 10 The device is described in detail.

[0183] Figure 10 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0184] like Figure 10 As shown, the video generation apparatus 1000 of this embodiment includes: an extraction module 1010, a processing module 1020, and a decoding module 1030.

[0185] The extraction module 1010 is used to extract the target audio features of the target audio and the virtual image features of the virtual image in the reference image. In one embodiment, the extraction module 1010 can be used to perform the operation S210 described above, which will not be repeated here.

[0186] Processing module 1020 is used to process the target audio features and initial video features based on an attention mechanism to obtain target video features. The initial video features are formed by concatenating the virtual image features with random noise. In one embodiment, processing module 1020 can be used to perform the operation S220 described above, which will not be repeated here.

[0187] The decoding module 1030 is used to decode the target video features to obtain the target video, wherein the target video includes video frames of the virtual image based on the target audio speech expression. In one embodiment, the decoding module 1030 can be used to perform the operation S230 described above, which will not be repeated here.

[0188] According to an embodiment of this disclosure, the processing module 1020 includes a cross-attention processing submodule.

[0189] The cross-attention processing submodule is used to perform cross-attention processing on the target audio features and the initial video features under the constraint of temporal relationship information to obtain the target video features. The temporal relationship information indicates the temporal correspondence between the initial video features and the target audio features.

[0190] According to embodiments of this disclosure, the cross-attention processing submodule includes a correction unit and a feature fusion unit.

[0191] The correction unit is used to correct the attention weights obtained through cross-attention processing by utilizing temporal relationship information, so as to obtain the corrected attention weights.

[0192] The feature fusion unit is used to fuse features based on the initial video features, target audio features, and corrected attention weights to obtain the target video features.

[0193] According to embodiments of this disclosure, the correction unit includes a determining subunit and a correction subunit.

[0194] A sub-unit is defined to determine the attention weights for the target audio features based on the similarity between the initial video features and the target audio features.

[0195] The correction subunit is used to correct the attention weights using temporal relationship information, thus obtaining the corrected attention weights.

[0196] According to embodiments of this disclosure, the feature fusion unit includes a processing subunit and a feature fusion subunit.

[0197] The processing subunit is used to process the target audio features using the corrected attention weights to obtain weighted audio features.

[0198] The feature fusion subunit is used to perform feature fusion processing on the initial video features and the weighted audio features.

[0199] According to embodiments of this disclosure, the above-described apparatus 1000 further includes a segmentation module.

[0200] The segmentation module is used to segment the target audio into multiple audio segments and extract the audio segment features of the audio segments.

[0201] According to embodiments of this disclosure, the processing module 1020 includes a first processing submodule and a splicing submodule.

[0202] The first processing submodule is used to process any audio segment feature and the initial video feature based on an attention mechanism to obtain the video segment feature for any audio segment feature.

[0203] The splicing submodule is used to splice the video segment features of multiple audio segments according to the time dimension to obtain the target video features.

[0204] According to embodiments of this disclosure, the first processing submodule includes: a first determining unit and a first processing unit.

[0205] The first determining unit is configured to determine historical video features for any audio segment feature based on the temporal information of any audio segment feature, wherein the historical video features include video segment features determined based on audio segment features that precede any audio segment feature in the temporal sequence.

[0206] The first processing unit is used to process any audio segment feature and the first spliced ​​video feature obtained by splicing historical video features and initial video features to obtain video segment features for any audio segment feature.

[0207] According to embodiments of this disclosure, the above-described apparatus 1000 further includes a compression module.

[0208] The compression module is used to compress historical video features according to preset compression rules to obtain compressed historical video features.

[0209] According to embodiments of this disclosure, the first processing unit is further configured to process any audio segment feature and a second spliced ​​video feature obtained by splicing compressed historical video features and initial video features to obtain video segment features for any audio segment feature.

[0210] According to embodiments of this disclosure, the compression module includes: a first compression submodule and a second compression submodule.

[0211] The first compression submodule is used to compress historical video features in the temporal dimension to obtain first-level compressed historical video features, wherein the first-level compressed historical video includes frame features of at least one historical video frame.

[0212] The second compression submodule is used to compress the frame features of at least one historical video frame based on the time distance between the time point of the historical video frame and the time point represented by the time sequence information, so as to obtain compressed historical video features.

[0213] According to embodiments of this disclosure, the second compression submodule includes: a second determining unit and a second processing unit.

[0214] The second determining unit is used to determine the compression weight for any historical video frame based on the frame characteristics of any historical video frame and the time distance between the time point of any historical video frame and the time point represented by the time sequence information.

[0215] The second processing unit is used to process the frame features of at least one historical video frame using the compression weights of each historical video frame to obtain compressed historical video features.

[0216] According to embodiments of this disclosure, the above-mentioned device 1000 further includes: a weighted processing module and a splicing module.

[0217] The weighted processing module is used to weight the virtual image features according to the reference weights for the reference image to obtain weighted virtual image features.

[0218] The splicing module is used to splice the weighted virtual image features with random noise to obtain the weighted initial video features.

[0219] According to embodiments of this disclosure, the first processing unit is further configured to process any audio segment feature and a third spliced ​​video feature obtained by splicing compressed historical video features and weighted initial video features to obtain video segment features for any audio segment feature.

[0220] It should be noted that the video generation device part in the embodiments of this disclosure corresponds to the video generation method part in the embodiments of this disclosure. For a detailed description of the video generation device part, please refer to the video generation method part, which will not be repeated here.

[0221] Embodiments of this disclosure also provide an AI (Artificial Intelligence) agent applied to work order processing, configured to execute the large model-based work order processing method described above.

[0222] Figure 11 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0223] In embodiments of this disclosure, the von Neumann architecture in modern computer theory is inspired, such as... Figure 11 As shown, the AI ​​agent 1100 may include five core modules: input module 1111, control module 1120, storage module 1130, computing module 1140, and output module 1150.

[0224] The input module 1111 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 1100 can understand and process. The input module 1111 is the primary link for the AI ​​agent 1100 to interact with the outside world. It enables the AI ​​agent 1100 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0225] In the example, input module 1111 can input the target audio features and initial video features described above.

[0226] In the example, the control module 1120 is the core support for the AI ​​agent 1100's ability to handle complex tasks. The control module 1120 can execute the video generation method described above.

[0227] In the example, the control module 1120 will continuously interact with the storage module 1130, the arithmetic module 1140, and / or the output module 1150 during operation. However, it should be noted that in the embodiments of this disclosure, the control module 1120 initiates communication with the storage module 1130, the arithmetic module 1140, and / or the output module 1150 as a single initiator, and there is no communication coupling between the storage module 1130, the arithmetic module 1140, and the output module 1150.

[0228] In the example, the performance of the control module 1120 is closely related to the large model on which the AI ​​agent 1100 is based. To fully leverage the capabilities of the large language model, the internal structure of the control module 1120 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0229] The storage module 1130 is responsible for remembering information such as historical dialogues and event streams. As mentioned above, target audio features, virtual avatar features, target video features, and target videos can be included in the storage module 1130.

[0230] In the example, after acquiring the target video and reference image, the AI ​​agent 1100 can extract the target audio features and the virtual image features of the virtual avatar in the reference image, and store them in the storage module 1130. The AI ​​agent 1100 can then determine the target audio features and the virtual image features of the virtual avatar in the reference image from the storage module 1130 and feed them back to the control module 1120. The control module 1120 can then process the target audio features and initial video features based on an attention mechanism to obtain the target video features, which are then passed to the output module 1150.

[0231] The arithmetic module 1140 can be viewed as a predefined tool library. Arithmetic units, etc., used to determine the jump parameters and / or search parameters as described above, can be included in the arithmetic module 1140.

[0232] In the example, when the AI ​​agent 1100 needs to extract target video features from the target video and virtual image features from the reference image, it can call the relevant processor from the processing module 1140 and feed it back to the control module 1120. Then, the control module 1120 can use the fed-back processor to generate target video features and virtual image features. It can also process the target audio features and initial video features based on an attention mechanism to obtain the target video features, which are then transmitted to the output module 1150. It is understandable that although large language models have excellent language understanding and generation capabilities, like humans, the tasks they can solve without any tools are very limited. When the AI ​​agent 1100 is given the ability to call tools, it can perform tasks such as using a calculator to perform mathematical calculations, using Python to perform data analysis, and using expert models to perform person recognition in images.

[0233] In the example, output module 1150 can output the target video features or target video described above.

[0234] The AI ​​agent 1100 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0235] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0236] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0237] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0238] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0239] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0240] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0241] Multiple components in device 1200 are connected to input / output (I / O) interface 1205, including: input unit 1206, such as a keyboard, mouse, etc.; output unit 1207, such as various types of displays, speakers, etc.; storage unit 1208, such as a disk, optical disk, etc.; and communication unit 1209, such as a network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0242] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as video generation methods. For example, in some embodiments, the video generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the video generation method by any other suitable means (e.g., by means of firmware).

[0243] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0244] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0245] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0246] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0247] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0248] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0249] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0250] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: Extract the target audio features and the virtual image features from the reference image; Based on the attention mechanism, the target audio features and the initial video features are processed to obtain the target video features, wherein the initial video features are formed by splicing the virtual image features with random noise; as well as The target video features are decoded to obtain the target video, wherein the target video includes video frames of the virtual character based on the target audio speech expression.

2. The method according to claim 1, wherein, The process of processing the target audio features and initial video features based on the attention mechanism to obtain target video features includes: Under the constraint of temporal relationship information, cross-attention processing is performed on the target audio features and the initial video features to obtain the target video features, wherein the temporal relationship information indicates the temporal correspondence between the initial video features and the target audio features.

3. The method according to claim 2, wherein, The cross-attention processing of the target audio features and the initial video features under the constraint of temporal relationship information includes: Using the temporal relationship information, the attention weights obtained through cross-attention processing are corrected to obtain the corrected attention weights. The target video features are obtained by fusing features based on the initial video features, the target audio features, and the corrected attention weights.

4. The method according to claim 3, wherein, The step of using the temporal relationship information to correct the attention weights obtained through cross-attention processing to obtain corrected attention weights includes: Based on the similarity between the initial video features and the target audio features, an attention weight for the target audio features is determined; The attention weights are corrected using the temporal relationship information to obtain the corrected attention weights; The feature fusion based on the initial video features, the target audio features, and the corrected attention weights includes: The target audio features are processed using the corrected attention weights to obtain weighted audio features; The initial video features and the weighted audio features are then fused together.

5. The method according to claim 1, further comprising: The target audio is divided into multiple audio segments, and the audio segment features of the audio segments are extracted; The process of processing the target audio features and initial video features based on an attention mechanism to obtain the target video features includes: For any audio segment feature, based on an attention mechanism, the audio segment feature and the initial video feature are processed to obtain a video segment feature for the audio segment feature. The video segment features of the multiple audio segments are concatenated according to the time dimension to obtain the target video features.

6. The method according to claim 5, wherein, The step of processing the any audio segment feature and the initial video feature to obtain the video segment feature for the any audio segment feature includes: Based on the temporal information of any audio segment feature, historical video features for any audio segment feature are determined, wherein the historical video features include video segment features determined based on audio segment features that precede the temporal information of any audio segment feature; The video segment features for any audio segment feature and the first spliced ​​video feature obtained by splicing the historical video features and the initial video features are processed to obtain the video segment features for any audio segment feature.

7. The method according to claim 6, further comprising: The historical video features are compressed according to a preset compression rule to obtain compressed historical video features. The process of processing the features of any audio segment and the first spliced ​​video features obtained by splicing the historical video features and the initial video features to obtain video segment features for any audio segment features includes: The features of any audio segment and the second spliced ​​video features obtained by splicing the compressed historical video features and the initial video features are processed to obtain video segment features for any audio segment features.

8. The method according to claim 7, wherein, The step of compressing the historical video features according to a preset compression rule to obtain compressed historical video features includes: The historical video features are compressed in the temporal dimension to obtain first-level compressed historical video features, wherein the first-level compressed historical video includes frame features of at least one historical video frame. Based on the time distance between the time point of the historical video frame and the time point represented by the time sequence information, the frame features of the at least one historical video frame are compressed to obtain the compressed historical video features.

9. The method according to claim 8, wherein, The step of compressing the frame features of at least one historical video frame based on the time distance between the time point of the historical video frame and the time point represented by the time sequence information includes: For any historical video frame, based on the frame features, the compression weight for that historical video frame is determined according to the time distance between the time point of that historical video frame and the time point represented by the time sequence information. The frame features of the at least one historical video frame are processed using their respective compression weights to obtain the compressed historical video features.

10. The method according to any one of claims 7 to 9, further comprising: The virtual avatar features are weighted according to the reference weights for the reference image to obtain weighted virtual avatar features; The weighted virtual image features are concatenated with the random noise to obtain the weighted initial video features; The process of processing the features of any audio segment and the second spliced ​​video features obtained by splicing the compressed historical video features and the initial video features to obtain video segment features for any audio segment features includes: The video segment features of any audio segment and the third spliced ​​video features obtained by splicing the compressed historical video features and the weighted initial video features are processed to obtain the video segment features for any audio segment features.

11. An intelligent agent of artificial intelligence, comprising: The input module is used to receive the target audio and reference image; The processing module is configured to determine a target task based on the target audio and reference image received by the input module, determine a target large model based on the target task, and obtain a target video by calling the target large model to execute the method of any one of claims 1-10. as well as An output module is used to output the target video obtained by the processing module.

12. A video generation apparatus, comprising: The extraction module is used to extract the target audio features of the target audio and the virtual image features of the virtual image in the reference image; The processing module is used to process the target audio features and the initial video features based on an attention mechanism to obtain the target video features. The initial video features are formed by concatenating the virtual image features with random noise and then processing them to obtain the target video features. as well as A decoding module is used to decode the features of the target video to obtain the target video, wherein the target video includes video frames of the virtual image based on the target audio speech expression.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

15. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1-10 when executed by a processor.

Images