Video generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a video generation method and device, electronic equipment, a storage medium and a program product, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes of digital human, content generation based on artificial intelligence and the like. The method comprises the following steps: acquiring text features of a description text, image features of a virtual image in a reference image and audio features of audio, wherein the description text indicates action description information for driving the virtual image based on the audio; the role feature is bound with the audio feature to obtain a target audio feature, the role feature is associated with a corresponding virtual image in the reference image, and the role feature is used for indicating an association relationship between the target audio feature and the image feature; and generating a target video based on the text feature, the image feature and the target audio feature, wherein the target video comprises a plurality of video frames for making a sound according to the action description information based on the audio driving virtual image.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to computer vision, deep learning, and large-scale modeling, and can be applied to scenarios such as digital humans and AI-based content generation. More specifically, this disclosure provides a video generation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence technology, an object image can be used as the reference frame for the first frame of a video, combined with voice input to generate a video. The video footage can display the object's posture and body movements, and can be matched with the voice, thus having wide applications in content creation, intelligent interaction, remote communication, education, and other fields. Summary of the Invention

[0003] This disclosure provides a video generation method, apparatus, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, a video generation method is provided, the method comprising: acquiring text features of descriptive text, image features of a virtual image in a reference image, and audio features of audio, wherein the descriptive text indicates action description information of the virtual image driven by audio; binding character features with audio features to obtain target audio features, wherein the character features are associated with a corresponding virtual image in the reference image, and the character features are used to indicate the association between the target audio features and the image features; and generating a target video based on the text features, image features, and target audio features, wherein the target video includes multiple video frames in which the virtual image emits sound according to the action description information driven by audio.

[0005] According to another aspect of this disclosure, a training method for a video generation model is provided. The method includes: acquiring text sample features describing a text sample, image sample features of a virtual character in a reference image sample, and audio sample features of an audio sample, wherein the text sample indicates action description information of the virtual character driven by the audio sample; binding character sample features with audio sample features to obtain target audio sample features, wherein the character sample features are associated with a corresponding virtual character in the reference image sample, and the character sample features are used to indicate the association between the target audio sample features and the image sample features; generating a target video sample using the video generation model to be trained based on the text sample features, image sample features, and target audio sample features, wherein the target video sample includes multiple video frames in which the virtual character emits sound according to the action description information driven by the audio; and adjusting the parameters of the video generation model based on the loss value between the target video sample and a preset reference video to obtain a trained video generation model.

[0006] According to another aspect of this disclosure, a video generation apparatus is provided, comprising: an initial feature acquisition module for acquiring text features of descriptive text, image features of a virtual image in a reference image, and audio features of audio, wherein the descriptive text indicates action description information of the virtual image driven by audio; a target audio feature acquisition module for binding character features with audio features to obtain target audio features, wherein the character features are associated with a corresponding virtual image in the reference image, and the character features are used to indicate the association between the target audio features and the image features; and a target video generation module for generating a target video based on the text features, image features, and target audio features, wherein the target video includes multiple video frames in which the virtual image emits sound according to the action description information driven by audio.

[0007] According to another aspect of this disclosure, a training apparatus for a video generation model is provided. The apparatus includes: an initial sample feature acquisition module, used to acquire text sample features of a descriptive text sample, image sample features of a virtual character in a reference image sample, and audio sample features of an audio sample, wherein the descriptive text sample indicates action description information driven by the audio sample; a target audio sample feature acquisition module, used to bind character sample features with audio sample features to obtain target audio sample features, wherein the character sample features are associated with a corresponding virtual character in the reference image sample, and the character sample features are used to indicate the association between the target audio sample features and the image sample features; a target video sample generation module, used to generate target video samples based on the text sample features, image sample features, and target audio sample features using the video generation model to be trained, wherein the target video samples include multiple video frames in which the virtual character emits sound according to the action description information driven by the audio; and a video generation model acquisition module, used to adjust the parameters of the video generation model based on the loss value between the target video sample and a preset reference video to obtain a trained video generation model.

[0008] According to another aspect of this disclosure, an intelligent agent is provided, comprising: an input module for receiving input information, the input information including descriptive text, reference images, and audio; a processing module for obtaining output information by invoking a large model based on the input information received by the input module; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to this disclosure.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided according to this disclosure.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to this disclosure.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is an exemplary system architecture diagram of a video generation method and apparatus applicable according to an embodiment of the present disclosure;

[0015] Figure 2 This is a flowchart of a video generation method according to an embodiment of the present disclosure;

[0016] Figure 3 This is a schematic diagram of a video generation method according to another embodiment of the present disclosure;

[0017] Figure 4 This is a schematic diagram of a video generation method according to another embodiment of the present disclosure;

[0018] Figure 5 This is a flowchart of a video generation model training method according to an embodiment of the present disclosure;

[0019] Figure 6 This is a schematic diagram illustrating the acquisition of a reference video according to an embodiment of the present disclosure;

[0020] Figure 7 This is a schematic diagram of the overall framework of a video generation method according to an embodiment of the present disclosure;

[0021] Figure 8 This is a block diagram of a video generation apparatus according to an embodiment of the present disclosure;

[0022] Figure 9 This is a block diagram of a video generation model training apparatus according to an embodiment of the present disclosure;

[0023] Figure 10 This is a structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure;

[0024] Figure 11 This is a block diagram of an electronic device to which a video generation method can be applied, according to an embodiment of the present disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1 This is a schematic diagram of an exemplary system architecture for applying a video generation method and apparatus according to an embodiment of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a data acquisition device 101, a network 102, and a server 103. The data acquisition device 101 may acquire descriptive text, reference images, and multiple audio files, and the network 102 serves as a medium for providing a communication link between the data acquisition device 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] The data acquisition device 101 can be any device with a camera that supports image or text acquisition, including but not limited to cameras, smartphones, tablets, and drones.

[0029] Server 103 can be a server that provides various services, such as a server that processes reference images acquired by acquisition device 101. Server 103 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.

[0030] It should be noted that the above are merely examples, and this disclosure is not limited thereto. The video generation method provided in the embodiments of this disclosure can generally be executed by server 103. Accordingly, the video generation device provided in the embodiments of this disclosure can generally be located in server 103. The video generation method provided in the embodiments of this disclosure can also be executed by a server or server cluster that is different from server 103 but can communicate with acquisition device 101. Accordingly, the video generation device provided in the embodiments of this disclosure can also be located in a server or server cluster that is different from server 103 but can communicate with acquisition device 101.

[0031] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0032] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, application, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0033] Figure 2 This is a flowchart of a video generation method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the method 200 may include operations S210 to S230.

[0035] In operation S210, the text features of the descriptive text, the image features of the virtual image in the reference image, and the audio features of the audio are obtained. The descriptive text indicates the action description information of the virtual image based on audio.

[0036] For example, reference images, descriptive text, and one or more audio files can be acquired from user-uploaded files, or reference images, descriptive text, and audio files can be acquired from an acquisition device. The reference images here include one or more virtual avatars.

[0037] Furthermore, assuming the reference image shows multiple virtual figures on the stage, the descriptive text could be "multiple virtual figures are singing and dancing on the stage," or "the first virtual figure is singing on the stage, the second virtual figure is dancing on the stage," etc. The virtual figures can be people, animals, or other virtual characters.

[0038] Furthermore, a pre-trained encoder can be used to convert the descriptive text into semantic vectors, and then an image patch embedding layer can be used to map the semantic vectors to a unified dimension to obtain text features. The reference image can be preprocessed with dimensionality reduction and noise reduction, and then the encoder can be used to downsample the preprocessed reference image to generate low-dimensional video features. Multiple image features of multiple virtual characters in the reference image can be extracted from the low-dimensional video features. Each audio segment is divided into temporal frames, and these temporal frames are input into a speech vector conversion model. The speech vector conversion model is used to obtain the semantic and frequency features of each audio segment. Specifically, the speech vector conversion model converts the audio into vector representations, performs dimensionality compression on these audio features, and assigns weights to each audio feature to distinguish the audio source, ultimately obtaining audio features with unified dimensions.

[0039] In operation S220, the character features are bound to the audio features to obtain the target audio features. The character features are associated with the corresponding virtual image in the reference image. The character features are used to indicate the association between the target audio features and the image features.

[0040] For example, each virtual character in the reference image has a corresponding character feature. This character feature can be pre-determined based on statistics, expert experience, training, etc., or it can be contour features, color features, texture features, and corner features extracted from the corresponding virtual character. The audio features corresponding to the virtual character also include the relevant character feature. Therefore, the correlation between the virtual character and its associated audio features can be increased through the character feature. Binding can be achieved by adding features, concatenating them, or by inputting the character feature and audio feature into a machine learning model and outputting the target audio feature.

[0041] For example, there is a first degree of correlation between audio features and image features. Through the binding of character features, there is a second degree of correlation between the target audio features and the image features. Because the character features are associated with the virtual image and are also bound to the audio features, the second degree of correlation is greater than the first degree of correlation. The first and second degrees of correlation can be obtained by calculating semantic similarity.

[0042] In operation S230, a target video is generated based on text features, image features, and target audio features. The target video includes multiple video frames based on audio-driven virtual images emitting sounds according to action description information.

[0043] In some embodiments, the descriptive text indicates action description information of multiple virtual figures in a reference image driven by multiple audio features based on multiple audio, and further includes: binding multiple character features to multiple audio features respectively to obtain multiple target audio features; generating a target video based on text features, multiple figure features and multiple target audio features, the target video including multiple video frames that drive multiple virtual figures to dialogue according to action description information based on multiple audio.

[0044] In one example, the first audio is the audio of the first virtual character, the second audio is the audio of the second virtual character, and the reference image shows the first and second virtual characters singing a duet on stage. Then the generated target video shows the first and second virtual characters singing a duet on stage, with descriptions based on descriptive text and actions that match the characteristics of the audio.

[0045] Through the embodiments of this disclosure, by binding the character features associated with the corresponding virtual image in the reference image to the audio features, the target audio features and image features are associated through the character features. Compared with the audio features, the target audio features and image features have a closer relationship. Thus, the connection between audio and virtual objects can be effectively focused on during the video generation process. It can adapt to one or more audio inputs, so that each virtual image is driven by the corresponding audio. In the generated target video, dynamic positioning of characters can be achieved, capturing individual actions or multi-person interactive actions, improving the matching degree of virtual images, actions and audio, and adapting to complex needs such as character movement, multi-person dialogue, and preservation of real-time background dynamic elements.

[0046] Figure 3 This is a schematic diagram of a video generation method according to another embodiment of the present disclosure.

[0047] In some embodiments, generating a target video based on text features, image features, and target audio features includes: determining multiple query vectors for a multi-head attention mechanism based on the positional encoding vectors of the character features and image features; performing cross-attention processing based on the query vectors, text features, and target audio features to obtain fused features obtained through the correlation between the target audio features and the query vectors, where the correlation is related to the correlation between the target audio features and the image features; and performing decoding operations based on the fused features to generate the target video.

[0048] like Figure 3As shown, in method 300, a query vector 303 is obtained based on the position encoding vector 302 of the character feature 301 and the image feature of the virtual image bound to it. One of the text feature 304 and the target audio feature 305 bound to the character feature is used as a key vector and the other as a value vector. Next, operation S310 is performed. In operation S310, cross-attention processing is performed based on the query vector, key vector and value vector to obtain the fusion feature 306. The fusion feature is decoded to obtain the target video 307.

[0049] Through the embodiments of this disclosure, in the cross-attention processing, the association between the target audio and the virtual image is realized based on the character characteristics, without relying on the fixed spatial area of ​​the character. Even if the character moves left or right or is obscured in front or behind in the video, the correct association between "audio and virtual image" can still be maintained.

[0050] In some embodiments, the position encoding vector of the image feature is obtained by: obtaining the target angle based on the position information in the reference image associated with the vector representation of the image feature; and rotating the vector representation of the image feature by the target angle in two-dimensional space to obtain the position encoding vector.

[0051] For example, positional information can be directly integrated into the query vector and key vector in the attention mechanism for rotational positional encoding. Specifically, rotational positional encoding of image features integrates their positional information into the query vector; if text features are used as key vectors, rotational positional encoding integrates their positional information into the key vector; and if target audio features are used as key vectors, rotational positional encoding integrates their positional information into the key vector. After integrating the positional information, weights are assigned to the query vector, key vector, and value vector, and cross-attention processing is performed to obtain the fused features.

[0052] Furthermore, by introducing position encoding vectors and character features, even if the virtual character undergoes dynamic changes such as movement in the video, it can still be correctly associated with the virtual character and the audio. For example, the activity area of ​​the virtual character corresponding to the first image feature is designated as the first region, and the activity area of ​​the virtual character corresponding to the second image feature is designated as the second region. Within the first region, the first image feature is positionally encoded to obtain a first positional encoding vector. Based on the first positional encoding vector and the first role feature associated with the virtual character corresponding to the first image feature, a first query vector is obtained. The first query vector, text features, and target audio features associated with the first role feature are subjected to cross-attention processing to obtain a first fusion feature. Within the second region, the second image feature is positionally encoded to obtain a second positional encoding vector. Based on the second positional encoding vector and the second role feature associated with the virtual character corresponding to the second image feature, a second query vector is obtained. The second query vector, text features, and target audio features associated with the second role feature are subjected to cross-attention processing to obtain a second fusion feature. The first and second fusion features are integrated into a fusion feature through a multi-head attention mechanism, and the fusion feature is decoded to obtain the target video. Even if the virtual character corresponding to the first image feature in the target video undergoes dynamic changes such as running, jumping, or moving left or right, the audio features bound to the first role feature can still accurately drive the virtual character corresponding to the first image feature, and the audio features bound to the second role feature can also accurately drive the virtual character corresponding to the second image feature.

[0053] In some embodiments, determining the query vector for the multi-head attention mechanism based on the position encoding vectors of role features and image features includes: weighting the position encoding vectors using multiple query weights of the multi-head attention mechanism to obtain candidate query vectors; and obtaining multiple query vectors by applying role features to the candidate query vectors.

[0054] Through the embodiments of this disclosure, a query vector is obtained by applying character features to the candidate query vector (e.g., by adding them). Furthermore, since character features are bound to audio features, the correlation between virtual characters and audio can be improved.

[0055] Figure 4 This is a schematic diagram of a video generation method according to another embodiment of the present disclosure.

[0056] In some embodiments, generating a target video by processing text features, multiple image features, and multiple target audio features includes: constraining the motion trend of a virtual image in the target video based on at least one of speech rate features, tone features, temporal features, and dialogue content features indicated by the multiple target audio features, and features of action description information indicated by the text features, wherein the motion trend indicates the changing trend of the virtual image's inter-frame motion vector; and generating the target video by processing text features, multiple image features, and multiple target audio features based on the motion trend, wherein the multiple virtual images conform to the motion trend during the execution of the action description information.

[0057] like Figure 4 As shown, in method 400, the motion trend 403 of each virtual character in the target video is constrained based on multiple target audio features 401 and text features 402 to ensure that the inter-frame motion vectors of each virtual character change smoothly. Finally, the text features, multiple character features, and multiple target audio features are processed based on the motion trend of each virtual character to obtain the target video 404. Specifically, to ensure the consistency between each virtual character and the audio in the target video and the stability of the appearance of each virtual character, multiple video segments can be generated first. For example, if the target video is a 30-second video, this 30-second video can be divided into 10 video segments, each of which is a 3-second video. The inter-frame motion vectors within each video segment are calculated. This motion vector can quantify the positional difference of the same virtual character in the current frame image and the previous frame image. Based on the text features of the descriptive text and the temporal features of the target audio features, the change trend of the motion vector is smoothed to remove outliers. Based on the smoothed motion trend, the fused features are decoded to obtain the target video.

[0058] For example, the text features describing the text indicate that a virtual character moved left or right at a certain moment. If the position of the virtual character changes abnormally between the current frame and the previous frame in a video clip containing that moment, the video clip is smoothed to remove the abnormal points.

[0059] For example, the trend of audio changes can also be constrained. If the tone of a virtual character obtained from the target audio features is relatively calm, but the tone of the virtual character in the video clip changes greatly between the current frame and the previous frame, and the tone in the current frame is very rapid, then the trend of audio changes of the virtual character in the video clip can be constrained to obtain a more accurate motion vector.

[0060] Long videos generally suffer from poor image consistency, with issues like "character distortion" and "background flickering" easily appearing in video frames. For example, when the input voice duration exceeds 5 seconds, inter-frame feature drift becomes significant, making it impossible to maintain the integrity of the first frame. This embodiment of the present disclosure constrains the movement trend of the virtual character within the target video, ensuring the continuity of long videos and improving video generation quality.

[0061] In some embodiments, generating a target video based on text features, image features, and target audio features includes: generating multiple video segments sequentially based on text features, image features, and target audio features; and fusing the first action feature of a later video segment with the second action feature of a previous video segment according to the temporal features indicated by the target audio features to obtain the target video.

[0062] For example, the first motion feature represents the motion information of the virtual object in at least one video frame of a later video segment, and the second motion feature represents the motion information of the virtual object in at least one video frame of a previous video segment.

[0063] For example, fusion could involve inputting temporal features, action description information (first action features and second action features) into a machine learning model to obtain the head frame of a later video segment and the tail frame of a previous video segment. Alternatively, fusion could involve extracting features from the tail frame of a previous video segment and using them as at least part of the input features for generating the head frame of a later video segment to produce a head frame with consistent action.

[0064] Through the embodiments of this disclosure, multiple video segments are spliced ​​by fusing action features based on the temporal features of the target audio features. This can eliminate the discontinuity of action of each virtual character in the target video, thereby ensuring the continuity of the video and improving the quality of video generation.

[0065] In some embodiments, the video generation method further includes: extracting a first motion feature based on the motion information of a virtual figure in the tail video frame of a preceding video segment; and extracting a second motion feature based on the motion information of a virtual figure in the head video frame of a subsequent video segment. The head may be a preset time period near the start time of the video segment, and the tail may be a preset time period near the end time of the video segment.

[0066] The embodiments disclosed herein facilitate the extraction of continuous motion between video segments.

[0067] In some embodiments, processing text features, multiple image features, and multiple target audio features to generate a target video includes: concatenating multiple noise features with multiple image features in the frame dimension to obtain multiple video latent features; and generating a target video based on the text features, multiple video latent features, and multiple target audio features using a diffusion model in the video generation model.

[0068] For example, the size of the noise feature is [1,C,F,H,W], and the size of the image feature is [1,C,1,H,W], where C, H, and W represent the channel dimension, height dimension, and width dimension of the feature, respectively, and F represents the frame dimension of the feature. By concatenating multiple noise features and multiple image features in the frame dimension, multiple video latent features (i.e., features used to generate video clips) can be obtained. Then, a diffusion model is used to denoise the video generated based on text features, multiple video latent features, and multiple target audio features to obtain the target video.

[0069] By combining noise features with image features at the frame level through the embodiments of this disclosure, it is beneficial to reduce the accumulation of errors during the generation of multiple segments of the target video.

[0070] Figure 5 This is a flowchart of a video generation model training method according to an embodiment of the present disclosure.

[0071] like Figure 5 As shown, the training method 500 for the video generation model includes operations S510 to S540.

[0072] In operation S510, the text sample features of the descriptive text sample, the image sample features of the virtual image in the reference image sample, and the audio sample features of the audio sample are obtained. The descriptive text sample indicates the action description information of the virtual image driven by the audio sample.

[0073] In operation S520, the character sample features are bound to the audio sample features to obtain the target audio sample features. The character sample features are associated with the corresponding virtual image in the reference image sample. The character sample features are used to indicate the association between the target audio sample features and the image sample features.

[0074] When operating the S530, the video generation model to be trained is used to generate target video samples based on text sample features, multiple image sample features, and multiple target audio sample features. The target video samples include multiple video frames based on audio-driven virtual images emitting sounds according to action description information.

[0075] In operation S540, the parameters of the video generation model are adjusted based on the loss value between the target video sample and the preset reference video to obtain the trained video generation model.

[0076] Through the embodiments of this disclosure, the video generation model trained by the above training method can associate virtual images in reference images with their corresponding audio, and accurately generate videos in which one or more virtual images emit sounds according to action description information, thereby improving video generation capabilities.

[0077] In some embodiments, the descriptive text sample indicates action description information of multiple virtual characters in a reference image sample driven by multiple audio sample features based on multiple audio samples, and further includes: binding multiple character sample features to multiple audio sample features respectively to obtain multiple target audio sample features; generating a target video sample based on the text sample features, multiple image features and multiple target audio sample features, the target video sample including multiple video frames that drive multiple virtual characters to dialogue according to action description information based on multiple audio samples respectively.

[0078] Through the embodiments of this disclosure, it is possible to accurately generate videos that allow multiple virtual avatars to engage in dialogue according to action description information by training on dialogues with multiple people, thereby improving video generation capabilities.

[0079] Figure 6 This is a schematic diagram illustrating the acquisition of a reference video according to an embodiment of the present disclosure.

[0080] In some embodiments, the reference videos are obtained by: determining multiple video frame samples from video samples, the video frame samples including multiple virtual avatar samples; obtaining multiple consistency evaluation values ​​of the multiple virtual avatar samples in the video frame samples, the consistency evaluation values ​​indicating the degree of matching between the virtual avatar samples and the audio samples; labeling multiple behavioral tags of the multiple virtual avatar samples in the video frame samples based on the multiple consistency evaluation values; and obtaining multiple video segment samples from the video samples based on the multiple behavioral tags of the multiple virtual avatar samples in the multiple video frame samples, to serve as multiple reference videos.

[0081] like Figure 6As shown, in method 600, the video sample 601 is standardized and converted into a fixed frame rate video stream. Multiple video frame samples 602 are determined from the video sample, including multiple virtual character samples such as the first virtual character sample 603 and the second virtual character sample 604. Audio signals are extracted synchronously and converted into a fixed frequency mono audio stream to ensure audio and video time axis alignment. The consistency evaluation value of each virtual character sample in each video frame sample is calculated using a pre-trained speech-visual association model to quantify the degree of matching of virtual character samples in dialogue based on audio samples in video frame samples. The consistency evaluation value curve is smoothed by an adaptive sliding window to eliminate instantaneous detection noise and improve the stability of the consistency evaluation value. Video frame samples with a consistency evaluation value greater than or equal to a preset threshold are labeled as speaking tags, and video frame samples with a consistency evaluation value less than the preset threshold are labeled as listening tags. That is, in operation S610, the behavior tags of virtual character samples in video frame samples are obtained. Based on these tags, multiple video segments are segmented from the video sample as reference videos 605. Furthermore, for video frame samples labeled with speaking or listening tags, if the speaking or listening time of the virtual avatar sample is less than the preset duration, the listening and speaking tags can be merged into one tag, thereby ensuring the consistency of the labeled data during model training.

[0082] Furthermore, multiple video frame samples are determined from the video samples, including: using a tracking algorithm to track the spatial position of each virtual avatar sample in each video sample; based on the tracking results, calculating the time interval where multiple virtual avatar samples coexist using an overlap interval algorithm, such as the time interval in a multi-person dialogue scenario; performing a quality assessment on the obtained time intervals; and using the video sample segments corresponding to the time intervals that meet the quality assessment requirements as video frame samples; further, the quality assessment requirements may be that the virtual avatar sample quality meets preset requirements (e.g., the face area is greater than a preset number of pixels, the ratio of the largest face area to the smallest face area is less than or equal to a preset ratio), the number of frames in the video sample segment is greater than a preset number of frames (to ensure the integrity of the dialogue segment), and the number of virtual avatar sample tracking interruptions is less than a preset number of interruptions, etc.

[0083] Through the embodiments of this disclosure, speaking labels and listening labels are determined based on consistency evaluation values. The segmented fragment samples can achieve accurate association between multi-person tracking boxes and speech consistency, improve the accuracy of "person-speech" matching, and thus improve the training effect.

[0084] In some embodiments, obtaining multiple video segment samples from multiple video samples based on multiple behavior tags of multiple virtual avatar samples in multiple video frame samples includes: in response to any virtual avatar sample having a speaking tag in the video frame sample, where the speaking tag indicates that the consistency evaluation value is greater than or equal to a preset threshold, retaining the audio track of any virtual avatar sample at the corresponding moment of the video frame sample; and in response to any virtual avatar sample having a listening tag in the video frame sample, muting the audio track of any virtual avatar sample at the corresponding moment of the video frame sample, obtaining a video segment sample of any virtual avatar sample, where the listening tag indicates that the consistency evaluation value is less than a preset threshold.

[0085] For example, based on the speaking tags of multiple virtual avatar samples in a video frame sample, the audio track of the virtual avatar sample marked with the speaking tag in the video frame sample is retained at the time of speaking. Based on the listening tags of multiple virtual avatar samples in a video frame sample, the virtual avatar sample marked with the listening tag in the video frame sample is muted at the time of listening. In this way, multi-track data can be obtained, and each track corresponds to one virtual avatar sample, ensuring accurate alignment of the audio timeline.

[0086] Through the embodiments of this disclosure, by ensuring that each sub-segment corresponds to a single speaking action, and setting other audio tracks in the segment to silent audio, dual-track (or multi-track) data corresponding to the video is obtained, with each audio track corresponding to a speaker, ensuring accurate alignment of the audio and video timelines.

[0087] In some embodiments, determining multiple video frame samples from video samples includes: obtaining multiple detection boxes of multiple virtual avatar samples from multiple video frames of the video samples; performing overlap detection on at least one of the display time and display area of ​​the multiple detection boxes for each video frame in the multiple video frames; and determining multiple video frame samples in response to the result of the overlap detection meeting a preset condition.

[0088] Specifically, a tracking algorithm is used to acquire multiple detection boxes for multiple virtual avatar samples. The spatial position of each virtual avatar sample in each video sample is tracked through the detection boxes. Based on the tracking results, the time interval where multiple virtual avatar samples coexist is calculated using an overlap interval algorithm, such as the time interval in a multi-person dialogue scenario. The quality of the obtained time interval is evaluated, and the video sample segments corresponding to the time intervals that meet the quality evaluation requirements are used as video frame samples. Further, the quality evaluation requirements may be that the virtual avatar sample quality meets preset requirements (e.g., the face area is greater than a preset number of pixels, the ratio of the largest face area to the smallest face area is less than or equal to a preset ratio), the number of frames in the video sample segment is greater than a preset number of frames (to ensure the integrity of the dialogue segment), the number of virtual avatar sample tracking interruptions is less than a preset number of interruptions, etc.; and / or, based on the tracking results, overlap detection is performed on the display area in each video frame, and the video frame corresponding to the display area with multiple virtual avatar samples is used as a video frame sample.

[0089] Through the embodiments of this disclosure, the accuracy of "virtual avatar-voice" matching can be improved and mismatch problems can be avoided by associating the dual features of "tracking box and consistency evaluation value".

[0090] In some embodiments, the method further includes: adjusting character sample features to reduce the loss value between the target video sample and the reference video, wherein adjusting the character sample features is used to indicate the updated association between the target audio sample features and the image sample features.

[0091] Through the embodiments of this disclosure, the characteristics of a virtual avatar can be learned during the training process, so as to effectively increase the correlation between audio and the virtual avatar.

[0092] In some embodiments, metadata can be added to the reference video, such as cropping effective segments to generate independent video files, synchronously extracting frame images, and generating structured metadata, including basic information (such as segment ID, source video, duration, frame rate), behavioral information (such as speaker ID, "speaking-listening" time interval (accurate to the frame)), and quality information (such as face clarity score, mean speech consistency). The output consists of a structured video segment set (containing independent videos and frame images) and a metadata dataset (containing speech behavior annotations, basic quality information, etc.). The "speaking-listening" time interval is determined based on the behavioral labels.

[0093] Figure 7 This is a schematic diagram of the overall framework of a video generation method according to an embodiment of the present disclosure.

[0094] like Figure 7As shown, in method 700, features are extracted from multiple audio files 701, descriptive text 702, and reference images 703 to obtain audio features 704, text features 705, and image features 706. These features are then input into a video generation model, and a target video 708 is generated through the video generation model 707. Further, the method for generating the target video using the video generation model is as follows: noise features 709 are concatenated with the image features 706 of each virtual image in the reference image to obtain video features 710, where the noise features are random noise features; position encoding is performed on the video features to obtain a position encoding vector 712; and based on the character features 711 and the position encoding vector 712, a... The query vector 713 is used, where character features are associated with image features and audio features. One image feature corresponds to one character feature, and one audio feature also corresponds to one character feature. For interrelated image and audio features, their character features are the same. Multi-head attention is divided into self-attention and cross-attention. Self-attention involves using the video features themselves as QKV, performing regular matrix calculations, and then combining them with image features. Cross-attention calculation is then performed based on the query vector, text features, and target audio features, i.e., operation S710 is executed to obtain the fused features. Here, target audio feature 715 refers to audio feature 714 bound to character features. Finally, decoding the fused features 716 generates the target video 708. During cross-attention calculation, the video features are Q, and the other conditional features are K and V. There are two conditions here, namely audio features and text features, so cross-attention calculation is performed twice. That is, self-attention calculation, cross-attention calculation, cross-attention calculation, and feedforward neural network processing are repeated N times in the model.

[0095] Figure 8 This is a block diagram of a video generation apparatus according to an embodiment of the present disclosure.

[0096] like Figure 8 As shown, the video generation device 800 may include an initial feature acquisition module 810, a target audio feature acquisition module 820, and a target video generation module 830.

[0097] The initial feature acquisition module 810 is used to acquire the text features of the descriptive text, the image features of the virtual image in the reference image, and the audio features of the audio. The descriptive text indicates the action description information of the virtual image based on audio.

[0098] The target audio feature acquisition module 820 is used to bind character features with audio features to obtain target audio features. The character features are associated with the corresponding virtual image in the reference image. The character features are used to indicate the relationship between the target audio features and the image features.

[0099] The target video generation module 830 is used to generate a target video based on text features, image features and target audio features. The target video includes multiple video frames based on audio-driven virtual images emitting sounds according to action description information.

[0100] For example, the target video generation module 830 is further configured to determine a query vector for the multi-head attention mechanism based on the position encoding vector of the character features and the image features; perform cross-attention processing based on the query vector, text features and target audio features to obtain fused features, which are obtained through the third correlation between the target audio features and the query vector, and the third correlation is positively correlated with the first correlation; and perform decoding operation based on the fused features to generate the target video.

[0101] For example, the position encoding vector of the image feature is obtained by: obtaining the target angle based on the position information in the reference image associated with the vector representation of the image feature; and rotating the vector representation of the image feature by the target angle in two-dimensional space to obtain the position encoding vector.

[0102] For example, the target video generation module 830 is also configured to weight the position encoding vector using the query weights of the multi-head attention mechanism to obtain a candidate query vector; and to obtain a query vector by applying role features to the candidate query vector.

[0103] For example, the target video generation module 830 is further configured to constrain the motion trend of the virtual image in the target video based on at least one of the speech rate features, tone features, temporal features and dialogue content features indicated by the target audio features, and the features of the action description information indicated by the text features; and to generate the target video by processing the text features, image features and target audio features based on the motion trend, wherein the virtual image conforms to the motion trend in the process of performing the action description information.

[0104] For example, the target video generation module 830 is further configured to generate multiple video segments sequentially based on text features, image features, and target audio features; and to fuse the first action feature of a later video segment with the second action feature of a previous video segment according to the temporal features indicated by the target audio features to obtain the target video.

[0105] For example, the video generation apparatus 800 is further configured to extract a first motion feature based on the motion information of the virtual image in the tail video frame of the preceding video segment; and to extract a second motion feature based on the motion information of the virtual image in the head video frame of the following video segment.

[0106] For example, the descriptive text indicates action description information of multiple virtual figures in a reference image driven by multiple audio features based on multiple audios. The video generation device 800 is also configured to: bind multiple character features to multiple audio features respectively to obtain multiple target audio features; generate a target video based on text features, multiple figure features and multiple target audio features, the target video including multiple video frames that drive multiple virtual figures to dialogue according to action description information based on multiple audios respectively.

[0107] Figure 9 This is a block diagram of a video generation model training apparatus according to an embodiment of the present disclosure.

[0108] like Figure 9 As shown, the video generation model training device 900 may include an initial sample feature acquisition module 910, a target audio sample feature acquisition module 920, a target video sample generation module 930, and a video generation model acquisition module 940.

[0109] The initial sample feature acquisition module 910 is used to acquire the text sample features of the descriptive text sample, the image sample features of the virtual image in the reference image sample, and the audio sample features of the audio sample. The descriptive text sample indicates the action description information of the virtual image driven by the audio sample.

[0110] The target audio sample feature acquisition module 920 is used to bind the character sample features with the audio sample features to obtain the target audio sample features. The character sample features are associated with the corresponding virtual image in the reference image sample. The character sample features are used to indicate the relationship between the target audio sample features and the image sample features.

[0111] The target video sample generation module 930 is used to generate target video samples based on text sample features, image sample features and target audio sample features using the video generation model to be trained. The target video samples include multiple video frames based on the audio-driven virtual image emitting sounds according to the action description information.

[0112] The video generation model acquisition module 940 is used to adjust the parameters of the video generation model based on the loss value between the target video sample and the preset reference video to obtain the trained video generation model.

[0113] For example, the video generation model acquisition module 940 is further configured to determine multiple video frame samples from the video samples, the video frame samples including multiple virtual avatar samples; acquire multiple consistency evaluation values ​​of the multiple virtual avatar samples in the video frame samples, the consistency evaluation values ​​indicating the degree of matching between the virtual avatar samples and the audio samples; label multiple behavioral tags of the multiple virtual avatar samples in the video frame samples based on the multiple consistency evaluation values; and obtain multiple video segment samples from the video samples based on the multiple behavioral tags of the multiple virtual avatar samples in the multiple video frame samples, as multiple reference videos.

[0114] For example, the video generation model acquisition module 940 is further configured to, in response to any virtual character sample's behavior label in the video frame sample being a speaking label, retain the audio track of any virtual character sample at the corresponding moment of the video frame sample, wherein the speaking label indicates that the consistency evaluation value is greater than or equal to a preset threshold; and in response to any virtual character sample's behavior label in the video frame sample being a listening label, mute the audio track of any virtual character sample at the corresponding moment of the video frame sample to obtain a video segment sample of any virtual character sample, wherein the listening label indicates that the consistency evaluation value is less than a preset threshold.

[0115] For example, the video generation model acquisition module 940 is further configured to acquire multiple detection boxes of multiple virtual image samples from multiple video frames of the video samples; for each of the multiple video frames, perform overlap detection on at least one of the display time and display area of ​​the multiple detection boxes; and determine multiple video frame samples in response to the result of the overlap detection meeting a preset condition.

[0116] For example, the video generation model training device 900 is also configured to adjust the character sample features to reduce the degree of loss between the target video sample and the reference video, wherein the adjusted character sample features, compared with the character sample features, enable the target audio sample features to have a greater first correlation with the image sample features.

[0117] For example, the descriptive text sample indicates the action description information of multiple virtual characters in the reference image sample driven by multiple audio sample features based on multiple audio samples. The video generation model training device 900 is also configured to bind multiple character sample features to multiple audio sample features respectively to obtain multiple target audio sample features; and generate target video samples based on text features, multiple image features and multiple target audio sample features. The target video samples include multiple video frames that drive multiple virtual characters to dialogue according to action description information based on multiple audio.

[0118] Figure 10 This is a structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure.

[0119] In the embodiments of this application, inspired by the von Neumann architecture in modern computer theory, such as... Figure 10 As shown, the intelligent agent 1000 may include multiple core modules: an input module 1010, a control module 1020, and an output module 1030.

[0120] In the example, input module 1010 is responsible for receiving or sensing information such as calls, queries, requests, instructions, signals, question information, or data from the outside world (e.g., users or the external environment), and converting it into a format that the intelligent agent 1000 can understand and process. Input module 1010 is the primary link for the intelligent agent 1000 to interact with the outside world, enabling the intelligent agent 1000 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information. In the example, input module 1010 can be used to receive input information. Input information may include descriptive text, reference images, and multiple audio files as described above.

[0121] In the example, the control module 1020 is the core support for the agent 1000's ability to handle complex tasks. The control module 1020 generates a target video based on the input information received by the input module 1010. A large model is determined based on the target video. By calling the large model, the input information can be processed to obtain the output information.

[0122] In the example, output module 1030 can be used to output the output information obtained by output control module 1020. The output information may include the target video as described above.

[0123] In the example, the control module 1020 may include a control unit 1021, a storage unit 1022, and a processing unit 1023.

[0124] During operation, the control unit 1021 will continuously interact with the storage unit 1022, the arithmetic unit 1023, and / or the output module 1030. However, in the embodiments of this application, the control unit 1021 initiates communication with the storage unit 1030, the arithmetic unit 1023, and / or the output module 1030 as a single initiator, while there may be no communication coupling between the storage unit 1022, the arithmetic unit 1023, and the output module 1030.

[0125] In the example, the performance of control unit 1021 is closely related to the large model on which agent 1000 is based. To fully leverage the capabilities of the large language model, the internal structure of control unit 1021 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0126] Storage unit 1022 can be responsible for remembering information such as historical dialogues and event streams. As described above, descriptive text, reference images, multiple audio files, and target videos can all be included in storage unit 1022.

[0127] In the example, after acquiring input information, agent 1000 can use the input information to retrieve descriptive text, reference images, multiple audio files, and target video from storage unit 1022, and then feed them back to control unit 1021. Control unit 1021 can then call upon a large model to process the input information and generate the target video.

[0128] The arithmetic unit 1023 can be viewed as a predefined tool library. Tools such as format conversion tools can be included in the arithmetic unit 1023.

[0129] In the example, when the AI ​​agent 1000 needs to render the output data, it can call the relevant renderer and display tools from the computing unit 1023 and feed them back to the control module 1020. Then, the control module 1020 can use the returned renderer and display tools to pass the rendered result to the output module 1030. It is understandable that although large language models have excellent language understanding and generation capabilities, like humans, the tasks they can solve without any tools are very limited. When the AI ​​agent 1000 is given the ability to call tools, it can perform tasks such as result display.

[0130] The intelligent agent 1000 according to the embodiments of this application can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0131] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0132] Figure 11 This is a block diagram of an electronic device to which a video generation method can be applied according to one embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0133] like Figure 11As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0134] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as text processing methods and / or deep learning framework deployment methods. For example, in some embodiments, the text processing methods and / or deep learning framework deployment methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the text processing method and / or the deployment method of the deep learning framework described above can be performed. Alternatively, in other embodiments, computing unit 1101 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0136] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM) or flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) monitor or a liquid crystal display (LCD)); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: The text features of the descriptive text, the image features of the virtual avatar in the reference image, and the audio features of the audio are obtained, wherein the descriptive text indicates action description information of the virtual avatar driven by the audio. By binding the character features with the audio features, a target audio feature is obtained. The character features are associated with the corresponding virtual image in the reference image. The character features are used to indicate the association between the target audio feature and the image feature. A target video is generated based on the text features, the image features, and the target audio features. The target video includes multiple video frames in which the virtual image emits sounds according to the action description information, driven by the audio.

2. The method according to claim 1, wherein, The generation of the target video based on the text features, the image features, and the target audio features includes: Based on the position encoding vectors of the character features and the image features, a query vector is determined for use in the multi-head attention mechanism; Cross-attention processing is performed on the query vector, the text features, and the target audio features to obtain fused features. The fused features are obtained by the correlation between the target audio features and the query vector, and the correlation is related to the correlation relationship. Decoding is performed based on fused features to generate the target video.

3. The method according to claim 2, wherein, The location encoding vector of the image feature is obtained using the following method: The target angle is obtained by using the vector representation of the image features to represent the positional information in the associated reference image. The position encoding vector is obtained by rotating the vector representation of the image feature in two-dimensional space by the target angle.

4. The method according to claim 2 or 3, wherein, The determination of the query vector for the multi-head attention mechanism based on the position encoding vectors of the character features and the image features includes: The position encoding vector is weighted using the query weights of the multi-head attention mechanism to obtain a candidate query vector; The query vector is obtained by applying the role characteristics to the candidate query vector.

5. The method according to claim 1 or 2, wherein, The generation of the target video based on the text features, the image features, and the target audio features includes: Based on at least one of the speech rate features, tone features, temporal features, and dialogue content features indicated by the target audio features, and the features of the action description information indicated by the text features, the motion trend of the virtual image in the target video is constrained. The target video is generated by processing the text features, image features, and target audio features based on the motion trend, wherein the virtual image conforms to the motion trend while performing the action description information.

6. The method according to claim 1 or 2, wherein, The generation of the target video based on the text features, the image features, and the target audio features includes: Based on the text features, the image features, and the target audio features, multiple video segments are generated sequentially. According to the temporal features indicated by the target audio features, the first action feature of the later video segment and the second action feature of the earlier video segment are fused to obtain the target video.

7. The method according to claim 6, wherein, The method further includes: Based on the motion information of the virtual figure in the last video frame of the preceding video segment, the first motion feature is extracted; Based on the motion information of the virtual image in the head video frame of the subsequent video segment, the second motion feature is extracted.

8. The method according to claim 1, wherein, The descriptive text indicates action description information of multiple virtual figures in the reference image driven by multiple audio features based on multiple audio sources, and the method further includes: The multiple character features are respectively bound to the multiple audio features to obtain multiple target audio features; The target video is generated based on the text features, multiple image features, and multiple target audio features. The target video includes multiple video frames in which multiple virtual images are driven to perform dialogues according to the action description information based on the multiple audio features.

9. A method for training a video generation model, comprising: The text sample features of the descriptive text sample, the image sample features of the virtual character in the reference image sample, and the audio sample features of the audio sample are obtained. The descriptive text sample indicates the action description information of the virtual character driven by the audio sample. By binding the character sample features with the audio sample features, a target audio sample feature is obtained. The character sample feature is associated with the corresponding virtual image in the reference image sample. The character sample feature is used to indicate the association between the target audio sample feature and the image sample feature. The target video sample is generated using the video generation model to be trained based on the text sample features, the image sample features, and the target audio sample features. The target video sample includes multiple video frames that drive the virtual image to emit sounds according to the action description information based on the audio. Based on the loss value between the target video sample and the preset reference video, the parameters of the video generation model are adjusted to obtain a trained video generation model.

10. The method according to claim 9, wherein, The reference video was obtained using the following method: Multiple video frame samples are determined from video samples, wherein the video frame samples include multiple virtual avatar samples; Obtain the consistency evaluation value of the multiple virtual character samples in the video frame samples, and the consistency evaluation value indicates the degree of matching between the virtual character samples and the audio samples; Based on the consistency evaluation value, multiple behavioral labels of the multiple virtual character samples are labeled on the video frame samples respectively; Based on multiple behavioral tags of the multiple virtual character samples in the multiple video frame samples, multiple video segment samples are obtained from the video samples to serve as multiple reference videos.

11. The method according to claim 10, wherein, The process of obtaining multiple video segment samples from the video samples based on multiple behavior tags of the multiple virtual avatar samples in the multiple video frame samples includes: In response to any virtual character sample among the plurality of virtual character samples having a speech tag in the video frame sample, the audio track of any virtual character sample at the corresponding moment in the video frame sample is retained, wherein the speech tag indicates that the consistency evaluation value is greater than or equal to a preset threshold; and In response to the behavior label of the virtual character sample in the video frame sample being a listening label, the audio track of any virtual character sample at the corresponding time of the video frame sample is muted to obtain a video segment sample of any virtual character sample. The listening label indicates that the consistency evaluation value is less than the preset threshold.

12. The method according to claim 10, wherein, The step of determining multiple video frame samples from video samples includes: Multiple detection boxes of the multiple virtual avatar samples are obtained from multiple video frames of the video samples; For each of the plurality of video frames, overlap detection is performed on at least one of the display time and display area of ​​the plurality of detection boxes; In response to the overlap detection result meeting the preset conditions, the multiple video frame samples are determined.

13. The method according to claim 9, wherein, The method further includes: The character sample features are adjusted to reduce the loss value between the target video sample and the reference video, wherein the adjusted character sample features are used to indicate the updated association between the target audio sample features and the image sample features.

14. The method according to claim 9, wherein, The descriptive text sample indicates action description information of multiple virtual figures in the reference image sample driven by multiple audio sample features based on multiple audio samples, and the method further includes: The multiple character sample features are respectively bound to the multiple audio sample features to obtain the multiple target audio sample features; The target video sample is generated based on the text sample features, multiple image sample features, and multiple target audio sample features. The target video sample includes multiple video frames that drive multiple virtual images to perform dialogues according to the action description information based on multiple audios.

15. A video generation apparatus, comprising: The initial feature acquisition module is used to acquire the text features of the descriptive text, the image features of the virtual image in the reference image, and the audio features of the audio, wherein the descriptive text indicates the action description information of the virtual image driven by the audio. The target audio feature acquisition module is used to bind the character features with the audio features to obtain the target audio features. The character features are associated with the corresponding virtual image in the reference image. The character features are used to indicate the association between the target audio features and the image features. The target video generation module is used to generate a target video based on the text features, the image features, and the target audio features. The target video includes multiple video frames in which the virtual image emits sound according to the action description information based on the audio.

16. A training device for a video generation model, comprising: The initial sample feature acquisition module is used to acquire the text sample features of the descriptive text sample, the image sample features of the virtual image in the reference image sample, and the audio sample features of the audio sample. The descriptive text sample indicates the action description information of the virtual image driven by the audio sample. The target audio sample feature acquisition module is used to bind the character sample features with the audio sample features to obtain the target audio sample features. The character sample features are associated with the corresponding virtual image in the reference image sample. The character sample features are used to indicate the association between the target audio sample features and the image sample features. The target video sample generation module is used to generate target video samples based on the text sample features, the image sample features and the target audio sample features using a video generation model to be trained. The target video sample includes multiple video frames based on the audio driving the virtual image to emit sound according to the action description information. The video generation model acquisition module is used to adjust the parameters of the video generation model based on the loss value between the target video sample and a preset reference video to obtain a trained video generation model.

17. An intelligent agent, comprising: An input module is used to receive input information, including descriptive text, reference images, and audio. The processing module is configured to obtain the target video by calling a large model to execute the method of any one of claims 1 to 8 based on the input information received by the input module; An output module is used to output the target video obtained by the processing module.

18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the video generation method according to any one of claims 1 to 14.

19. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the video generation method according to any one of claims 1 to 14.

20. A computer program product comprising a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1 to 14.

Images