Video generation method, training method of video generation model and task platform

Abstract

The embodiment of the invention provides a video generation method, a training method of a video generation model and a task platform, and the method comprises the steps: obtaining a reference image and a reference audio, and enabling the reference image to at least comprise the visual information of a reference object; extracting visual features of the reference object according to the visual information in the reference image, and extracting audio features according to the reference audio; based on interaction characteristics between the visual characteristics and the audio characteristics, reference action information of a target object under the influence of the audio characteristics is predicted, and the target object is obtained based on the reference object; and generating a video corresponding to the target object based on the reference action information and the reference audio. By using the method, the matching degree of the visual information of the target object in the video and the corresponding audio can be improved, and the video presentation effect is correspondingly improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a video task processing, video generation model training, information processing method based on the video generation model, and task platform. Background Technology

[0002] With the rapid development of computer technology and artificial intelligence technology, it is now possible to generate images or videos required by users based on brief information input by users, and the requirements for the accuracy of video generation are also getting higher and higher.

[0003] In animated film production, virtual reality, and games, it is often necessary to create character videos and ensure that the character's visual information (such as body movements and lip movements) matches its corresponding audio (such as the character's spoken words).

[0004] However, in the currently generated videos of people, the degree of matching between the visual information of the people and their corresponding audio still needs to be improved, and the presentation effect of the videos needs to be improved. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a video generation method that can improve the matching degree between the visual information of the target object in the video and the corresponding audio, thereby improving the video presentation effect. One or more embodiments of this specification also relate to a training method for a video generation model, a task platform, a computing device, a computer-readable storage medium, and a computer program product.

[0006] According to a first aspect of the embodiments of this specification, a video generation method is provided, comprising:

[0007] Acquire a reference image and a reference audio, wherein the reference image contains at least visual information of a reference object;

[0008] Visual features of the reference object are extracted from the visual information in the reference image, and audio features are extracted from the reference audio.

[0009] Based on the interaction characteristics between the visual features and the audio features, the reference action information of the target object under the influence of the audio features is predicted, wherein the target object is obtained based on the reference object;

[0010] Based on the reference motion information and the reference audio, a video corresponding to the target object is generated.

[0011] According to a second aspect of the embodiments of this specification, a video generation method is provided, comprising:

[0012] Receive a video generation request uploaded from the front end, wherein the video generation request includes description information of the target object and reference audio;

[0013] Based on the description information, obtain a target image that at least contains the facial region of the target object;

[0014] Facial features are extracted from the facial region of the target object in the target image, and audio features are extracted from the reference audio.

[0015] Based on the interaction characteristics between the facial features and the audio features, reference facial motion information of the target object is predicted under the influence of the audio features, wherein the target object is obtained based on the reference object;

[0016] Based on the reference facial motion information and the reference audio, a video corresponding to the target object is generated;

[0017] The video is sent to the front end.

[0018] According to a third aspect of the embodiments of this specification, a method for training a video generation model is provided, comprising:

[0019] Obtain training samples and video labels, wherein the training samples include sample images and sample audio, and the sample images contain at least visual information of the sample objects;

[0020] Using an initial video generation model, visual features are extracted from the visual information in the sample image, and audio features are extracted from the reference audio.

[0021] Using the initial video generation model, and based on the interaction characteristics between the visual features and the audio features, the reference action information of the sample object under the influence of the audio features is predicted.

[0022] Using the initial video generation model, based on the reference motion information and the sample audio, a result video corresponding to the target object is generated;

[0023] Based on the comparison results between the resulting video and the video tags, the initial video generation model is adjusted to obtain the trained video generation model.

[0024] According to a fourth aspect of the embodiments of this specification, a video generation method is provided, applied to a task platform, comprising:

[0025] The device receives a model request sent by a terminal device, wherein the model request includes at least one of the following: a scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0026] Based on the model request, a corresponding video generation model is determined from at least one model, wherein the video generation model is trained based on the above-described video generation model training method.

[0027] According to a fifth aspect of the embodiments of this specification, a task platform is provided, including a request interface and a response unit;

[0028] The request interface is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: a scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0029] The response unit is configured to determine a corresponding video generation model from at least one model based on the model request, wherein the video generation model is trained based on the above-described video generation model training method.

[0030] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0031] Memory and processor;

[0032] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which implement the steps of the above method when executed by the processor.

[0033] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0034] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0035] In one embodiment of this specification, visual features can be extracted from the visual information of a reference object in a reference image, and audio features can be extracted from the reference audio. Based on the interaction characteristics between the visual and audio features, reference action information of the target object obtained based on the reference object under the influence of the audio features can be predicted. Then, a video corresponding to the target object can be generated by combining the reference audio. This approach, based on the influence of audio on visual features, achieves a precise correspondence between the reference audio and the visual information of the target object, improving the matching degree between the visual information of the target object and the reference audio in the obtained video, and enhancing the presentation effect of the generated video. Attached Figure Description

[0036] Figure 1 This is an application architecture diagram of a video generation method provided in one embodiment of this specification;

[0037] Figure 2 This is a flowchart of a video generation method provided in one embodiment of this specification;

[0038] Figure 3 This is a schematic diagram of a feature pyramid network provided in one embodiment of this specification;

[0039] Figure 4 This is a schematic diagram of the structure of a video generation model provided in one embodiment of this specification;

[0040] Figure 5 This is a flowchart of another video generation method provided in one embodiment of this specification;

[0041] Figure 6 This is a flowchart of a training method for a video generation model provided in one embodiment of this specification;

[0042] Figure 7 This is a comparison diagram of video generation effects provided in one embodiment of this specification;

[0043] Figure 8 This is a flowchart of another video generation method provided in one embodiment of this specification;

[0044] Figure 9 This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0045] Figure 10 This is a schematic diagram of the structure of a video generation device provided in one embodiment of this specification;

[0046] Figure 11 This is a schematic diagram of the structure of a training device for a video generation model provided in one embodiment of this specification;

[0047] Figure 12 This is a schematic diagram of another video generation device provided in one embodiment of this specification;

[0048] Figure 13 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0049] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0050] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0052] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0053] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0054] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0055] Currently, artificial intelligence (AI) technology can be applied to perform various tasks in different scenarios. For example, AI can be used to drive a character to perform corresponding actions (such as speaking) based on audio to generate corresponding videos. This audio-driven character technology can be applied to various situations such as digital humans, filmmaking, virtual reality, and video games. In videos generated using this technology, the degree of matching between the character's actions and the audio is crucial.

[0056] One related technology utilizes a one-shot talking head generation method to generate talking head videos. This method requires only a photograph of an individual with a given identity and a short audio clip to generate a talking head video for that identity, offering convenience and speed. A talking head refers to a portrait of a person whose facial expressions and lip movements can be driven by different control signals. Current specific methods include SadTalker and StyleTalk. This method requires mapping the audio signal to intermediate facial representations, such as 3D Morphable Model (3DMM) coefficients or keypoints, and then using a renderer to generate the video. Because the process of mapping audio signals to facial deformation is highly uncertain, and the intermediate representations lose some information from the audio, the talking head videos generated using this method are often of poor quality.

[0057] Another related technique utilizes neural radiance field (NeRF) speaker generation methods to generate speaker videos, such as AD-NeRF, ER-NeRF, and GeneFace. This method models the speaker's image by training on video frames and audio of the target speaker, thus generating relatively accurate facial deformation videos based on given audio. However, the model obtained by this method is tied to a specific speaker identity, making it difficult to generalize to different speakers. A separate model needs to be built for each speaker, resulting in high limitations on application scenarios and high complexity in the video generation process.

[0058] This specification provides a video generation method in the following embodiments. This method requires only an image of the target object and a driving audio clip to generate a high-quality video corresponding to the target object, thereby improving the quality of the generated video and reducing the complexity of video generation. This specification provides a video generation method. One or more embodiments of this specification also relate to a training method for a video generation model, a task platform, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0059] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the video generation method provided in this application can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. Figure 1 This is an application architecture diagram of a video generation method provided in one embodiment of this specification. Figure 1 In the application scenario shown, a large model (such as a video generation model) is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client devices 20 can include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can connect via, for example,... Figure 1 The graphical user interface shown interacts with the user to invoke the large model, thereby implementing the video generation method provided in the embodiments of this specification. If the client device has sufficient computing resources, the large model may not belong to the client device itself, and the client device may directly utilize the large model to implement the video generation method provided in the embodiments of this specification.

[0060] Figure 2 This is a flowchart illustrating a video generation method according to an embodiment of this specification. This method can be applied to a video generation apparatus, such as a video generation apparatus that can be... Figure 1 Server 10 in the middle. For example... Figure 1 As shown, the method may include steps 202 to 208.

[0061] Step 202: Obtain a reference image and a reference audio, wherein the reference image contains at least visual information of the reference object.

[0062] In one implementation, the reference image and reference audio can be sent by a user using a client device. For example, the client device can connect to the video generation device using an installed application with video generation capabilities, or a website or app with video generation capabilities, and then send the reference image and reference audio to the video generation device.

[0063] In another implementation, the user can send image and audio descriptions to the video generation device via a client device. The video generation device then retrieves corresponding reference images and audio from the internet or storage space based on this description, or generates the reference images and audio itself based on the description. For example, the user can input only the object type, object description, and image style that the image should contain, and the video generation device can generate an image that meets these requirements. Similarly, the user can input the content, style, and accent of the audio, and the video generation device can generate audio that meets these requirements. The video generation device can utilize a large model to generate the image or audio.

[0064] Optionally, the reference image and reference audio may also be acquired by the video generation device itself based on certain set tasks (such as certain functional test tasks), without being sent or specified by the client device, which is not limited here.

[0065] The reference image can be an image of a reference object. The reference object may have a head, face, and torso, and may be capable of uttering speech and performing other actions. The reference object may be a person, animal, anthropomorphic object, or animated character. The visual information of the reference object may include at least one of its facial region, head region, shape information, and clothing information. In addition to the visual information of the reference object, the reference image may also include other information, such as background objects or other objects. The reference audio in the embodiments of this specification can be used to drive the object to perform corresponding actions.

[0066] Step 204: Extract visual features of the reference object from the visual information of the reference object in the reference image, and extract audio features from the reference audio.

[0067] The video generation apparatus can first acquire visual information of a reference object from a reference image, and then extract visual features from that visual information. For example, it can first identify the reference object from the reference image, and then extract information from the identified object region to obtain the visual information of the reference object. The video generation apparatus can perform masking processing on the reference image to obtain a mask image that only includes the visual information of the reference object. The video generation apparatus can extract all visual features of the reference object from the reference image, or it can extract only the visual features of a portion of the reference object as needed, such as extracting only the facial visual features of the reference object. For example, the video generation apparatus can perform sampling processing on the visual information to achieve feature extraction.

[0068] In some implementations, in step 202, the reference image acquired by the video generating device may only include the visual features of the reference object. In this case, feature extraction can be performed directly on the reference image in step 204.

[0069] In some implementations, the video generation apparatus may use a one-dimensional fully convolutional network to process reference audio and obtain the audio features of the reference audio.

[0070] Step 206: Based on the interaction characteristics between visual features and audio features, predict the reference action information of the target object under the influence of audio features, wherein the target object is obtained based on the reference object.

[0071] The purpose of the video generation method provided in the embodiments of this specification is to generate a video corresponding to a target object, which is obtained based on a reference object. The target object can be the reference object directly, or it can be an object obtained by adjusting or transforming the reference object. For example, if the reference object is a real person, the target object can be a cartoon character, comic character, or a predicted image of the real person after the target duration, generated based on that real person.

[0072] Although the target object and the reference object may have some differences, since the target object is obtained based on the reference object, the relationship between the visual features and audio features of the target object may still be the same as that of the reference object. The video generation device can determine the state of the target object under audio drive by analyzing the visual features and audio features of the reference object.

[0073] The video generation device can analyze the correlation between visual and audio features to obtain the interaction characteristics between them. These interaction characteristics refer to the properties of the interaction between visual and audio features, reflecting the influence of audio features on visual features at various locations on a reference object. Furthermore, based on these interaction characteristics, the influence of audio features on the target object's actions is determined, obtaining reference action information of the target object driven by the reference audio. This reference action information can characterize the actions performed by the target object, such as facial movements like smiling, opening the mouth, or closing the eyes, or other body movements like shaking the head, waving the hand, or shaking the leg. In the embodiments of this specification, by determining the interaction characteristics between visual and audio features, the relationship between audio and visual features at various locations on the target object can be captured, thereby enabling accurate prediction of the target object's actions.

[0074] Step 208: Generate the video corresponding to the target object based on the reference motion information and reference audio.

[0075] Based on reference motion information, the video generation device can determine the movement of a target object driven by a reference audio. For example, it can determine the facial movements of the target object when it speaks the speech corresponding to the reference audio. Furthermore, the video generation device can use the reference motion information to generate multiple frames of images of the target object, obtaining an image frame sequence that reflects the facial movements of the target object. The video generation device can determine the number of image frames to be generated according to a set frame rate and correspondingly match the image frames with the audio frames of the reference audio. By combining the image frame sequence with the reference audio in chronological order, a video corresponding to the target object can be generated. For example, this video could be a video of the target object speaking the speech corresponding to the reference audio. In some embodiments, the video may also incorporate the target object's body movements and clothing movements.

[0076] In some implementations, step 208 may include generating an initial image frame sequence based on reference motion information; performing super-resolution processing on each image frame in the initial image frame sequence to obtain a target image frame sequence; and combining the target image frame sequence with reference audio to generate a video corresponding to the target object.

[0077] For example, the reference image can have a lower resolution. After obtaining the initial image frame sequence, the video generation device can invoke super-resolution processing to upscale the initial image frames, allowing them to carry richer background and more detailed information, thus obtaining the target image frame sequence. Optionally, the target image frames can also contain more texture features of the target object. In this way, the video generated based on the target image frame sequence can have better quality and display effect.

[0078] After generating a video corresponding to the target object, the video generating device can send the video to a client device. This client device can be a client device that sends or specifies reference images and reference audio. The video generating device can also send this video as a sample video to all its connected client devices.

[0079] In summary, the video generation method provided in this specification can extract visual features from the visual information of a reference object in a reference image and extract audio features from the reference audio. Based on the interaction characteristics between these visual and audio features, it predicts the reference action information of the target object obtained based on the reference object under the influence of the audio features, and then combines the reference audio to generate a video corresponding to the target object. This approach, based on the influence of audio on visual features, achieves a precise correspondence between the reference audio and the visual information of the target object, improving the matching degree between the visual information of the target object and the reference audio in the obtained video, and enhancing the presentation effect of the generated video.

[0080] The optional implementation methods of the video generation method provided in the embodiments of this specification are further explained below.

[0081] In step 206 above, the video generation device performs cross-modal (i.e., visual and audio modal) feature interaction. Step 206 may include: performing correlation analysis on visual features and audio features to obtain correlation information between visual features and audio features; wherein, the visual features include features of multiple visual positions on the target object, and the correlation information characterizes the degree of correlation between each visual position and the audio features; based on the correlation information, predicting reference action information of the target object under the influence of audio features.

[0082] Video generation devices can perform correlation analysis on visual and audio features. For example, cross-attention computation can be used to analyze the interaction between visual and audio features to determine their interaction characteristics. Cross-attention computation is widely used in machine learning and deep learning for multimodal processing, sequence-to-sequence (Seq2Seq) models, and other tasks that require fusing information from different sources. It is an extension of the self-attention mechanism, which not only considers the dependencies within a sequence but also establishes dynamic associations between two different sequences or data structures, enabling cross-sequence information interaction.

[0083] In cross-attention computation, the model simultaneously considers the query, key, and value. It is typically used to fuse information from one sequence (the source sequence) with information from another sequence (the target sequence). For example, in image caption generation, the target sequence (i.e., the text description) uses a cross-attention mechanism to focus on the source sequence (i.e., the image feature vector), selecting the most relevant parts to generate a more accurate text description. This mechanism first calculates the similarity scores between the query and all keys, then weights each value based on these scores, finally obtaining the adjusted representation of the target sequence according to the source sequence. This mechanism allows the model to focus on specific parts of the input sequence, enhancing the interactivity and understanding of information.

[0084] The video generation device can perform multi-head cross attention (MHCA) calculations on visual and audio features. For example, visual features can be used as query information, and audio features as key-value pairs, to obtain corresponding correlation information, which can be represented by a set of numerical values. This correlation information characterizes the interaction between features of different modalities; for instance, it can characterize the correlation between each visual position of the target object and the audio features. In the case where the visual feature is the facial visual feature of the object, this correlation information can characterize the correlation between the features of each position or sub-region of the face and the audio features. This correlation information can serve as prior information for modeling dynamic target objects in a region-aware manner. Based on this correlation information, the video generation device can predict the influence of audio features on various positions of the target object, thereby obtaining reference action information of the target object.

[0085] For example, the video generating device can calculate this correlation information according to the following formula. If the formula is... Among them, F cm Indicates reference action information, F represents visual features, a represents audio features, and W represents... Q W K and W V Let T represent the projection matrices with hidden dimension d corresponding to the query information, key, and value, respectively, and T denotes transpose.

[0086] The video generation device can utilize a cross-attention module to process visual and audio features to obtain correlation information. This cross-attention module can be a deep learning model, W. Q W K and W V These can be the parameters of the model, which can be trained based on the changes in the visual features of objects corresponding to multiple audio sources.

[0087] In some embodiments, the video generation apparatus can also generate video based on a desired viewpoint, such as generating a video of a target object from the target viewpoint. The video generation method provided in the embodiments of this specification may further include acquiring information about the target viewpoint; and determining the spatial position features of the portion of the target object presented from the target viewpoint based on the extracted visual features and the target viewpoint information. Accordingly, in step 208 above, generating a video corresponding to the target object based on reference motion information and reference audio includes: obtaining target motion information of the target object from the target viewpoint based on reference motion information, audio features, and spatial position features; and generating a video corresponding to the target object based on the target motion information and reference audio.

[0088] The target viewpoint may differ from the viewpoint corresponding to the reference image (e.g., referred to as the source viewpoint). The target viewpoint can be specified by the user through a client device; for example, the target viewpoint information can be sent by the client device along with the reference image and reference audio. Alternatively, the target viewpoint can be determined automatically by the video generation device based on other information; this is not limited here. The viewpoint corresponding to the image is determined by the camera pose. In the embodiments of this specification, the target viewpoint can be characterized by the camera pose. For example, the camera pose corresponding to the target viewpoint is P = (R, T), where R represents the rotation matrix and T represents the translation matrix; that is, the camera pose is represented by the rotation and translation information of the camera relative to a certain reference camera state.

[0089] The visual information required to be presented by the same object differs depending on the viewing angle. For example, from a frontal viewpoint, the visual information of a person includes all information about the face and some information about the left and right sides; from a left-side viewpoint, the visual information of a person includes all information about the left side and some information about the face. In the embodiments of this specification, the reference image acquired by the video generation device can only represent the visual information of the reference object from one viewpoint, and the visual features extracted based on this visual information also represent the features from that viewpoint. The video generation device can process these visual features to determine the parts of the target object that need to be presented from the target viewpoint, as well as the specific features of these parts, such as the specific spatial positions of points on these parts. The specific shape of the target object can be determined by the positions of its various regions or points. For example, the shape of the nose is determined by the positions of the nostrils, the tip of the nose, and the bridge of the nose. Therefore, the spatial positions of the regions or points in the parts of the target object that need to be presented can be used to represent the target object. In the embodiments of this specification, spatial position features are used to describe the spatial position of the parts of the target object that need to be presented from the target viewpoint.

[0090] The video generation device can construct a 3D image of a target object based on visual features extracted from a source viewpoint of a reference image. It then determines the features of the 3D spatial points to be presented within this 3D image from the target viewpoint, thus obtaining the spatial position features of the target object from that viewpoint. From the target viewpoint, it can be considered that there are multiple rays emanating from the camera position corresponding to that viewpoint, all of which pass through the 3D image of the target object. The points located on each ray in the 3D image are the 3D spatial points to be presented. These spatial position features can be used in the subsequent pixel rendering process.

[0091] The reference motion information obtained earlier represents the motion information that the target object should present under the influence of audio features from the source viewpoint. This reference motion information reflects the relationship between audio and motion. When the video to be generated corresponds to the target viewpoint, this motion information needs to be adjusted accordingly. Spatial position features characterize the positions of each point of the target object that should be presented from the target viewpoint, that is, the form of the target object when it is not moving from the target viewpoint. When the target object moves, it moves these points. Therefore, the video generation device can adjust the spatial position features based on the reference motion information, so that each point in the part presented from the target viewpoint moves according to the corresponding action, thus obtaining the form of the target object when it is moving from the target viewpoint. Since the influence of audio features on different positions in the target object varies, and the positions of the target object to be presented from the target viewpoint and the source viewpoint are different, the motion information of each position from the target viewpoint cannot be fully determined based solely on the reference motion information. Therefore, the adjustment of the spatial position features also needs to be combined with the audio features. In this way, after adjustment, the target motion information of the target object from the target viewpoint can be obtained, and then the video corresponding to the target object from the target viewpoint can be generated.

[0092] In one implementation, to adjust the reference motion information, the video generation device can predict object deformation based on spatial location features, audio features, and the reference motion information to obtain feature offset information of the spatial location features; wherein, the feature offset information characterizes the visual deformation features of the target object under the influence of the target viewpoint and audio features. By combining the spatial location features and the feature offset information, the target motion information of the target object under the target viewpoint is obtained.

[0093] Spatial location features represent the position that the target object should appear in from the target's perspective, given that the same action is taken by the reference object in the reference image. Reference audio is audio over a period of time, during which the target object should perform corresponding actions based on the audio. For example, when a person speaks, their eyes, nose, and face will also move, and the movements of these parts will differ depending on the content being spoken.

[0094] Video generation devices can utilize object deformation prediction models to process spatial location features, audio features, and reference motion information to obtain feature offset information of the spatial location features, that is, the movement of the spatial location features under the influence of the audio features. Feature offset information Δx = deform(x, a, F) cm ), where x represents the spatial location feature. This object deformation prediction model can be trained based on the object's image information from a first-viewpoint and the audio and corresponding object action information from a second-viewpoint. After obtaining the feature offset information, the spatial location feature can be added to this feature offset information to determine the adjusted spatial location feature, that is, the adjusted spatial location feature x' = x + Δx. This adjusted spatial location information can then be used as the target action information.

[0095] In the embodiments described in this specification, videos of target objects from different perspectives can be generated based on images from only one viewpoint, which can improve the ease of video generation and reduce the complexity of user operations.

[0096] In some implementations, step 202 above, which involves extracting visual features of a reference object from a reference image, includes: extracting visual features of the reference object at different resolutions based on the visual information in the reference image to obtain multi-scale visual features; wherein, the multi-scale visual features include visual features of the reference object at different resolutions. This can improve the richness of feature extraction.

[0097] In the embodiments of this specification, different scales refer to spatial scales, which indicate the level of detail in which features are collected or represented in spatial dimensions. This can affect the quality of sampling results, detail preservation, and subsequent processing and analysis capabilities. Spatial scales can include resolution, pixel size, etc. Taking resolution as an example, the embodiments of this specification sample image frames at 1 / 8, 1 / 16, and 1 / 32 of the original resolution, respectively, to obtain visual features at 1 / 8, 1 / 16, and 1 / 32 resolutions. Each resolution's visual feature represents a scale's visual feature, resulting in multi-scale visual features encompassing visual features at multiple resolutions. The features extracted at each scale in the embodiments of this specification can include structural and texture information of the reference image. The proportions of structural and texture information contained in features at different scales differ; for example, larger scale features (corresponding to smaller resolutions) contain more structural information and less texture information, while smaller scale features contain more texture information and less structural information.

[0098] In some implementations, the video generation apparatus utilizes a feature pyramid network to extract visual features. Figure 3This is a schematic diagram of a feature pyramid network provided in one embodiment of this specification. The feature pyramid network may include n downsampling blocks and n upsampling blocks arranged sequentially, where n ≥ 2, and the nth downsampling block is adjacent to the nth upsampling block. Lower-level downsampling blocks can be connected to higher-level upsampling blocks to fuse visual information at different scales, obtaining more complete visual information. For example, the output of the i-th downsampling block can be connected to the input of the (i+1)-th upsampling block, where 1 ≤ i ≤ n-1. Figure 3 As shown in the embodiment of this specification, n=4 is used as an example. The n downsampling blocks are D0, D1, D2 and D3, and the n upsampling blocks are U0, U1, U2 and U3. The downsampling block D3 and the upsampling block U3 are adjacent.

[0099] The video generation device processes the visual information of the reference object in the reference image through n downsampling blocks arranged sequentially in a feature pyramid network to obtain features at n scales. Each downsampling block produces features at one scale, with larger scales and smaller resolutions at each subsequent block. The video generation device concatenates the features output from the i-th downsampling block with the features output from the (i+1)-th upsampling block and reduces the number of feature channels to obtain the features output from the i-th upsampling block. For example, each upsampling block can include this convolutional layer; the video generation device can use a 1*1 convolutional layer to reduce the number of feature channels. The feature scales output from the i-th downsampling block and the (i+1)-th upsampling block can be the same. The features output from the n-th downsampling block can be used as the features output from the n-th upsampling block. These n upsampling block outputs represent the visual features of the reference object at n resolutions, thus obtaining multi-scale visual features.

[0100] For example, upsampling block U i Output characteristics Among them, Ui(F i+1 ) indicates the upsampling block U i The input is the upsampled block U i+1 The output F i+1 ;conv i (.) indicates the upsampling block U i A 1x1 convolutional layer is used for feature mapping; [...] indicates concatenation along the channel direction.

[0101] When extracting multi-scale visual features of a reference object from a reference image, step 206 above, based on the interaction characteristics between visual and audio features, predicts the reference action information of the target object under the influence of audio features. This can include: reshaping the multi-scale visual features into visual features of the same scale and concatenating them along the feature channel dimension to obtain auxiliary visual features; flattening the auxiliary visual features according to the target feature size and aggregating the flattened auxiliary visual features to obtain aggregated visual features; and performing feature interaction between the aggregated visual features and audio features to predict the reference action information of the target object under the influence of audio features. The aggregated visual features can contain rich information about the reference object.

[0102] Multi-scale visual features represent the characteristics of an object in different aspects. Video generation devices need to predict the precise movements of the target object based on the overall information of the target object. Video generation devices can aggregate these multi-scale visual features, allowing information exchange between features at different scales, to ensure that the aggregated visual features represent the overall information of the target object. Feature aggregation has certain requirements on the size of the input features, such as the target feature size. Since the resolution of visual features at different scales is also different, it is difficult to directly convert the feature size of multi-scale visual features. Therefore, video generation devices can first reshape the multi-scale visual features into visual features of the same scale.

[0103] Reshaping refers to the process of transforming feature data from one shape to another so that the transformed features meet the input requirements of a specific layer or model. Feature reshaping can involve the transformation of one-dimensional, two-dimensional, or multi-dimensional data. Through feature reshaping, effects such as data format adjustment, data integration, and simplification of data structures can be achieved. In the embodiments of this specification, the scale (e.g., resolution) of the visual features to be satisfied after reshaping can be specified, and multi-scale visual features are transformed into visual features of that scale through feature reshaping. The visual features in the embodiments of this specification are all image features; therefore, these features can also be considered as forming a feature map. The size or shape of the features mentioned below refers to the size or shape of the corresponding feature map. For example, multi-scale visual features can be reshaped to a resolution of H... s *W s The visual feature map, where H s This refers to the height, W s It refers to the width.

[0104] The reshaped visual features of the same scale are concatenated along the feature channel dimension. This concatenation operation can be implemented using connection functions provided by deep learning frameworks. Concatenating along the feature channel dimension means merging features corresponding to different feature channels to obtain a higher-dimensional feature representation, ensuring that the concatenated result can express richer features. For example, if both feature maps have a shape of C*H*W, where C represents the feature channel, concatenating these two feature maps along the feature channel dimension will result in an auxiliary visual feature with a shape of 2C*H*W. During concatenation, the video generation device can concatenate multi-scale visual features together to obtain a single auxiliary visual feature; alternatively, the video generation device can also divide multi-scale visual features into multiple groups for concatenation to obtain multiple auxiliary visual features of the same scale.

[0105] Next, the obtained auxiliary visual features can be flattened, which refers to converting the multi-dimensional feature map into a one-dimensional vector. In the embodiments of this specification, a target feature size that the flattened feature must meet can be specified. The video generation device flattens the auxiliary visual features based on this target feature size, that is, it converts the auxiliary visual features into features of the target feature size. Assuming the target feature size is C*D, the video generation device can convert H*W in the stitched auxiliary visual features to D, thus flattening the auxiliary visual features according to the target feature size. If multiple auxiliary visual features are obtained during stitching, each auxiliary visual feature can be flattened according to the target feature size during flattening.

[0106] The video generation device can utilize a slot attention module to aggregate flattened auxiliary visual features, performing feature exchange at different scales to obtain aggregated visual features. The shape of the aggregated visual features can be B*(H s *W s )*D slot Where B refers to the number of images processed in a batch, and H... s This refers to the height, W s This refers to the width, D slot This refers to the feature vector dimension of each slot, that is, the length of the feature vector of the object represented by each slot. In the embodiments of this specification, each slot of the slot attention module represents a visual feature of one scale.

[0107] In some implementations, when the video generation device acquires information from the target viewpoint, it can further reconstruct multi-scale visual features into three-plane features; combining the target viewpoint information and the three-plane features, it determines the three-plane features of each three-dimensional spatial point in the portion of the target object presented from the target viewpoint, thereby obtaining the spatial position features of the portion of the target object presented from the target viewpoint; wherein, the spatial position features include the three-plane features of each three-dimensional spatial point. Accordingly, step 208 above, generating a video corresponding to the target object based on reference motion information and reference audio, includes: combining spatial position features, reference motion information, and reference audio to generate a video corresponding to the target object.

[0108] After extracting multi-scale visual features, the video generation device can reshape the visual features at each scale into a tri-plane feature. This tri-plane refers to three feature planes with mutually orthogonal axes, each with dimensions N*N*C, where N*N represents the spatial resolution and C represents the number of feature channels. The video generation device projects the visual features at each scale onto these three feature planes, obtains the corresponding feature vectors, and then sums the feature vectors corresponding to the three feature planes to obtain the corresponding tri-plane feature. The shape of the tri-plane feature at the i-th scale is B*3*C. i *H i *W i Where B refers to the batch size, "3" indicates a three-plane configuration, and C... i H i and W i These represent the number of channels, height, and width at the i-th scale, respectively.

[0109] The method for determining the spatial position features of a target object from the target's perspective based on these three-dimensional planar features can be found in the previously described introduction to determining spatial position features, and will not be elaborated upon here. The video generation device can combine these three-dimensional planar features to construct a three-dimensional image of the target object. Furthermore, by combining information from the target's perspective and using projection or bilinear interpolation, the features of the three-dimensional spatial points to be presented in the three-dimensional image from the target's perspective are determined, thus obtaining the spatial position features of the target object from the target's perspective. Since the three-dimensional planar features can better represent features in three-dimensional space, the construction effect of the three-dimensional image of the target object can be improved, and the subsequent pixel rendering effect based on the spatial position features can also be improved.

[0110] In some implementations, generating a video corresponding to a target object by combining spatial location features, reference motion information, and reference audio may include: performing neural rendering based on spatial location features and reference motion information, and combining information other than the reference object in the reference image to obtain an image frame sequence; combining the image frame sequence and reference audio to generate a video corresponding to the target object. The method of combining spatial location features, reference motion information, and reference audio during the generation of the video corresponding to the target object can be referred to the previous description regarding adjustments to spatial location features.

[0111] Video generation devices can render pixels of the target object based on the spatial location features and reference motion information corresponding to each time point in the video, and also render information other than the reference object in the reference image, resulting in an image frame sequence. For example, neural rendering models can be used to render pixels through neural rendering. Neural rendering learns rendering from existing observations using neural networks. The main idea of ​​neural rendering is to combine insights from classical (physically based) computer graphics with deep learning. The goal of neural rendering is to generate photorealistic images in a controllable manner. Neural rendering can control various aspects of the scene, such as lighting, camera parameters, pose, geometry, appearance, and semantic structure. This rendering method ensures that the resulting image frames have a strong sense of realism, thus improving the display quality of the resulting video.

[0112] During this rendering process, the video generation device can utilize a three-plane decoder to output corresponding colors and densities based on the result of adjusting spatial position features according to reference motion information, and then perform pixel rendering based on these colors and densities. This three-plane decoder can be a multilayer perceptron (MLP).

[0113] For example, the video generation device can render pixels on an image frame according to the NeRF training paradigm and the following volume rendering formula. The volume rendering formula is: Where C(r) represents the pixel color, r(t) represents the target viewpoint information, a represents the audio feature, σ represents the opacity, c represents the color density, and T(t) represents the spatial length (or step size) of pixel rendering, such as the distance between adjacent pixels; t f and t n These represent the upper and lower limits of the rendering range, respectively.

[0114] The video generation method provided in the embodiments of this specification can be executed using a video generation model, which includes a feature extraction module, an action prediction module, and a video generation module. Accordingly, each step in the above video generation method can be executed using its respective module. These modules can also be deep learning models.

[0115] Step 204 above involves extracting visual features of the reference object from the visual information in the reference image and extracting audio features from the reference audio. This includes using a feature extraction module to extract features from the visual information in the reference image and the reference audio, respectively, to obtain the visual features and audio features of the reference object. For example, this feature extraction module may include a visual feature extraction submodule and an audio feature extraction submodule, which extract features from the reference image and the reference audio, respectively.

[0116] Step 206 above, based on the interaction characteristics between visual features and audio features, predicts the reference action information of the target object under the influence of audio features, including: using the action prediction module to process the visual features and audio features, and predicting the reference action information of the target object under the influence of audio features.

[0117] Step 208 above generates a video corresponding to the target object based on the reference motion information and reference audio, including: processing the reference motion information and reference audio using a video generation module to generate a video corresponding to the target object.

[0118] In some implementations, after generating a video corresponding to the target object based on reference motion information and reference audio in step 208 above, the video generation method may further include: adjusting at least one of the feature extraction module, motion prediction module, and video generation module upon receiving feedback information regarding issues with the video.

[0119] Following step 208 above, the video generation device can send the generated video to a client device that specifies the reference image and reference audio. If the user on the client device believes the video does not meet their needs, they can send feedback information to the video generation device, which may include a specific description of the user's opinion. The video generation device can then continue training the video generation model based on this feedback to improve the quality of subsequent videos generated by the model. For example, the video generation device may fine-tune at least one module in the video generation model based on the feedback, and adjust the parameters of that module. This module may be a sub-model responsible for certain functions within the video generation model. Based on user instructions, the video generation device can re-execute the video generation method using the adjusted video generation model to regenerate the video corresponding to the target object and provide feedback to the user.

[0120] This video generation model can be trained based on a target loss function. This target loss function can be... Among them, L pix For pixel-level reconstruction loss, L per It is perceived loss, Ladv It is to combat losses, L sync For audio-visual consistency loss, L deform The offset prediction loss can be the loss that is used during the training of the multiple sub-modules included in the video generation model.

[0121] For example, L pix A loss can be provided for the visual feature extraction sub-model, which is used to extract visual features from a reference image. Optionally, it can also be used to reconstruct the visual features into three-plane features and determine the spatial location features corresponding to the target viewpoint. per It could be the loss corresponding to a feature interaction sub-model, which is used to interact visual and audio features; for example, it could be a cross-attention model. deform It could be the loss corresponding to an object deformation prediction model, which can be used to generate feature offset information of spatial location features under the influence of target viewpoint and reference audio. sync This could be the loss function corresponding to the audio-visual consistency analysis model. This model can also be used in the video generation model to analyze the generated video. If the audio-visual consistency requirements are not met, the video is regenerated.

[0122] Figure 4 This is a schematic diagram of the structure of a video generation model provided in one embodiment of this specification. The schematic diagram also illustrates a simplified flow of the video generation method provided in this embodiment. Figure 4 As shown, the video generation model includes a feature extraction module 401, an action prediction module 402, and a video generation module 403. The feature extraction module 401 includes a visual feature extraction submodule 4011 and a feature reconstruction submodule 4012; the action prediction module 402 includes a feature interaction submodule 4021 and an object deformation prediction submodule 4022; the video generation module 403 may include an image color determination submodule 4031, an image information combination submodule 4032, and a super-resolution module 4033. The visual feature extraction submodule 4011 includes a feature pyramid network, and the feature reconstruction submodule 4012 is used to construct the three-plane features of each three-dimensional spatial point under the target's viewpoint. The feature interaction submodule 4021 includes a multi-head cross-attention (MHCA) model, a layer normalized low-level neural network (LN) unit, and a forward neural network (FFN). The object deformation prediction submodule 4022 includes an encoder and a decoder.

[0123] The reference image can be input into the video visual feature extraction submodule 4011 to extract multi-scale visual features using each sampling layer. The feature reconstruction submodule 4012 can convert these multi-scale visual features into three-plane features, and construct three-plane features of each three-dimensional spatial point under the target's viewpoint based on the target's vision to obtain spatial position features. After the multi-scale visual features are reshaped and channel-connected, and aggregated by the slot attention module to obtain aggregated visual features, they are input into the feature interaction submodule 4021; the audio features extracted from the reference audio are also input into the feature interaction submodule 4021. The feature interaction submodule 4021 uses the audio features as key information and the aggregated visual features as query information to perform cross-attention calculation. The calculation results are then passed through the LN unit and FFN unit and input into the object deformation prediction submodule 4022. The spatial position features and audio features are also input into the object deformation prediction submodule 4022, and then the object deformation prediction submodule 4022 can output target action information. After the object deformation prediction submodule 4022, an adder is connected. The spatial position features and the target motion information are added together by this adder to obtain the target motion information. The target motion information can still be represented using three-plane features. The image color determination submodule 4031 (three-plane decoder) decodes the target motion information to obtain the color of the target pixel of the target object. This color is then used for rendering to obtain an auxiliary image containing the target object. The image information combination submodule 4032 combines this auxiliary image with other information in the reference image to obtain a more refined image frame. The super-resolution module 4033 performs super-resolution processing on this image frame to obtain a target image frame with a more refined display effect. This frame is then combined with the reference audio to obtain the video corresponding to the target object.

[0124] In some embodiments, the video generation apparatus can also perform audio-visual consistency analysis on the generated video to ensure that a high-quality video is generated. For example, after step 208 above, the video generation method provided in this specification further includes: acquiring multiple image frames and multiple audio frames that correspond one-to-one in the video; extracting features from the multiple image frames and multiple audio frames, and determining the feature similarity between each image frame and its corresponding audio frame to obtain the audio-visual consistency analysis result of the video. Optionally, this audio-visual consistency analysis process can be independent of the aforementioned video generation process, such as not affecting the video generated during the video generation process.

[0125] The video generation device can acquire multiple corresponding image frames and multiple audio frames in a video according to a set acquisition frequency. For example, if the acquisition frequency is 25 frames per second, the video generation device can acquire one image frame every 1 / 25 of a second and divide the audio within one second into 25 audio frames. The video generation device may include a visual feature extraction module and an audio feature extraction module. The visual feature extraction module takes the image frame sequence as input and outputs the first embedded feature e1 of each extracted image frame. The audio feature extraction module takes the audio sequence as input and outputs the second embedded feature e2 of each extracted audio frame.

[0126] The video generation device can analyze the consistency level between audio and video sequences by calculating the cosine similarity between each first embedded feature e1 and its corresponding second embedded feature e2. For example, this similarity can be... Where ε represents the set threshold.

[0127] In the embodiments of this specification, the video generation device can analyze all information in an image frame to determine audio-visual consistency. Optionally, the video generation device can also analyze only a portion of the information in an image frame to determine whether that portion of the information matches the reference audio. For example, the video corresponding to the target object is a video of the target object speaking the reference audio. The audio-visual consistency analysis performed by the video generation device can be an analysis of whether the lip movements of the target object when speaking the reference audio are accurate. In this approach, the video generation device can only acquire the mouth region information of the target object in the image frame, and perform feature extraction on the mouth region information to perform audio-visual consistency analysis. For example, the lip region information can be obtained by cropping using a pre-extraction strategy of the actual image bounding box.

[0128] The video generation device can utilize an audio-visual consistency analysis model to perform this audio-visual consistency analysis. This model is trained based on visual information from multiple objects performing actions based on audio. The loss value used to train this audio-visual consistency analysis model can be formulated as follows: in, e represents the embedding features of an image frame sequence gt This represents the embedding features of the real audio. During training, the obtained asynchronous audio and visual feature pairs can be used as negative samples to retrain the model. The audio-visual consistency analysis model can be trained before the aforementioned video generation model and can be kept frozen during the training of the video generation model.

[0129] This specification demonstrates a method to quickly generate videos of a target object performing actions based on a given reference image and audio, achieving better audio-visual consistency. It enables the generation of high-quality, realistic videos of target objects even with a single sample. For example, if the target object is a speaker, the video can include the speaker's speech. This specification utilizes a feature pyramid structure to model different levels of visual information from coarse to fine, fully extracting the texture and structural features of the reference image. Furthermore, it employs cross-attention-based cross-modal action offset prediction to model the actions of the target object influenced by audio (such as facial movements), accurately modeling the mapping from audio to target object actions (such as lip movements). Additionally, audio-visual consistency analysis is introduced to constrain the temporal consistency of the generated video, further enhancing the realism of the audio-driven speaker.

[0130] The following is in conjunction with the appendix Figure 5 Taking the video generation method provided in this manual as an example of generating a video of a target object speaking, the video generation method will be further explained. Figure 5 This is a flowchart of another video generation method provided in one embodiment of this specification. This method can be applied to a video generation apparatus, which can be... Figure 1 Server 10 is shown in the image. (As shown) Figure 5 As shown, the method may include steps 502 to 512. This method can be referenced from the aforementioned method for... Figure 2 The video generation method shown here will not be repeated here, as the content introduced earlier has already been covered.

[0131] Step 502: Receive the video generation request uploaded by the front end. The video generation request includes the description information of the target object and reference audio.

[0132] This front-end can correspond to the aforementioned client device. The method for the front-end to upload video generation requests can be found in the relevant description of the video generation device obtaining reference images and reference audio in step 202. The front-end can display a video generation page by triggering an application, webpage, or mini-program. Users can use this video generation page to trigger the front-end to send a video generation request to the video generation device, instructing the device to generate a video and specifying the conditions that the generated video must meet. For example, users can enter information indicating these conditions on the video generation page and click the "Confirm Video Generation" control to trigger the front-end to send the video generation request. This condition information may include a description of the target object and reference audio; the video generation request can carry this condition information.

[0133] Step 504: Obtain a target image that contains at least the facial region of the target object based on the description information.

[0134] For example, the description information of the target object carried in the video generation request can be textual. The user can instruct the video generation device to first generate an image of the target object that meets the description information, and then generate a video corresponding to the target object based on the generated image. The description information may include the type, approximate state, and image style of the target object. For example, if a description includes "a girl smiling in a flower garden, illustration style," the video generation device can first construct a corresponding image based on the description information. This image may include the target object, other objects, and corresponding background information. The video generation device can obtain the facial region of the target object from this image to obtain the target image. Optionally, the image constructed by the video generation device based on the description information may only include the target object, and correspondingly, the constructed image can be directly determined as the target image.

[0135] For example, the description information can be an image of the target object. Users can directly identify the image-based description information as the target image, or they can extract the facial region of the target object from the image-based description information to obtain the target image.

[0136] Step 506: Extract facial features from the facial region of the target object in the target image, and extract audio features from the reference audio.

[0137] In the embodiments of this specification, the facial feature is a visual feature of the target object's face. Step 506 can refer to the aforementioned description of step 204, and will not be repeated here.

[0138] Step 508: Based on the interaction characteristics of facial features and audio features, predict the reference facial action information of the target object under the influence of audio features, wherein the target object is obtained based on the reference object.

[0139] In the embodiments of this specification, step 508 can refer to the foregoing description of step 206, and will not be repeated here.

[0140] Step 510: Generate a video corresponding to the target object based on the reference facial motion information and reference audio.

[0141] In the embodiments of this specification, step 510 can refer to the foregoing description of step 208, and will not be repeated here. The video corresponding to the target object can be a video of the target object speaking the reference audio.

[0142] Step 512: Send the video to the front end.

[0143] After generating the video corresponding to the target object, the video generation device can send the video back to the front end via a communication connection for display to the user. The user can play the video and determine if it meets their needs. If the user is dissatisfied with the generated video, they can send feedback to the video generation device through the front end. The video generation device can then regenerate the video corresponding to the target object based on this feedback. Alternatively, the device can adjust its video generation model based on the feedback to regenerate the video corresponding to the target object.

[0144] exist Figure 5 The provided video generation method can also perform audio-visual consistency analysis after video generation to ensure that the lip movements of the target object in the generated video match the audio. For details, please refer to the previous section on... Figure 2 This section provides an introduction to the video generation methods provided.

[0145] Before executing the video generation method provided in the embodiments of this specification, the video generation device may first execute a training method for the video generation model to train the video generation model. Then, the trained video generation model is used to execute the video generation method. Figure 6 This is a flowchart illustrating a training method for a video generation model according to an embodiment of this specification. This training method can be applied to a model training device, which may be the same as or different from the video generation device that performs the video generation method described above. In different cases, after the model training device has trained the video generation model, the video generation device then acquires the video generation model to execute the video generation method described above.

[0146] Figure 6 This is a flowchart of a training method for a video generation model provided in one embodiment of this specification, specifically including the following steps 602 to 610. Figure 6 The steps in this process can be referred to in the previous section. Figure 2 The implementation methods of the corresponding steps are not elaborated here as they have been introduced earlier.

[0147] Step 602: Obtain training samples and video labels. The training samples include sample images and sample audio. The sample images contain at least visual information of the sample objects.

[0148] The training samples are the data that the video generation model can process, and can be historical data that has already yielded the processing results corresponding to the training task. The video labels corresponding to the training samples are the correct processing results that should be obtained when performing the training task on those training samples. They are used to determine the prediction loss of the initial video generation model for the training task, thereby adjusting the model parameters of the initial video generation model.

[0149] Step 604: Using the initial video generation model, extract visual features from the visual information in the sample images and extract audio features from the reference audio.

[0150] Step 604 is the feature extraction process for training samples during the model training phase. Its specific implementation process is similar to that for... Figure 2 The process of feature extraction from reference images and reference audio during the model usage phase, as described in the introduction, is similar. For details, please refer to the relevant introduction about step 204 above.

[0151] Step 606: Using the initial video generation model, based on the interaction characteristics between visual and audio features, predict the reference action information of the sample object under the influence of audio features.

[0152] Step 606 describes the processing of visual and audio features during the model training phase. Its specific implementation process and targeted... Figure 2 The process of processing visual and audio features during the model usage phase is similar as described in the introduction. For details, please refer to the relevant introduction about step 206 above.

[0153] Step 608: Using the initial video generation model, generate the result video corresponding to the sample object based on the reference motion information and sample audio.

[0154] Step 608 is the video generation process during the model training phase, and its specific implementation process is similar to that for... Figure 2 The video generation process in the model usage stage is similar to that described above; please refer to the relevant description of step 208 above for details.

[0155] Step 610: Based on the comparison results of the result video and the video tag, adjust the initial video generation model to obtain the trained video generation model.

[0156] The model training device can adjust the model parameters of the initial video generation model based on the comparison results of the result video and the video label, and return to the operation step of obtaining training data (i.e., step 602) until the training stop condition is met, and obtain the trained video generation model.

[0157] Specifically, a loss value can be calculated based on the resulting video and video tags. This loss value is then used for backpropagation to adjust the initial video generation model parameters. The process then returns to step 602 to continue training the initial video generation model using training samples and video tags until a training stopping condition is met, resulting in a fully trained video generation model. This training stopping condition can be that the loss value is less than a loss value threshold, the number of iterations reaches a threshold, or a combination of both thresholds.

[0158] The loss threshold is a critical value for the loss. If the loss value is greater than or equal to the threshold, it indicates that there is still a certain deviation between the prediction result of the initial video generation model and the actual result, and the model parameters of the initial video generation model still need to be adjusted. At this point, the training stopping condition has not been met. If the loss value is less than the threshold, it indicates that the prediction result of the initial video generation model is close enough to the actual result, and training can be stopped. At this point, the training stopping condition can be met. The specific value of the loss threshold is selected based on the actual situation.

[0159] In addition, the number of iterations threshold is set according to the actual situation. When the number of iterations reaches the threshold, it means that the initial video generation model has been trained enough times. At this time, the prediction result of the initial video generation model is very close to the real result, and training can be stopped.

[0160] The video generation model trained using the training method provided in the embodiments of this specification can, based on a given reference image and a reference audio clip, fully extract the texture and structural features of the reference image, accurately predict the actions of the target object affected by the audio, and quickly generate a video of the target object's actions based on the reference audio, achieving better audio-visual consistency. It realizes the generation of high-quality, realistic videos of target objects in a single-sample scenario.

[0161] Optionally, the video generation model includes a feature interaction module. In step 606 above, the initial video generation model is used to predict the reference action information of the sample object under the influence of audio features based on the interaction characteristics between visual and audio features. This includes: inputting facial features and audio features into the feature interaction module for feature interaction, and predicting the reference action information of the sample object under the influence of audio features. This process can refer to [the following text is incomplete and requires further context: "for..."] Figure 2 The details regarding feature interactions described in the introduction to the model usage process will not be repeated here.

[0162] Optionally, the training samples also include sample viewpoints. The training method for the video generation model provided in this specification further includes: determining the spatial location features of the portion of the sample object presented in the sample viewpoint based on facial features. Correspondingly, the video generation model also includes an action prediction module. In step 608 above, generating the result video corresponding to the sample object based on reference action information and sample audio includes: inputting the spatial location features, reference action information, and audio features of the sample audio into the action prediction module for processing to obtain the target action information of the sample object in the sample viewpoint; and generating the video corresponding to the sample object based on the target action information and sample audio. This process can be referred to for... Figure 2The details regarding spatial location features and target action information in the introduction to the model usage process will not be repeated here.

[0163] Optionally, the video generation model includes a feature extraction module with a feature pyramid network. Step 606 above, which extracts visual features of a reference object from a reference image, includes: inputting the sample image into the feature extraction module to extract features at different resolutions, thereby obtaining multi-scale visual features of the visual information of the sample object in the sample image; wherein, the multi-scale visual features include the visual features of the sample object at different resolutions.

[0164] Optionally, the training samples also include sample viewpoints, and the video generation model further includes a feature construction module. The training method for the video generation model provided in this embodiment further includes: inputting multi-scale features and sample viewpoint information into the feature construction module for processing to obtain the three-plane features of each three-dimensional spatial point corresponding to the sample object under the sample viewpoint; generating a video corresponding to the sample object based on reference action information and sample audio, including: combining the three-plane features, reference action information, and sample audio to generate the video corresponding to the sample object. This process can be referenced for... Figure 2 The details regarding the three-plane features described in the introduction to the model usage process will not be repeated here.

[0165] Optionally, the video generation model further includes an audio-visual consistency analysis module. The training method of the video generation model provided in the embodiments of this specification further includes: inputting the video into the audio-visual consistency analysis module for processing to obtain the audio-visual consistency analysis result of the video; correspondingly, the adjustment of the initial video generation model based on the comparison result of the result video and the video tag in the above step 610 includes: adjusting the initial video generation model based on the comparison result of the result video and the video tag, as well as the audio-visual consistency analysis result.

[0166] Figure 7 This is a comparison diagram of video generation effects provided in one embodiment of this specification. Figure 7 The image shown is an image frame from a video generated using the video generation method provided in the embodiments of this specification (last row), as well as image frames from videos generated using six other models in related technologies. The image on the left, magnified showing the mouth area, represents the image frame corresponding to the audio frame output by each model that pronounces "yi"; the image on the right, magnified showing the mouth area, represents the image frame corresponding to the audio frame output by each model that pronounces "an". Figure 7 The comparison shows that the video generation method provided in the embodiments of this specification produces a video with a high degree of consistency between the mouth shape of the person and the audio.

[0167] Figure 8This is a flowchart of another video generation method provided in an embodiment of this specification, applied to a task platform, specifically including the following steps 702 and 704.

[0168] Step 702: Receive a model request sent by the terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0169] It should be noted that the task platform is an online platform that provides model services, allowing developers, enterprises, or individual users to submit tasks, obtain resources, or implement specific functions through API calls, web interfaces, or other interactive methods. In the embodiments of this specification, the task platform is a cloud service platform capable of providing deep learning models, allowing external applications (such as e-commerce platforms, navigation applications, social applications, film and television production applications, and online education platforms) to access and use it. The task platform ensures efficient and accurate responses to requests from terminal devices by managing the training, updating, optimization, load balancing, scheduling, and inference of the models. The terminal device is the one requesting to obtain the video generation model.

[0170] In practice, a model request is a request sent by the terminal device to the task platform to obtain or use a specific model service. The model request may include at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters.

[0171] The target scenario refers to the specific environment or context in which the video generation model will be applied, describing the purpose and expected effects of the model service. The scenario identifier is the identification information used to identify the target scenario, assisting in quickly locating models related to that specific scenario, such as "audio-driven video generation," "video question answering," "video navigation," "video surveillance," "video quality evaluation," "video object localization," "video product localization and search," etc. The scenario input data can be the input data used to train the initial video model to adapt to the target scenario; for example, in a video question answering scenario, the scenario input data could be sample videos with answers. Additionally, model specification parameters are attribute specifications defining the model architecture, performance, or configuration, such as model model, model type, and model size. These model specification parameters are used to determine the model that best suits the user's needs.

[0172] For example, on the task platform client of a terminal device, a user selects the target scene identifier as "audio-driven video generation," selects the required model size as 32GB, uploads the scene input data of the target scene, and the terminal device generates a model request and sends it to the task platform. The task platform receives the model request sent by the terminal device, which includes the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters.

[0173] In the embodiments described in this specification, users can independently select at least one of the following on the terminal device: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters to trigger a model request. The task platform can receive the model request sent by the terminal device and provide reference information for selecting a video generation model based on at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters, thus making it applicable to a wider range of scenarios.

[0174] Step 704: Based on the model request, determine the corresponding video generation model from at least one model, wherein the video generation model is trained based on the above-described video generation model training method.

[0175] It should be noted that, based on the model request, the corresponding video generation model is determined from at least one model. One possible approach is to search for the corresponding video generation model from at least one model included in the model library based on the model request; another possible approach is to train the video generation model based on the model request; yet another possible approach is to construct the video generation model based on the model request, which is not limited here.

[0176] For example, one can first search for at least one pre-trained model from the model library based on the scene identifier of the target scene, then select the model of the corresponding size from the at least one model based on the model specification parameters, and then train the model of the corresponding size based on the scene input data of the target scene to obtain a model suitable for user needs.

[0177] The video generation model selected from at least one model is based on... Figure 6 The video generation model provided is trained using the training method described in this specification. Figure 6 The embodiments in this specification are based on the same inventive concept, and the specific methods are described in the above-mentioned model training content, which will not be repeated here.

[0178] In an optional implementation of this embodiment, the model request includes a scene identifier of the target scene; step 704, based on the model request, determines the corresponding video generation model from at least one model, including: based on the scene identifier of the target scene, searching for a video generation model suitable for the target scene from the model library, wherein the model library stores at least one model suitable for different scenes.

[0179] It should be noted that the model library is a database for storing and managing various pre-trained deep learning models. Multiple models adapted to different data processing scenarios cover different application scenarios and needs. The model library allows users to select the appropriate model according to their needs, or directly use the model to perform corresponding tasks through API calls.

[0180] Multiple models adapted to different scenarios are stored in the model library, each specifically designed for different scenarios. Each model is optimized for a specific application environment, and the video generation model is trained according to the aforementioned video generation model training method. For example, based on the scene identifier "audio-driven video generation" of the target scene, a video generation model suitable for the scenario of using audio to drive video generation of the target object can be found in the model library.

[0181] In the embodiments described in this specification, based on the requirements of the scenario, a video generation model suitable for the scenario is accurately found through scenario identification, so that the target processing results are more accurate and fit the scenario, thereby improving the user experience and video generation quality.

[0182] As an example, the task platform can provide models for various scenarios. For instance, in a video question-answering scenario, it can provide a corresponding video generation model based on model requests sent by the terminal device. Since the video generation model is selected from at least one model trained using the aforementioned training method, and has undergone precise joint training in both visual and auditory modalities, it can accurately understand visual and audio content at a fine-grained level and answer related questions. Alternatively, it can identify the source of sound, determine the location of objects described by audio signals in the video, and answer object-level questions. This capability is of great significance for improving perception capabilities in navigation, monitoring, and interactive technology fields.

[0183] In addition, it can also be applied to the e-commerce field. In the "video product positioning and search" scenario, based on the selected video generation model, a video can be accurately understood or a description of the video can be obtained. The sound in the video can be used to accurately search for and locate products, thereby improving user experience and search efficiency.

[0184] In an optional implementation of this embodiment, the model request includes scene input data of the target scene; in step 704 above, determining the corresponding video generation model from at least one model based on the model request includes: determining an initial video generation model adapted to the target scene from at least one model; and training the initial video generation model based on the scene input data of the target scene to obtain the video generation model.

[0185] In practice, the model request may include scene input data for the target scene, and the video generation model is a video generation model suitable for the target scene. For example, a general video generation model is a basic video generation model trained to adapt to different scenes, but not optimized for any specific scene. For instance, based on input data for an audio-driven video generation scene, the general video generation model can be trained to obtain a video generation model adapted to that scene.

[0186] In the embodiments described in this specification, based on scenario requirements, a general video generation model is further trained using scenario input data to obtain a video generation model adapted to the scenario, making the target processing results more accurate and relevant to the scenario, thereby improving the user experience and the processing quality of video tasks.

[0187] In an optional implementation of this embodiment, the model request includes model specification parameters; in step 704 above, determining the corresponding video generation model from at least one model based on the model request includes: searching for the corresponding video generation model in the model library based on the model specification parameters, wherein the model library stores multiple video generation models with different model specification parameters.

[0188] The model specification parameter can be the model size, such as searching for a video generation model of the corresponding size from the model library based on a model size of 32GB. In the embodiments of this specification, the corresponding video generation model is accurately found based on the model specification requirements and the model specification parameter, ensuring the efficient and stable operation of the video generation model and improving the user experience.

[0189] In an optional implementation of this embodiment, after determining the corresponding video generation model from at least one model based on a model request, the method further includes: deploying the video generation model and constructing a video generation interface based on the video generation model, so that the terminal device can schedule the video corresponding to the target object of the video generation model. The video generation interface is an interactive programming interface for the terminal device to schedule the video generation model, typically provided in the form of an API. Through the video generation interface, users can input task data for the video generation task, such as reference images and reference audio, and effectively control the model's output (such as the generated video content).

[0190] In practice, one possible approach to deploying the video generation model is to deploy it on a distributed system of the task platform. For example, the video generation model can be deployed on the distributed system of the task platform, and a video generation interface can be built based on the model and provided to the terminal device, enabling the terminal device to schedule the video generation model to perform the task of generating the video corresponding to the target object.

[0191] In the embodiments of this specification, efficient terminal invocation is achieved, video task processing is optimized, and the processing quality and response speed of video tasks are improved. Applying the video generation method provided in the embodiments of this specification, video generation models are obtained to meet user needs, realizing personalized model services and providing users with an efficient, flexible, and easy-to-use model service method, thus enhancing the user experience.

[0192] Corresponding to the above method embodiments, this specification also provides task platform embodiments. Figure 9 This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification. Figure 9 As shown, the task platform 800 includes: a request interface 802 and a response unit 804;

[0193] Request interface 802 is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0194] The response unit 804 is used to determine the corresponding video generation model from at least one model based on the model request, wherein the video generation model is trained based on the above-described video generation model training method.

[0195] Optionally, the task platform also includes a video generation interface, which is built based on a video generation model; and a video processing interface, which is used for terminal devices to schedule and generate videos corresponding to the target objects.

[0196] Optionally, the model request includes a scene identifier of the target scene; the response unit 804 is further configured to: search for a video generation model adapted to the target scene from the model library based on the scene identifier of the target scene, wherein the model library stores at least one model adapted to different scenes.

[0197] Optionally, the model request includes scene input data of the target scene; the response unit 804 is further configured to: determine an initial video generation model adapted to the target scene from at least one model; and train the initial video generation model based on the scene input data of the target scene to obtain a video generation model.

[0198] Optionally, the model request includes model specification parameters; the response unit 804 is further configured to: search for the corresponding video generation model in the model library based on the model specification parameters, wherein the model library stores multiple video generation models with different model specification parameters.

[0199] Optionally, the task platform also includes a deployment module, configured to: deploy a video generation model and, based on the video generation model, build a video processing interface so that the terminal device can schedule the video generation model to generate a video corresponding to the target object.

[0200] In the embodiments described in this specification, the task platform adapts to user needs to obtain video generation models, realizes personalized model services, provides users with an efficient, flexible and easy-to-use model service platform, and improves user experience.

[0201] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the video generation method described above belong to the same concept. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution based on the video generation method described above.

[0202] Corresponding to the above method embodiments, this specification also provides embodiments of a video generation apparatus. Figure 10 This is a schematic diagram of the structure of a video generation device provided in one embodiment of this specification. Figure 10 As shown, the device includes:

[0203] The first acquisition module 902 acquires a reference image and a reference audio, wherein the reference image contains at least visual information of the reference object;

[0204] The feature extraction module 904 extracts visual features of the reference object from the visual information in the reference image, and extracts audio features from the reference audio.

[0205] The prediction module 906 predicts the reference action information of the target object under the influence of the audio features based on the interaction characteristics between visual features and audio features, wherein the target object is obtained based on the reference object;

[0206] The video generation module 908 generates a video corresponding to the target object based on reference motion information and reference audio.

[0207] Optionally, the prediction module 906 is used to: perform correlation analysis on visual features and audio features to obtain correlation information between visual features and audio features; wherein, visual features include features of multiple visual positions on the target object, and correlation information characterizes the correlation between the features of each visual position and the audio features; based on the correlation information, predict the reference action information of the target object under the influence of audio features.

[0208] Optionally, the video generation apparatus further includes:

[0209] The second acquisition module is used to acquire information from the target's perspective.

[0210] The first determining module is used to determine the spatial location features of the part of the target object presented from the target viewpoint based on visual features and target viewpoint information;

[0211] The video generation module 908 includes:

[0212] The adjustment submodule is used to obtain the target action information of the target object from the target's perspective based on reference action information, audio features, and spatial location features.

[0213] The video generation submodule is used to generate a video corresponding to the target object based on the target action information and reference audio.

[0214] Optionally, the adjustment submodule is used to: predict object deformation based on spatial location features, audio features and reference action information to obtain feature offset information of spatial location features; wherein, the feature offset information represents the visual deformation features of the target object under the action of the target viewpoint and audio features; and combine the spatial location features and feature offset information to obtain the target action information of the target object under the target viewpoint.

[0215] Optionally, the feature extraction module 904 is used to: extract visual features of the reference object at different resolutions based on the visual information in the reference image, thereby obtaining multi-scale visual features; wherein, the multi-scale visual features include the visual features of the reference object at different resolutions.

[0216] Optionally, the prediction module 906 is used to: reshape multi-scale visual features into visual features of the same scale and stitch them together along the feature channel dimension to obtain auxiliary visual features; flatten the auxiliary visual features according to the target feature size and aggregate the flattened auxiliary visual features to obtain aggregated visual features; and perform feature interaction between the aggregated visual features and audio features to predict the reference action information of the target object under the influence of the audio features.

[0217] Optionally, the video generation apparatus further includes:

[0218] The third acquisition module is used to acquire information from the target's perspective.

[0219] The reshaping module is used to reshape multi-scale visual features into three-plane features;

[0220] The second determining module is used to combine the information from the target viewpoint and the three-plane features to determine the three-plane features of each three-dimensional spatial point in the part of the target object presented from the target viewpoint, thereby obtaining the spatial position features of the part of the target object presented from the target viewpoint; wherein, the spatial position features include the three-plane features of each three-dimensional spatial point;

[0221] The video generation module 908 is used to generate a video corresponding to the target object by combining spatial location features, reference motion information and reference audio.

[0222] Optionally, the video generation module 908 is used to: perform neural rendering based on spatial location features and reference action information, and combine information other than the reference object in the reference image to obtain an image frame sequence; combine the image frame sequence and reference audio to generate a video corresponding to the target object.

[0223] Optionally, the video generation module 908 is used to: generate an initial image frame sequence based on reference motion information; perform super-resolution processing on each image frame in the initial image frame sequence to obtain a target image frame sequence; and combine the target image frame sequence with reference audio to generate a video corresponding to the target object.

[0224] Optionally, the video generation apparatus further includes:

[0225] The fourth acquisition module is used to acquire multiple image frames and multiple audio frames that correspond one-to-one in the video after generating the video corresponding to the target object based on the reference action information and reference audio.

[0226] The third determination module is used to extract features from multiple image frames and multiple audio frames, and determine the feature similarity between each image frame and its corresponding audio frame, thereby obtaining the audio-visual consistency analysis results of the video.

[0227] Optionally, the video generation device uses a video generation model to generate video. The video generation model includes a feature extraction module, an action prediction module, and a video generation module. The feature extraction module 904 is used to: extract features from the visual information and reference audio in the reference image, respectively, to obtain the visual features and audio features of the reference object.

[0228] Prediction module 906 is used to: process visual and audio features using the action prediction module to predict reference action information of the target object under the influence of audio features;

[0229] The video generation module 908 is used to process the reference motion information and reference audio to generate a video corresponding to the target object.

[0230] Optionally, the video generation apparatus further includes:

[0231] The adjustment module is used to adjust at least one of the feature extraction module, action prediction module, and video generation module after generating a video corresponding to the target object based on reference action information and reference audio, upon receiving feedback information regarding issues with the video.

[0232] Corresponding to the above method embodiments, this specification also provides embodiments of a training device for a video generation model. Figure 11 This is a schematic diagram of the structure of a training device for a video generation model provided in one embodiment of this specification. Figure 11 As shown, the device includes:

[0233] The acquisition module 1002 is used to acquire training samples and video labels, wherein the training samples include sample images and sample audio, and the sample images contain at least visual information of the sample objects;

[0234] The extraction module 1004 is used to extract visual features from the visual information in the sample image and extract audio features from the reference audio using the initial video generation model.

[0235] Prediction module 1006 is used to predict the reference action information of the sample object under the influence of audio features by utilizing the initial video generation model and based on the interaction characteristics between visual features and audio features.

[0236] The video generation module 1008 is used to generate the result video corresponding to the target object based on the reference action information and sample audio using the initial video generation model.

[0237] The adjustment module 1010 is used to adjust the initial video generation model based on the comparison results of the result video and the video tags, so as to obtain the trained video generation model.

[0238] Optionally, the video generation model includes a feature interaction module, and the prediction module 1006 is used to: input facial features and audio features into the feature interaction module for feature interaction, and predict the reference action information of the target object under the influence of the audio features.

[0239] Optionally, the training samples also include sample perspectives, and the training device further includes:

[0240] The determination module is used to determine the spatial location features of the target object as presented in the sample view based on facial features;

[0241] The video generation module 1008 is used to: input the spatial location features, reference action information and audio features of the sample audio into the action prediction module for processing, to obtain the target action information of the target object from the sample viewpoint; and generate the video corresponding to the target object based on the target action information and the sample audio.

[0242] Optionally, the video generation model includes a feature extraction module with a feature pyramid network. The extraction module 1004 is used to: input the sample image into the feature extraction module to extract features at different resolutions, thereby obtaining multi-scale visual features for the visual information of the sample object in the sample image; wherein, the multi-scale visual features include the visual features of the sample object at the different resolutions.

[0243] Optionally, the training samples also include sample viewpoints, the video generation model also includes a feature construction module, and the training device also includes:

[0244] The first processing module is used to input multi-scale features and sample view information into the feature construction module for processing, so as to obtain the three-plane features of each three-dimensional spatial point of the target object under the sample view.

[0245] The video generation module 1008 is used to generate a video corresponding to the target object by combining the three-plane features, reference motion information and sample audio.

[0246] Optionally, the video generation model also includes an audio-visual consistency analysis module, and the training device also includes:

[0247] The second processing module is used to process the video input audio-visual consistency analysis module to obtain the audio-visual consistency analysis results of the video.

[0248] The adjustment module 1010 is used to adjust the initial video generation model based on the comparison results of the resulting video and video tags, as well as the audio-visual consistency analysis results.

[0249] The above is an illustrative scheme of a training device for a video generation model according to this embodiment. It should be noted that the technical solution of this training device for a video generation model and the technical solution of the above-described training method for a video generation model belong to the same concept. For details not described in detail in the technical solution of the training device for a video generation model, please refer to the description of the technical solution of the above-described training method for a video generation model.

[0250] Corresponding to the above method embodiments, this specification also provides embodiments of a video generation apparatus. Figure 12 This is a schematic diagram of another video generation device provided in one embodiment of this specification. (See diagram below.) Figure 12 As shown, the device includes:

[0251] The receiving module 1102 is used to receive a video generation request uploaded by the front end, wherein the video generation request includes description information of the target object and reference audio;

[0252] The acquisition module 1104 is used to acquire a target image containing at least the facial region of the target object based on the description information;

[0253] The extraction module 1106 is used to extract facial features from the facial region of the target object in the target image and to extract audio features from the reference audio.

[0254] The prediction module 1108 is used to predict the reference facial action information of the target object under the influence of the audio features based on the interaction characteristics of facial features and audio features, wherein the target object is obtained based on the reference object;

[0255] The video generation module 1110 is used to generate a video corresponding to the target object based on reference facial motion information and reference audio.

[0256] The sending module 1112 is used to send video to the front end.

[0257] Figure 13 This is a structural block diagram of a computing device according to one embodiment of this specification. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0258] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0259] In one embodiment of this specification, the aforementioned components of the computing device 1200 and Figure 13Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 13 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0260] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1200 can also be a mobile or stationary server.

[0261] The memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute the computer programs / instructions stored in the memory 1210. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned video generation method and the training method of the video generation model.

[0262] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0263] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0264] Computer programs / instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0265] The technical solutions of the aforementioned computing devices, computer-readable storage media, and computer program products belong to the same concept as the aforementioned video generation method and video generation model training method. For details not described in detail in the technical solutions of the computer program products, please refer to the description of the technical solutions of the aforementioned methods.

[0266] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0267] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0268] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A video generation method, comprising: Acquire a reference image and a reference audio, wherein the reference image contains at least visual information of a reference object; Visual features of the reference object are extracted from the visual information in the reference image, and audio features are extracted from the reference audio. Based on the interaction characteristics between the visual features and the audio features, the reference action information of the target object under the influence of the audio features is predicted, wherein the target object is obtained based on the reference object; Based on the reference motion information and the reference audio, a video corresponding to the target object is generated.

2. The method according to claim 1, wherein predicting reference action information of the target object under the influence of the audio features based on the interaction characteristics between the visual features and the audio features comprises: A correlation analysis is performed on the visual features and the audio features to obtain correlation information between the visual features and the audio features; wherein, the visual features include features at multiple visual locations on the target object, and the correlation information characterizes the degree of correlation between the features at each visual location and the audio features; Based on the correlation information, the reference action information of the target object under the influence of the audio features is predicted.

3. The method according to claim 1 or 2, further comprising: Obtain information from the target's perspective; Based on the visual features and the target viewpoint information, determine the spatial position features of the portion of the target object presented under the target viewpoint; The step of generating a video corresponding to the target object based on the reference motion information and the reference audio includes: Based on the reference motion information, the audio features, and the spatial location features, the target motion information of the target object under the target's viewpoint is obtained; Based on the target action information and the reference audio, a video corresponding to the target object is generated.

4. The method according to claim 3, wherein obtaining the target action information of the target object from the target's perspective based on the reference action information, the audio features, and the spatial location features includes: Based on the spatial location features, the audio features, and the reference motion information, object deformation prediction is performed to obtain feature offset information of the spatial location features; wherein, the feature offset information characterizes the visual deformation features of the target object under the action of the target viewpoint and the audio features; By combining the spatial location features and the feature offset information, the target action information of the target object under the target's viewpoint is obtained.

5. The method according to claim 1 or 2, wherein extracting the visual features of the reference object from the visual information in the reference image comprises: Based on the visual information in the reference image, visual features of the reference object are extracted at different resolutions to obtain multi-scale visual features; wherein, the multi-scale visual features include the visual features of the reference object at the different resolutions.

6. The method according to claim 5, wherein predicting reference action information of the target object under the influence of the audio features based on the interaction characteristics between the visual features and the audio features comprises: The multi-scale visual features are reshaped into visual features of the same scale and spliced ​​along the feature channel dimension to obtain auxiliary visual features. The auxiliary visual features are flattened according to the target feature size, and the flattened auxiliary visual features are aggregated to obtain aggregated visual features; The aggregated visual features and the audio features are interacted to predict the reference action information of the target object under the influence of the audio features.

7. The method according to claim 5, further comprising: Obtain information from the target's perspective; The multi-scale visual features are reshaped into three-plane features; By combining the information from the target viewpoint and the three-plane features, the three-plane features of each three-dimensional spatial point in the portion of the target object presented from the target viewpoint are determined, thereby obtaining the spatial position features of the portion of the target object presented from the target viewpoint; wherein, the spatial position features include the three-plane features of each three-dimensional spatial point; The step of generating a video corresponding to the target object based on the reference motion information and the reference audio includes: By combining the spatial location features, the reference motion information, and the reference audio, a video corresponding to the target object is generated.

8. The method according to claim 7, wherein generating a video corresponding to the target object by combining the spatial location features, the reference motion information, and the reference audio comprises: Neural rendering is performed based on the spatial location features and the reference action information, and information other than the reference object in the reference image is combined to obtain an image frame sequence; The image frame sequence and the reference audio are combined to generate a video corresponding to the target object.

9. The method according to claim 1 or 2, wherein generating the video corresponding to the target object based on the reference motion information and the reference audio comprises: Based on the reference action information, an initial image frame sequence is generated; The initial image frame sequence is subjected to super-resolution processing to obtain the target image frame sequence. The target image frame sequence is combined with the reference audio to generate a video corresponding to the target object.

10. The method according to any one of claims 1, 2, 4, or 6 to 8, wherein after generating the video corresponding to the target object based on the reference motion information and the reference audio, the method further comprises: Obtain multiple image frames and multiple audio frames that correspond one-to-one in the video; Features of the multiple image frames and multiple audio frames are extracted, and the feature similarity between each image frame and its corresponding audio frame is determined to obtain the audio-visual consistency analysis results of the video.

11. The method according to claim 1, executed using a video generation model, wherein the video generation model includes a feature extraction module, an action prediction module, and a video generation module; The step of extracting visual features of the reference object from the visual information in the reference image and extracting audio features from the reference audio includes: Using the feature extraction module, features are extracted from the visual information in the reference image and the reference audio, respectively, to obtain the visual features and audio features of the reference object; The method of predicting reference action information of the target object under the influence of the audio features based on the interaction characteristics between the visual features and the audio features includes: The motion prediction module processes the visual features and the audio features to predict reference motion information of the target object under the influence of the audio features; The step of generating a video corresponding to the target object based on the reference motion information and the reference audio includes: The video generation module processes the reference motion information and the reference audio to generate a video corresponding to the target object.

12. The method according to claim 11, after generating the video corresponding to the target object based on the reference motion information and the reference audio, further comprising: Upon receiving feedback regarding issues with the video, at least one of the following modules is adjusted: the feature extraction module, the action prediction module, and the video generation module.

13. A video generation method, comprising: Receive a video generation request uploaded from the front end, wherein the video generation request includes description information of the target object and reference audio; Based on the description information, obtain a target image that at least contains the facial region of the target object; Facial features are extracted from the facial region of the target object in the target image, and audio features are extracted from the reference audio. Based on the interaction characteristics between the facial features and the audio features, reference facial motion information of the target object is predicted under the influence of the audio features, wherein the target object is obtained based on the reference object; Based on the reference facial motion information and the reference audio, a video corresponding to the target object is generated; The video is sent to the front end.

14. A training method for a video generation model, comprising: Obtain training samples and video labels, wherein the training samples include sample images and sample audio, and the sample images contain at least visual information of the sample objects; Using an initial video generation model, visual features are extracted from the visual information in the sample image, and audio features are extracted from the reference audio. Using the initial video generation model, and based on the interaction characteristics between the visual features and the audio features, the reference action information of the sample object under the influence of the audio features is predicted. Using the initial video generation model, based on the reference motion information and the sample audio, a result video corresponding to the target object is generated; Based on the comparison results between the resulting video and the video tags, the initial video generation model is adjusted to obtain the trained video generation model.

15. The method according to claim 14, wherein the video generation model includes a feature interaction module, and the step of predicting reference action information of the sample object under the influence of the audio features based on the interaction characteristics between the visual features and the audio features using the initial video generation model includes: The facial features and the audio features are input into the feature interaction module for feature interaction, and the reference action information of the target object under the influence of the audio features is predicted.

16. The method of claim 15, wherein the training samples further include a sample perspective, and the method further includes: Based on the facial features, determine the spatial location features of the portion of the target object presented in the sample view. The video generation model further includes an action prediction module, which generates a result video corresponding to the target object based on the reference action information and the sample audio, including: The spatial location features, the reference action information, and the audio features of the sample audio are input into the action prediction module for processing to obtain the target action information of the target object under the sample viewpoint; Based on the target action information and the sample audio, a video corresponding to the target object is generated.

17. The method according to any one of claims 14 to 16, wherein the video generation model includes a feature extraction module having a feature pyramid network, and the step of extracting visual features of the reference object from the reference image includes: The sample image is input into the feature extraction module for feature extraction at different resolutions to obtain multi-scale visual features of the visual information of the sample object in the sample image; wherein, the multi-scale visual features include the visual features of the sample object at the different resolutions.

18. The method according to claim 17, wherein the training samples further include sample viewpoints, the video generation model further includes a feature construction module, and the method further includes: The multi-scale features and the information from the sample perspective are input into the feature construction module for processing to obtain the three-plane features of each three-dimensional spatial point of the sample object under the sample perspective. The step of generating a video corresponding to the sample object based on the reference motion information and the sample audio includes: By combining the three-plane features, the reference motion information, and the sample audio, a video corresponding to the sample object is generated.

19. The method according to any one of claims 14 to 16, wherein the video generation model further includes an audio-visual consistency analysis module, and the method further includes: The video is input into the audio-visual consistency analysis module for processing to obtain the audio-visual consistency analysis result of the video; The step of adjusting the initial video generation model based on the comparison results of the resulting video and the video tags includes: Based on the comparison results of the resulting video and the video tags, as well as the audio-visual consistency analysis results, the initial video generation model is adjusted.

20. A video generation method, applied to a task platform, comprising: The device receives a model request sent by a terminal device, wherein the model request includes at least one of the following: a scene identifier of the target scene, scene input data of the target scene, and model specification parameters. Based on the model request, a corresponding video generation model is determined from at least one model, wherein the video generation model is trained based on the training method of the video generation model as described in any one of claims 14 to 19.

21. The method of claim 20, wherein the model request includes a scene identifier of the target scene; and the video of the target object speaking the target audio from the target perspective includes: Based on the scene identifier of the target scene, a video generation model suitable for the target scene is searched from the model library, wherein the model library stores at least one model suitable for different scenes; The model request includes scene input data of the target scene; determining the corresponding video generation model from at least one model based on the model request includes: From at least one model, determine an initial video generation model adapted to the target scene; Based on the scene input data of the target scene, the initial video generation model is trained to obtain a video generation model; The model request includes model specification parameters; determining the corresponding video generation model from at least one model based on the model request includes: Based on the model specification parameters, the corresponding video generation model is searched from the model library, which stores multiple video generation models with different model specification parameters.

22. The method of claim 20 or 21, further comprising, after determining the corresponding video generation model from at least one model based on the model request: The video generation model is deployed, and a video generation interface is constructed based on the video generation model, so that the terminal device can schedule the video generation model through the video generation interface to generate a video corresponding to the target object.

23. A task platform, comprising a request interface and a response unit; The request interface is used to receive model requests sent by the terminal device, wherein... The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a corresponding video generation model from at least one model based on the model request, wherein the video generation model is trained based on the training method of the video generation model as described in any one of claims 14 to 19.

24. The task platform according to claim 23 further includes a video generation interface, wherein the video generation interface is constructed based on the video generation model; The video generation interface is used for the terminal device to schedule and generate a video corresponding to the target object.

25. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein when the computer programs / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 19.

26. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 19.

27. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 19.

Images