Video generation method, apparatus, device, and storage medium

Abstract

According to embodiments of this disclosure, a video generation method, apparatus, device, and storage medium are provided. The method includes: acquiring a video generation request associated with a reference image, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one object among the plurality of objects; and providing a first control signal and image features corresponding to the reference image to a video generation model to generate video content, wherein the first control signal indicates at least positional information of at least one object in the reference image, and the video content includes motion content associated with at least one object, the motion content corresponding to audio content determined based on the video generation request. In this manner, embodiments of this disclosure can quickly generate video content and ensure the correlation and accuracy between the motion content in the generated video content and the selected object.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to video generation methods, apparatus, devices, and computer-readable storage media. Background Technology

[0002] With the development of computer technology, some machine learning techniques allow users to drive images through control signals, thereby generating dynamic video content. For example, users can input audio signals to drive digital humans to perform actions corresponding to the audio signals. Summary of the Invention

[0003] In a first aspect of this disclosure, a video generation method is provided. The method includes: acquiring a video generation request associated with a reference image, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one object among the plurality of objects; and providing a first control signal and image features corresponding to the reference image to a video generation model to generate video content, wherein the first control signal indicates at least positional information of at least one object in the reference image, and the video content includes motion content associated with the at least one object, the motion content corresponding to audio content determined based on the video generation request.

[0004] In a second aspect of this disclosure, a video generation apparatus is provided. The apparatus includes a request acquisition module and a video generation module. The request acquisition module is configured to acquire a video generation request associated with a reference image, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one object among the plurality of objects; and the video generation module is configured to provide a first control signal and image features corresponding to the reference image to a video generation model to generate video content, wherein the first control signal indicates at least positional information of at least one object in the reference image, and the video content includes motion content associated with at least one object, the motion content corresponding to audio content determined based on the video generation request.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 An example structure of a video generation model according to some embodiments of the present disclosure is shown;

[0012] Figures 3A to 3C A schematic diagram of data preprocessing according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A flowchart illustrating an example process for video generation according to some embodiments of this disclosure is shown;

[0014] Figure 5 A schematic structural block diagram of an example apparatus for video generation according to some embodiments of the present disclosure is shown; and

[0015] Figure 6 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0021] Audio-driven portrait video generation technology aims to use spoken audio to drive portrait images. While there has been continuous progress in this field in recent years, related technologies often focus on single-person scenes. Typically, these solutions require preprocessing to determine the number of people in the input image before model processing, ensuring that the input image contains only a single person. The model then uses the input single-person image and audio to generate a video of the person speaking.

[0022] However, in practical use, users face an unavoidable need: the input image contains multiple people, and it's necessary to drive a specific person within it, rather than having multiple people in the image speak simultaneously. For models in multi-person scenarios, in addition to designing the model architecture, it's also necessary to collect a large amount of target data; both tasks are extremely time-consuming and labor-intensive.

[0023] Embodiments of this disclosure propose a video generation scheme. The scheme includes: acquiring a video generation request associated with a reference image, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one object among the plurality of objects; and providing a first control signal and image features corresponding to the reference image to a video generation model to generate video content, wherein the first control signal indicates at least the positional information of at least one object in the reference image, and the video content includes motion content associated with at least one object, the motion content corresponding to audio content determined based on the video generation request.

[0024] The embodiments of this disclosure can directly generate video content based on the selection of at least one object in a reference image and image features, combined with audio content. This video content includes motion content associated with the at least one object. In this way, the embodiments of this disclosure can distinguish the effective region of occurrence corresponding to a speaker in the reference image, thereby generating video content featuring a specified speaker, and ensuring the correlation and accuracy between the motion content in the generated video content and the selected object, thus guaranteeing the efficiency and quality of video content generation.

[0025] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.

[0028] like Figure 1 As shown, the electronic device 110 can acquire control signals 130 and reference images 120. The control signals 130 may include control signals of various modalities, such as text content, audio content, and video content. The reference image 120 includes multiple objects of a preset type. As an example, the preset type may include object types that may generate dynamic content, such as people or animals, for example, two or more people or animals.

[0029] As an example, Figure 1The reference image 120 shown includes two objects: designated object 125 and another object. In other scenarios or implementations, the reference image 120 may also include three or more objects, which is not limited here.

[0030] In some embodiments, the electronic device 110 can identify multiple objects in the reference image 120 and present multiple indicator elements (e.g., identification boxes) corresponding to the multiple objects. Then, based on the selection of the indicator element corresponding to the specified object 125 among the multiple indicator elements, a video generation request associated with the reference image 120 is obtained.

[0031] As will be described in detail below, electronic device 110 can provide model 140 with a reference image 120 having multiple objects and control signals 130 of multiple modes to generate video content 150. Specifically, electronic device 110 can determine image features corresponding to the reference image 120 based on the reference image 120, including object features corresponding to a specified object 125, and may also include image features corresponding to other objects besides the specified object 125.

[0032] As an example, video content 150 may include motion content associated with a designated object 125 in reference image 120, such as the designated object 155 speaking. As an example, this motion content corresponds to a received control signal 130 in multiple modalities. The designated object 155 speaking in video content 150 corresponds to the designated object 125 among multiple objects in reference image 120.

[0033] As an example, control signal 130 may include reference video content and audio content. The audio content may be determined based on multimedia content such as audio or video content uploaded by the user, or it may be determined based on text content uploaded by the user.

[0034] As an example, motion content in video content 150 may include head movements such as mouth and / or face generated based on speech content, and may also include body movements generated based on skeletal information from reference video content.

[0035] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0036] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0037] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and electronic device 110 can achieve signaling interaction through the communication connection between them.

[0038] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0039] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0040] Example Model

[0041] Figure 2 An example model 200 for generating video according to some embodiments of this disclosure is shown. Reference will be made below. Figure 2This describes the specific process by which Model 200 generates video content.

[0042] refer to Figure 2 As shown, model 200 may include multiple processing layers 210. The output of the previous processing layer can be used as the input of the next processing layer, and the output of the last processing layer is used as the output of model 200, i.e., the generated video content 202.

[0043] In some embodiments, each processing layer 210 may include a first attention unit 211, a second attention unit 212, and a merging unit 213. The first attention unit 211 may be used to generate first video features, such as video features corresponding to a speaker; the second attention unit 212 may be used to generate second video features, such as video features corresponding to a listener (not a speaker); the merging unit 213 is used to merge the first and second video features to generate input video features for the next processing layer or generated video content 202.

[0044] In some embodiments, the first attention unit 211 is configured to: determine the first video feature corresponding to a specified speaker based on a first control signal, a reference image, audio content and preset first prompt information.

[0045] As an example, the first control signal can at least indicate the position information of a designated speaker (e.g., designated object 125) in the reference image 120, for example... Figure 2 Speaker marker 240. Encoding the reference image 120 yields a reference token 211b corresponding to the specified object 125. Encoding the audio content yields a spoken-audio token 220 corresponding to the specified object 125. Based on preset first prompt information, a spoken-text token 211c corresponding to the specified object 125 is obtained.

[0046] like Figure 2 As shown, the first attention unit 211 may include a first attention module 211-1 and a second attention module 211-2. As an example, the first attention module 211-1 may have three input channels, and the output of the first attention module 211-1 corresponds to at least a portion of the input of the second attention module 211-2. The first input channel of the first attention module 211-1 may correspond to input video features.

[0047] As an example, the first attention module 211-1 can be a multimodal attention module, and the second attention module 211-2 can be a cross-attention module.

[0048] In some embodiments, model 200 can be a generative model based on a diffusion model. Accordingly, a video token 211a can be obtained by encoding the input video 201 to serve as input information for the first input channel of the first attention module 211-1. The video token 211a corresponds to the dynamic features of the speaker in the input video 201, such as head movement features like the mouth or face, or limb movement features determined based on skeletons.

[0049] In some embodiments, preset noise may be superimposed on the video token 211a and then used as input information for the first input channel of the first attention module 211-1.

[0050] In some embodiments, the input video 201 may include other video content associated with the video content to be generated. As an example, the input video 201 may be a reference video uploaded by a user, a video or video feature output by the previous processing layer 210, or a portion of a video frame from a temporally continuous previous video segment.

[0051] In some embodiments, the second input channel of the first attention module 211-2 may correspond to input image features.

[0052] As an example, the input image features may include a reference token 211b determined by encoding the reference image 120. For example, the reference token 211b may correspond to a specified object 125 in the reference image 120.

[0053] In some embodiments, the third input channel of the first attention module 211-2 may correspond to input text features.

[0054] As an example, where the control signal includes reference text content (e.g., a prompt word), the input text features may include a speak-text token 211c determined by encoding the reference text content. For example, “This person is speaking.”

[0055] In some embodiments, depending on the modality of the control signal, the first attention module 211-1 can perform an attention-based update process on at least two types of input features received via multiple input channels. For example, when the control signal includes input video content, the first attention module 211-1 can update the input video features (e.g., video token 211a) and the input image features (e.g., reference token 211b) based on a multimodal attention mechanism.

[0056] refer to Figure 2As shown, the second attention module 211-2 includes multiple input channels. As an example, in addition to some input channels corresponding to the output channels of the first attention module 211-1, the second attention module 211-2 may also include a fourth input channel and a fifth input channel.

[0057] In some embodiments, the fourth input channel may correspond to input audio features, and the fifth input channel may correspond to input image features.

[0058] In some embodiments, the fourth input channel may correspond to a speak-audio token 220 obtained by encoding the input audio content. The speak-audio token 220 corresponds to an audio feature in the input audio content that specifies the speaker.

[0059] As an example, when the control signal includes input audio content, audio features of the input audio content can be determined. As an example, the audio features may include Mel-spectral features of the audio content. Further, an encoding unit can be used to process the audio features to obtain the corresponding spoken-audio token 220. As an example, the encoding unit may include, for example, a multilayer perceptron (MLP) or other suitable model.

[0060] In some embodiments, the fifth input channel may correspond to a speaker tag 240 obtained by recognizing and identifying based on an input image. As an example, the speaker tag 240 may be an indicator element for a target speaking object among multiple objects in the input image, such as a bounding box corresponding to the location or region of the target speaking object.

[0061] In some embodiments, the second attention module 211-2 may update the input video features based on a cross-attention mechanism and based on the spoken-audio token 220 and the speaker tag 240 to obtain the first video features corresponding to the speaker.

[0062] In some embodiments, the second attention unit 212 is configured to: determine second video features corresponding to objects other than the designated speaker based on the second control signal, the reference image and the preset second cue information.

[0063] As an example, the second control signal can at least indicate the positional information of objects other than the designated speaker (e.g., designated object 125) in the reference image 120, for example... Figure 2 The listener marker 250 is used. Encoding the reference image 120 can also yield reference tokens 212b corresponding to objects other than the specified object 125. Based on preset second prompt information, listen-text tokens 212c corresponding to objects other than the specified object 125 can be obtained.

[0064] Continue to refer to Figure 2 As shown, the second attention unit 212 may include a third attention module 212-1 and a fourth attention module 212-2. As an example, the third attention module 212-1 may have three input channels, and the output of the third attention module 212-1 corresponds to at least a portion of the input of the fourth attention module 212-2. Specifically, the first input channel of the third attention module 212-1 may correspond to input video features.

[0065] As an example, the third attention module 212-1 can be a multimodal attention module, and the fourth attention module 212-2 can be a cross-attention module.

[0066] In some embodiments, a video token 212a can be obtained by encoding the input video 201 to serve as input information for the first input channel of the third attention module 212-1. As an example, the video token 212a may correspond to video features in the input video 201 that do not belong to the speaker.

[0067] In some embodiments, preset noise may be superimposed on the video token 212a and then used as input information for the first input channel of the third attention module 211-1.

[0068] In some embodiments, the input video 201 may include other video content associated with the video content to be generated. As an example, the input video 201 may be a reference video uploaded by a user, a video or video feature output by the previous processing layer 210, or a portion of a video frame from a temporally continuous previous video segment.

[0069] In some embodiments, the second input channel of the third attention module 212-2 may correspond to input image features.

[0070] As an example, the input image features may include a reference token 212b determined by encoding a reference image. For instance, the reference token 212b may correspond to an object other than the specified object 125 in the reference image 120.

[0071] In some embodiments, the third input channel of the third attention module 212-2 may correspond to input text features.

[0072] As an example, where the control signal includes reference text content (e.g., a prompt word), the input text features may include a listen-text token 212c determined by encoding the reference text content. For example, “This person is listening.”

[0073] In some embodiments, depending on the modality of the control signal, the third attention module 212-1 can perform an attention-based update process on at least two types of input features received via multiple input channels. For example, when the control signal includes input video content, the third attention module 212-1 can update the input video features (e.g., video token 212a) and the input image features (e.g., reference token 212b) based on a multimodal attention mechanism.

[0074] refer to Figure 2 As shown, the fourth attention module 212-2 includes multiple input channels. As an example, in addition to some input channels corresponding to the output channel of the third attention module 212-1, the fourth attention module 212-2 may also include a fourth input channel and a fifth input channel.

[0075] In some embodiments, the fourth input channel corresponding to the fourth attention module 212-2 may correspond to input audio features, and the fifth input channel may correspond to input image features.

[0076] In some embodiments, the fourth input channel corresponding to the fourth attention module 212-2 may correspond to the listen-audio token 230 obtained by encoding the input audio content. As an example, the input audio content corresponding to the listen-audio token 230 may be a silent audio segment or it may be empty.

[0077] In some embodiments, the fifth input channel corresponding to the fourth attention module 212-2 may correspond to the listener marker 250 obtained by recognition and identification based on the input image. As an example, the listener marker 250 may be an indicator element for objects other than the target speaking object among multiple objects in the input image, such as a marker box corresponding to the location or area of ​​other objects other than the target speaking object.

[0078] In some embodiments, the fourth attention module 212-2 may update the input video features based on a cross-attention mechanism and based on the listen-audio token 230 and the listener tag 250 to obtain the second video features corresponding to the listener.

[0079] like Figure 2 As shown, the merging unit 213 includes two input channels, which correspond to the output channels of the first attention unit 211 and the second attention unit 212, respectively.

[0080] In some embodiments, the merging unit 213 merges the first video features output by the first attention unit 211 and the second video features output by the second attention unit 212 to generate input video features for the next processing layer 210.

[0081] In this way, after iterative processing through multiple processing layers 210, model 200 can output the final video features to generate video content 202.

[0082] Therefore, upon receiving a user's generation request, the video driving process can be controlled by utilizing the feature injection methods corresponding to different types of control signals, based on the type of control signals provided by the user.

[0083] In some embodiments, longer video content can also be generated by segmentation. Specifically, a first video segment generated by a video generation model based on image features and control features can be obtained. As an example, the first video segment may have a preset duration. In some embodiments, when generating the first video segment, video tokens 211a and 212a may be set to empty, for example.

[0084] Furthermore, the first video segment can be used to generate a subsequent second video segment. As an example, the second video segment may also have a preset event length. Specifically, reference video features can be determined based on multiple video frames in the first video segment. These reference video features may, for example, include video tokens 211a and 212a determined based on the multiple video frames.

[0085] Furthermore, image features, control features, and reference video features can be provided to the video generation model to generate a second video segment. That is, the original reference image can be provided during the generation of subsequent video segments. Therefore, the embodiments of this disclosure can further improve the consistency of video content while ensuring the continuity of video continuation.

[0086] Furthermore, a target video can be generated based on the generated first and second video segments. Specifically, the target video can be generated by mixing the overlapping portions of the first and second video segments. In some embodiments, since the second video segment is generated based on a portion of video frames from the first video segment, the consistency and continuity of the generated video content can be ensured. Simultaneously, at the overlapping portions of the two video segments, temporal smoothing techniques can be employed to eliminate abrupt transitions.

[0087] In this way, embodiments of the present disclosure can generate continuous video content with a longer duration based on reference images.

[0088] As described above, the embodiments of this disclosure can achieve audio-driven specified speaker video generation by using a training-free audio-driven specified speaker video generation algorithm, which decouples the "target person region" and "other regions" in the input image and provides different generation strategies. It can be plugged and used on a pre-trained single-person digital human model to achieve audio-driven specified speaker video generation.

[0089] Figures 3A to 3C Data preprocessing procedures 300A to 300C according to some embodiments of the present disclosure are illustrated. Procedures 300A to 300C may, for example, be performed by… Figure 1 The electronic device 110 shown is provided.

[0090] refer to Figure 3A As shown, the electronic device 110 can perform audio feature extraction (310) on the input audio 301 to obtain the corresponding audio token 302. As an example, the electronic device 110 can extract audio features from the input audio 301 to obtain an audio token corresponding to a specified speaker, for example... Figure 2 The “Say-Audio Token” 220 shown.

[0091] In some embodiments, the input audio 301 may be a first audio content determined based on media content such as audio or video content uploaded by the user, or a second audio content generated based on text content uploaded by the user.

[0092] refer to Figure 3B As shown, the electronic device 110 can also perform dynamic region identification (320) on the input image 303. As an example, the dynamic regions identified from the input image 303 may include human body contour regions, face regions, animal contour regions, vehicle contour regions, and other object regions that may generate dynamic activities, corresponding to each object.

[0093] As an example, input image 303 can be referenced. Figure 1 The reference image 120 is shown. The identified dynamic region may include the head region or human body region corresponding to the specified object 125 in the reference image 120, and may also include the location region of other objects besides the specified object 125.

[0094] In some embodiments, the electronic device 110 may also receive the selection or designation of at least one dynamic region in the identified dynamic region to designate the occurrence region (330) and thereby obtain the corresponding speaker label 304; or by receiving the selection or designation of at least one dynamic region to designate the listening region (340) and thereby obtain the listener label 305.

[0095] As an example, speaker marker 304 may correspond to the head or face region, and / or limb region, etc., of the specified object 125 in reference image 120. Listener marker 305 may correspond to the location region of other objects besides the specified object 125 in reference image 120.

[0096] In some embodiments, the electronic device 110 can generate a first mask corresponding to a first region (e.g., a head region or a human body region) in an input image 303 (e.g., a reference image 120), based on at least one object (e.g., a designated object 125) in the input image 303 (e.g., a reference image 120), and further generate a first control signal for the at least one object (e.g., the designated object 125) based on the first mask. For example, the first control signal corresponds to a speaking object.

[0097] In some embodiments, the electronic device 110 may also generate a second mask corresponding to a second region outside the first region in the input image 303, and generate a second control signal for objects other than the specified object based on the second mask. For example, the second control signal may correspond to a non-speaking object.

[0098] In some embodiments, the electronic device 110 can direct to Figure 2 The model 200 shown provides the first control signal, the second control signal, and the image features of the reference image 120 to generate video content spoken by the specified object 125.

[0099] As an example, the first control signal may correspond to speaker marker 304, which may serve as... Figure 2 The speaker marker 240 in the model 200 shown is used. Correspondingly, a second control signal can correspond to the listener marker 305, which can serve as... Figure 2 The listener marker 250 is used in the model shown.

[0100] refer to Figure 3C As shown, the electronic device 110 can also perform an encoding (350) operation on the input image 303 to obtain a reference token 306.

[0101] As an example, reference token 306 can be used as Figure 2 Reference token 211b and / or reference token 212b are used in the model shown.

[0102] Example process

[0103] Figure 4 A flowchart of an example process 400 for video generation according to some embodiments of the present disclosure is shown. Process 400 can be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 400.

[0104] like Figure 4As shown, in block 410, electronic device 110 obtains a video generation request associated with a reference image, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one of the plurality of objects.

[0105] As an example, the reference image can be Figure 3B and Figure 3C Input image 303, multiple objects can be referenced Figure 3B The dynamic region identified in the text, the selection of at least one of multiple objects can correspond to Figure 3B Speaker flag 304 or Figure 2 Speaker marker 240 in the text.

[0106] In box 420, electronic device 110 provides a first control signal and image features corresponding to a reference image to a video generation model to generate video content. The first control signal at least indicates the positional information of the at least one object in the reference image, and the video content includes motion content associated with the at least one object, which corresponds to audio content determined based on the video generation request.

[0107] As an example, image features can be referenced. Figure 2 Reference tokens 211b and 212b in the reference tokens.

[0108] This disclosure enables the direct generation of video content based on the selection of at least one object in a reference image and its image features, combined with audio content. The video content includes motion content associated with the at least one object. In other words, this solution can generate video content with the target object as the speaker, based on the selection of a target object from multiple objects in a reference image, combined with audio content, without affecting the motion state of other objects besides the target object. In this way, this disclosure enables the rapid generation of video content and ensures the correlation and accuracy between the motion content in the generated video content and the selected object, thus guaranteeing video quality.

[0109] In some embodiments, obtaining a video generation request associated with a reference image includes: identifying a plurality of objects in the reference image; presenting a plurality of indicator elements corresponding to the plurality of objects; and obtaining a video generation request associated with the reference image based on the selection of at least one indicator element among the plurality of indicator elements corresponding to at least one object.

[0110] In this way, the embodiments of this disclosure can quickly and accurately locate multiple objects and at least one selected object, ensuring the correlation between the video generation request and the selected object, thereby ensuring the accuracy of the subsequently generated video content.

[0111] In some embodiments, the audio content includes: first audio content determined based on uploaded media content; or second audio content generated based on text content indicated by a video generation request.

[0112] In this way, embodiments of this disclosure can generate audio content based on media content or text content, and thereby generate motion content associated with the selected object, effectively expanding the types of source content for audio content and providing convenience for users.

[0113] In some embodiments, process 400 further includes: electronic device 110 generating a first mask corresponding to a first region in a reference image based on at least one object; and generating a first control signal based on the first mask.

[0114] As an example, the first control signal can be referenced. Figure 2 Reference token 211b in the text.

[0115] In this way, embodiments of the present disclosure can generate a first mask based on a first region of the selected object in a reference image, and further generate a first control signal, effectively ensuring the correlation between the first control signal and the first region corresponding to the selected object, thereby ensuring the correlation between the motion content in the subsequently generated video content and the selected object.

[0116] In some embodiments, providing a first control signal and image features corresponding to a reference image to a video generation model to generate video content includes: generating a second mask corresponding to a second region in the reference image, the second region being located outside the first region; generating a second control signal based on the second mask; and providing the first control signal, the second control signal, and image features to the video generation model to generate video content.

[0117] As an example, the second control signal can be referenced. Figure 2 Reference token 212b in the text.

[0118] In this manner, embodiments of the present disclosure can generate a second mask for a second region outside the first region corresponding to the selected object, and further generate a second control signal based on it, thereby ensuring the correlation between the second control signal and other objects outside the selected object, further ensuring the accuracy of control over the selected object and other objects, thereby ensuring the correlation and accuracy between the motion content in the generated video content and the selected object, and ensuring the quality of the generated video content.

[0119] In some embodiments, the video generation model includes multiple processing layers, each processing layer including a first attention unit, a second attention unit, and a merging unit; wherein, the first attention unit is configured to: determine a first video feature based on a first control signal, a reference image, audio content, and preset first prompt information; the second attention unit is configured to: determine a second video feature based on a second control signal, a reference image, and preset second prompt information; and the merging unit is configured to: merge the first video feature and the second video feature to generate input video features for the next processing layer.

[0120] In this way, the embodiments of this disclosure generate video content by iterating through multiple processing layers in the video generation model, which can effectively ensure the correlation and accuracy between the first video feature and the selected object, and ensure the correlation between the second video feature and other objects and features besides the selected object, thereby ensuring the video quality of the generated video content.

[0121] In some embodiments, determining a first video feature based on a first control signal, a reference image, audio content, and a preset first prompt information includes: performing a first attention processing procedure on the input video feature, the image feature of the reference image, and the first prompt information to generate a third video feature; and performing a second attention processing procedure on the third video feature, the first control signal, and the audio feature of the audio content to generate the first video feature.

[0122] As an example, input video features may include Figure 2 The video token 211a shown can be referenced for image features. Figure 2 The reference token 211b shown can have a first prompt message that can be a speaker prompt message, such as... Figure 2 The “speak-text token” 211c is shown. The first control signal can be referenced. Figure 2 The speaker marker 240 shown can be referenced for audio features. Figure 2 The audio token shown is 220.

[0123] In some embodiments, the first attention processing procedure can be a multimodal attention processing procedure, and the second attention processing procedure can be a cross-attention processing procedure.

[0124] In this way, the embodiments of this disclosure can accurately locate the speech features in the video features through the first attention processing process, and obtain the video features corresponding to the speaker through the second attention processing process in combination with the first control signal and audio content, thereby ensuring the accuracy and effectiveness of the video features corresponding to the speaker, and ensuring the correlation and accuracy between the audio content in the motion content and the selected object.

[0125] In some embodiments, determining a second video feature based on a second control signal, a reference image, and preset second prompt information includes: performing a third attention processing procedure on the input video feature, the image feature of the reference image, and the second prompt information to generate a fourth video feature; and performing a fourth attention processing procedure on the fourth video feature and the first control signal to generate the second video feature.

[0126] As an example, input video features may include Figure 2 The image features of the video token 212a shown can be referenced. Figure 2 The reference token 212b shown can provide a second prompt message, such as a prompt message for the listener. Figure 2 The “listen-text token” 212c is shown. The second control signal can be referenced. Figure 2 The listener marker 250 shown can be referenced for audio characteristics. Figure 2 The audio token 230 shown is a listening token.

[0127] In some embodiments, the third attention processing procedure can be a multimodal attention processing procedure, and the fourth attention processing procedure can be a cross-attention processing procedure.

[0128] In this way, the embodiments of this disclosure can accurately locate the listening features in the video features through the third attention processing process, and obtain the video features corresponding to the listener through the fourth attention processing process in combination with the second control signal, thereby ensuring the accuracy and effectiveness of the video features corresponding to the listener and ensuring that other objects in the video content other than the selected object remain in a non-speaking state.

[0129] In some embodiments, a first prompt message indicates that the corresponding object is a speaker; and / or, a second prompt message indicates that the corresponding object is a non-speaker.

[0130] In this way, by specifying the object as a speaking object or a non-speaking object through prompt information, the present embodiment can effectively ensure that the specified object is in a speaking or non-speaking state in the generated video content, thereby quickly generating video content in which the specified person speaks.

[0131] Example devices and equipment

[0132] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an example device 500 for video generation according to certain embodiments of the present disclosure is shown. Device 500 may be implemented as or included in electronic device 110. Various modules / components in device 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0133] like Figure 5As shown, the apparatus 500 includes a request acquisition module 510 and a video generation module 520. The request acquisition module 510 is configured to acquire a video generation request associated with a reference video, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one object among the plurality of objects; and the video generation module 520 is configured to provide a first control signal and image features of the reference image objects to a video generation model to generate video content, wherein the first control signal indicates at least the position information of at least one object in the reference image, and the video content includes motion content associated with at least one object, the motion content corresponding to audio content determined based on the video generation request.

[0134] In some embodiments, obtaining a video generation request associated with a reference image includes: identifying a plurality of objects in the reference image; presenting a plurality of indicator elements corresponding to the plurality of objects; and obtaining a video generation request associated with the reference image based on the selection of at least one indicator element among the plurality of indicator elements corresponding to at least one object.

[0135] In some embodiments, the audio content includes: first audio content determined based on uploaded media content; or second audio content generated based on text content indicated by a video generation request.

[0136] In some embodiments, the apparatus 500 further includes a signal generation module configured to: generate a first mask corresponding to a first region in a reference image based on at least one object; and generate a first control signal based on the first mask.

[0137] In some embodiments, providing a first control signal and image features corresponding to a reference image to a video generation model to generate video content includes: generating a second mask corresponding to a second region in the reference image, the second region being located outside the first region; generating a second control signal based on the second mask; and providing the first control signal, the second control signal, and image features to the video generation model to generate video content.

[0138] In some embodiments, the video generation model includes multiple processing layers, each processing layer including a first attention unit, a second attention unit, and a merging unit; wherein, the first attention unit is configured to: determine a first video feature based on a first control signal, a reference image, audio content, and preset first prompt information; the second attention unit is configured to: determine a second video feature based on a second control signal, a reference image, and preset second prompt information; and the merging unit is configured to: merge the first video feature and the second video feature to generate input video features for the next processing layer.

[0139] In some embodiments, determining a first video feature based on a first control signal, a reference image, audio content, and a preset first prompt information includes: performing a first attention processing procedure on the input video feature, the image feature of the reference image, and the first prompt information to generate a third video feature; and performing a second attention processing procedure on the third video feature, the first control signal, and the audio feature of the audio content to generate the first video feature.

[0140] In some embodiments, determining a second video feature based on a second control signal, a reference image, and preset second prompt information includes: performing a third attention processing procedure on the input video feature, the image feature of the reference image, and the second prompt information to generate a fourth video feature; and performing a fourth attention processing procedure on the fourth video feature and the first control signal to generate the second video feature.

[0141] In some embodiments, a first prompt message indicates that the corresponding object is a speaker; and / or, a second prompt message indicates that the corresponding object is a non-speaker.

[0142] In the apparatus 500 provided in this embodiment, the specific processing procedures of each module and the technical effects they bring can be referred to the relevant description of the example process 400 above, and will not be repeated here.

[0143] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, at least one processor 610 or processing unit, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0144] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0145] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0146] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0147] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0148] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0149] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0151] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A video generation method, comprising: Obtain a video generation request associated with a reference image, wherein the reference image comprises a plurality of objects of a preset type, and the video generation request indicates the selection of at least one of the plurality of objects; and A first control signal and image features corresponding to the reference image are provided to a video generation model to generate video content, wherein the first control signal indicates at least the position information of the at least one object in the reference image, and the video content includes motion content associated with the at least one object, the motion content corresponding to audio content determined based on the video generation request.

2. The method according to claim 1, wherein, The video generation request associated with the reference image includes: Identify the plurality of objects in the reference image; Present multiple indicator elements corresponding to the multiple objects; and Based on the selection of at least one indicator element among the plurality of indicator elements corresponding to at least one object, the video generation request associated with the reference image is obtained.

3. The method according to claim 1, wherein, The audio content includes: The first audio content determined based on the uploaded media content; or The second audio content generated based on the text content indicated by the video generation request.

4. The method according to claim 1, further comprising: Based on a first region of the at least one object in the reference image, a first mask corresponding to the first region is generated; as well as The first control signal is generated based on the first mask.

5. The method according to claim 4, wherein, The step of providing a first control signal and image features corresponding to the reference image to the video generation model to generate video content includes: Generate a second mask corresponding to a second region in the reference image, wherein the second region is located outside the first region; A second control signal is generated based on the second mask; and The video generation model is provided with the first control signal, the second control signal, and the image features to generate the video content.

6. The method according to claim 5, wherein, The video generation model includes multiple processing layers, and each processing layer includes a first attention unit, a second attention unit, and a merging unit. The first attention unit is configured to: determine a first video feature based on the first control signal, the reference image, the audio content, and a preset first prompt information; The second attention unit is configured to: determine a second video feature based on the second control signal, the reference image, and preset second cue information; The merging unit is configured to merge the first video feature and the second video feature to generate input video features for the next processing layer.

7. The method according to claim 6, wherein, The step of determining the first video feature based on the first control signal, the reference image, the audio content, and the preset first prompt information includes: A first attention processing procedure is performed on the input video features, the image features of the reference image, and the first prompt information to generate a third video feature; and A second attention processing procedure is performed on the third video feature, the first control signal, and the audio features of the audio content to generate the first video feature.

8. The method according to claim 6, wherein, The step of determining the second video features based on the second control signal, the reference image, and the preset second prompt information includes: A third attention processing procedure is performed on the input video features, the image features of the reference image, and the second cue information to generate a fourth video feature; and A fourth attention processing procedure is performed on the fourth video feature and the first control signal to generate the second video feature.

9. The method according to claim 6, wherein, The first prompt message indicates that the corresponding object is the speaker; and / or The second prompt indicates that the corresponding object is not a speaker.

10. A video generation apparatus, comprising: A request acquisition module is configured to acquire a video generation request associated with a reference video, wherein the reference image includes a plurality of objects of a preset type, and the video generation request indicates the selection of at least one of the plurality of objects; and A video generation module is configured to provide a first control signal and image features of the reference image object to a video generation model to generate video content, wherein the first control signal indicates at least the position information of the at least one object in the reference image, and the video content includes motion content associated with the at least one object, the motion content corresponding to audio content determined based on the video generation request.

11. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9.

Images