Video generation method and device, computer device, storage medium and program product

Abstract

The application relates to a video generation method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a reference image of a target object and description information for generating a video; extracting subject features of the reference image and noise features of each noise image through a video diffusion model; the video diffusion model is obtained by training sample reference images of different categories of objects; the subject features are spliced with the noise features of each noise image respectively to obtain corresponding spliced features; self-attention processing is performed on the spliced features to obtain attention features; and a target video of the target object and conforming to the description information is generated according to the attention features and encoded features of the description information. The method can effectively improve the diversity of the generated video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video generation method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of computer and internet technologies, and the emergence of the stable diffusion model, significant progress has been made in the field of video generation. The stable diffusion model is a large generative model in computer vision, applicable to a wide range of scenarios, including text-to-image, image-to-image, and image-to-video generation. One widely discussed application is fine-tuning specific concepts (such as a user's avatar or a pet dog) into the generative model to provide a customized generator.

[0003] However, current video generation methods primarily rely on fine-tuning some parameters of a diffusion model to teach it features of a new reference object, thereby achieving customized video generation. Alternatively, heuristic methods are used, employing external models and modules to extract and inject features of the reference object. Regardless of the approach, accurately capturing the main features of the reference object is difficult. For instance, using fine-tuning some parameters to teach the model features of a new reference object requires retraining when the reference object changes, incurring significant training costs. This also limits the generation of customized videos, restricting them to a single category. For example, if the new reference object is a panda, the trained fine-tuned diffusion model can only generate videos related to that panda to maintain complete similarity between the generated video and the reference image. This results in relatively homogeneous customized videos. Therefore, effectively improving the diversity of generated videos is a pressing issue. Summary of the Invention

[0004] Therefore, it is necessary to provide a video generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems, which can effectively improve the diversity of generated videos.

[0005] In a first aspect, this application provides a video generation method. The method includes: acquiring a reference image of a target object and descriptive information for generating the video; extracting the main features of the reference image and the noise features of each noisy image using a video diffusion model; the video diffusion model being trained based on sample reference images of different categories of objects; concatenating the main features with the noise features of each of the noisy images to obtain corresponding concatenated features; performing self-attention processing on the concatenated features to obtain attention features; and generating a target video of the target object that conforms to the descriptive information based on the attention features and the encoded features of the descriptive information.

[0006] Secondly, this application also provides a video generation apparatus. The apparatus includes: an acquisition module for acquiring a reference image of a target object and descriptive information for generating a video; an extraction module for extracting the main features of the reference image and the noise features of each noise image using a video diffusion model; the video diffusion model is trained based on sample reference images of different categories of objects; a stitching module for stitching the main features with the noise features of each noise image to obtain corresponding stitched features; a processing module for performing self-attention processing on the stitched features to obtain attention features; and a generation module for generating a target video of the target object that conforms to the descriptive information based on the attention features and the encoded features of the descriptive information.

[0007] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, performs the following steps: acquiring a reference image of a target object and descriptive information for generating a video; extracting the main features of the reference image and the noise features of each noise image using a video diffusion model; the video diffusion model is trained based on sample reference images of different categories of objects; concatenating the main features with the noise features of each noise image to obtain corresponding concatenated features; performing self-attention processing on the concatenated features to obtain attention features; and generating a target video of the target object that conforms to the descriptive information based on the attention features and the encoded features of the descriptive information.

[0008] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps: acquiring a reference image of a target object and descriptive information for generating a video; extracting the main features of the reference image and the noise features of each noise image using a video diffusion model; the video diffusion model being trained based on sample reference images of different categories of objects; concatenating the main features with the noise features of each noise image to obtain corresponding concatenated features; performing self-attention processing on the concatenated features to obtain attention features; and generating a target video of the target object that conforms to the descriptive information based on the attention features and the encoded features of the descriptive information.

[0009] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps: acquiring a reference image of a target object and descriptive information for generating a video; extracting the main features of the reference image and the noise features of each noise image using a video diffusion model; the video diffusion model being trained based on sample reference images of different categories of objects; concatenating the main features with the noise features of each noise image to obtain corresponding concatenated features; performing self-attention processing on the concatenated features to obtain attention features; and generating a target video of the target object that conforms to the descriptive information based on the attention features and the encoded features of the descriptive information.

[0010] The aforementioned video generation method, apparatus, computer equipment, storage medium, and computer program product acquire a reference image of the target object and descriptive information for generating the video. They then extract the main features of the reference image and the noise features of each noisy image using a video diffusion model. The video diffusion model is trained based on sample reference images of different object categories. Further, the main features are concatenated with the noise features of each noisy image to obtain corresponding concatenated features, and self-attention processing is applied to the concatenated features to obtain attention features. Based on the attention features and the encoded features of the descriptive information, a target video of the target object that conforms to the descriptive information is generated. Since the main features of the reference image and the noise features of each noise image are directly extracted by the video diffusion model, and the video diffusion model in this application is trained based on sample reference images of different categories of objects, the spliced ​​features obtained by concatenating the main features with the noise features of each noise image contain reference information of reference images of different categories of objects. The attention features obtained by performing self-attention processing on the spliced ​​features have been injected with the main reference features of different categories of objects closely related to the content of the generated target video. Based on the attention features and the encoding features of the description information, target videos of different types of target objects that conform to the description information can be generated accurately and efficiently, thereby effectively improving the diversity of generated customized videos. Even after changing the target object (reference image), there is no need to readjust the model parameters and retrain the model. It can ensure that the generated target video has better fidelity of main features while maintaining the diversity of the generated target video. Attached Figure Description

[0011] Figure 1 This is a diagram illustrating the application environment of a video generation method in one embodiment;

[0012] Figure 2 This is a flowchart illustrating a video generation method in one embodiment;

[0013] Figure 3 This is a schematic diagram of the display interface of the video generation method on the product side in one embodiment;

[0014] Figure 4 This is a schematic diagram illustrating the process of inferring and generating a target video using a diffusion model in one embodiment.

[0015] Figure 5 This is a schematic diagram illustrating the training process of a diffusion model in one embodiment.

[0016] Figure 6 This is a schematic diagram comparing the effects of a customized character video generation method in one embodiment;

[0017] Figure 7This is a comparative illustration of the effects of a customized character video generation method in another embodiment;

[0018] Figure 8 This is a schematic diagram illustrating a video sample of a specific person generated using the method provided in this application in one embodiment.

[0019] Figure 9 This is a schematic diagram of a video example of a specified item generated by applying the method provided in this application in one embodiment.

[0020] Figure 10 This is a structural block diagram of a video generation device in one embodiment;

[0021] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] It should be noted that in the following description, the terms "first, second, and third" are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0024] The video generation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Specifically, terminal 102 can interact with the video generation platform, i.e., server 104. In response to a user-initiated video generation request, terminal 102 sends the request to the video generation platform, i.e., server 104, so that server 104 obtains a reference image of the target object and descriptive information for video generation. It then extracts the main features of the reference image and the noise features of each noisy image using a video diffusion model. Further, server 104 concatenates the main features with the noise features of each noisy image to obtain corresponding concatenated features, and performs self-attention processing on the concatenated features to obtain attention features. Based on the attention features and the encoded features of the descriptive information, server 104 generates a target video of the target object that conforms to the descriptive information and returns the target video to terminal 102 so that terminal 102 can display or publish the target video.

[0025] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, IoT device, or portable wearable device. IoT devices can include smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0026] Server 104 can be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (Peer To Peer) network. The Peer To Peer protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP).

[0027] In addition, server 104 can also be a server cluster consisting of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0028] Terminal 102 and server 104 can be connected via Bluetooth, USB (Universal Serial Bus) or network, etc., and this application does not impose any restrictions.

[0029] In one embodiment, such as Figure 2As shown, a video generation method is provided. This method can be executed by a server or a terminal alone, or by both a server and a terminal together. This method can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0030] Step 202: Obtain a reference image of the target object and descriptive information for generating the video.

[0031] The reference image refers to an image containing the target object. For example, the reference image in this application can be any image containing the target object. It is understood that the target object in this application can include objects of different categories. For example, the target object in this application includes, but is not limited to, people, animals, and objects.

[0032] Description information refers to information used to describe the video to be generated (i.e., the target video). For example, the description information in this application can be text information describing the characteristics of objects contained in the video to be generated (i.e., the target video). That is, the description information in this application can be a piece of descriptive text input in text form. It is understood that the description information in this application includes, but is not limited to, description information in text form, and can also be in other forms, without specific limitations here.

[0033] Specifically, the video generation method provided in this application can be widely applied to various personalized creative fields such as social networking, games, and film and television production. For example, given a descriptive text and a reference image containing a specific person, the video generation method provided in this application can generate a video that conforms to the text content contained in the text and includes the specific person from the reference image. This has wide applications in fields such as artistic creation, entertainment, advertising design, and film and television production. That is, the devices used by different users (operating objects) can interact with the multimedia information platform (or application). When a user (operating object) wants to generate a personalized video containing a specific object and conforming to the text description, the user can open the multimedia application (APP) on the terminal through a trigger operation and enter the main page of the multimedia application through a selection operation. That is, the user can log in to the multimedia application (such as a video application) through a trigger operation. Furthermore, the user can initiate a video generation request through a trigger operation on the main page displayed by the multimedia application. For example, on the main page of a game application displayed on the terminal, various objects using the game application (such as developers) can view the specific content and related functional information on the main page. Each object using the game application can also trigger a video generation request for a specific object. In response to the video generation request triggered by the object using the game application on the main page of the game application, the terminal obtains the reference image of the target object and the descriptive information used to generate the video. When the terminal obtains the descriptive information and reference image input in real time by the object using the game application on the main page of the game application, the terminal can display the specific content corresponding to the descriptive information and the reference image of the target object on the page.

[0034] It is understood that the method provided in this application can be implemented through interaction between the terminal and the backend server of the video application, or through interaction between the frontend and backend of the terminal. That is, the frontend of the terminal is used to display the reference image of the target object and the descriptive information for generating the video, while the backend of the terminal is equivalent to the backend server, which is used to perform logical processing such as recognition and encoding on the reference image of the target object and the descriptive information for generating the video input by the operation object.

[0035] For example, let's illustrate this with a scenario of customized video generation in game development. Assume the multimedia application is a game application. For example... Figure 3 The diagram shown illustrates the display interface of the video generation method provided in this application on the product side. Specifically, when user A (the object of operation) wants to generate a video containing a specific object (such as a panda named Xiaobai), user A can trigger an operation to open the game application A on the terminal and enter the video through a selection operation. Figure 3As shown in the main page of game application A, user A can log in to game application A by triggering an action. Furthermore, user A can view the following on game application A: Figure 3 On the main page shown, a video generation request is initiated by triggering an action. For example, as displayed on the terminal... Figure 3 On the main page of the game application shown, user A (such as the developer) can view the specific content and related functional information of the game's main page. User A can click on things like... Figure 3 The "Generate Input" control on the main page of the game application shown is used to make the terminal respond to user A's click operation, displaying an input box or input page. After user A completes the input operation, user A can further click as shown... Figure 3 The "Generate Start" control on the main page of the game application is used to trigger a video generation request for a specific target object. In response to user A's click on "Generate Start" (i.e., video generation request) on the main page of the game application, the terminal obtains the descriptive text A (i.e., the descriptive information used to generate the video) and the reference image A containing the target object. When the terminal obtains the descriptive text A and the reference image A containing the target object that user A inputs in real time on the main page of the game application, the terminal can display the specific text content of the descriptive text A and the reference image A containing the target object on the page.

[0036] Step 204: Extract the main features of the reference image and the noise features of each noisy image using the video diffusion model; the video diffusion model is trained based on sample reference images of different categories of objects.

[0037] The video diffusion model refers to a pre-trained diffusion model used to generate videos. In this application, the video diffusion model can be a component of the overall diffusion model; for example, it could be a Video Diffusion Model (VDM). Furthermore, the video diffusion model in this application is trained based on sample reference images of different object categories. Therefore, the pre-trained video diffusion model can extract the main features of different object categories from the reference images using its own network structure, without requiring an external main feature extraction model or module, and without needing to fine-tune the parameters for each new reference image.

[0038] Subject features refer to features used to characterize the relevant features of a subject in a reference image. For example, the subject features in this application may be the appearance features and attribute features of the subject extracted from the reference image. It is understood that the subject features in this application include, but are not limited to, appearance features and attribute features, and may also include other features characterizing the features of the subject.

[0039] Noise features refer to the features used to characterize each frame of a noisy image. It can be understood that the noise features in this application can be multi-frame noise features extracted from multiple frames of noisy images. For example, if the target video to be generated in this application contains 16 video frames, then the noise features of each of the 16 noisy images can be extracted to obtain 16 frames of noise features.

[0040] A noisy image refers to an image generated based on initialization noise. For example, the noisy image in this application may be a multi-frame noisy image automatically generated by a diffusion model based on initialization noise. It can be understood that the number of frames of the noisy image generated in this application may also be determined based on the initialization noise.

[0041] Specifically, after the terminal acquires the reference image of the target object and the descriptive information used to generate the video, the terminal can call a pre-trained diffusion model for processing. That is, the terminal can use the acquired reference image and descriptive information as input parameters and input them into the pre-trained diffusion model so that the encoder in the diffusion model encodes the reference image and the initial noise respectively, to obtain the encoded reference image and the noise image of the preset number of frames. Further, the terminal can use the diffusion model to combine the encoded reference image and the noise image of the preset number of frames into a stitched image, and extract the mixed image features from the stitched image through the sub-model in the diffusion model, namely the video diffusion model. From the mixed image features, the main features of the reference image and the noise features of each noise image can be separated, thus obtaining the main features of the reference image and the noise features of each noise image.

[0042] For example, such as Figure 4 The diagram illustrates the process of inference and generation of the target video using a diffusion model. Assume the reference image of the target object (female celebrity A) acquired by the terminal is as follows: Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 After the description text shown ("A person playing an acoustic guitar"), the terminal can call a pre-trained program such as... Figure 4 The diffusion model shown takes the acquired reference image (portrait of actress A) and descriptive information (text) as input parameters and feeds them into the pre-trained diffusion model to achieve the desired effect. Figure 4 The encoder in the diffusion model shown, i.e., the VAE Encoder, encodes the reference image and the initial noise respectively, obtaining an encoded reference image r0 and a noise image z0 with a preset number of frames. Furthermore, the terminal can use the diffusion model to combine the encoded reference image r0 and the noise image z0 with a preset number of frames to form a stitched image z. t And through a sub-model in the diffusion model, namely the video diffusion model (VDM), from the stitched image zt Extract the mixed image features f, and separate the main features f of the reference image from the mixed image features f. r Noise features f of each noisy image z This allows us to obtain the main features f of the reference image. r Noise features f of each noisy image z .

[0043] Step 206: The main features are concatenated with the noise features of each noise image to obtain the corresponding concatenated features.

[0044] Here, the splicing feature refers to the splicing feature obtained by splicing the main features extracted from the reference image with the noise features of each frame of the noisy image. In this application, the splicing feature is used as the input parameter of the spatial self-attention layer of each frame of the image. The spatial self-attention layer processes the input splicing feature and outputs the attention feature of each frame of the image.

[0045] Step 208: Perform self-attention processing on the spliced ​​features to obtain attention features.

[0046] Among them, attention features refer to the fused features output after information is fused through spatial self-attention mechanism. It can be understood that the attention features in this application are used to characterize the relationship between pixels in a single frame.

[0047] Self-attention processing refers to the processing of the relationship between pixels in a single frame image based on the spatial self-attention mechanism. For example, in this application, the input parameters can be processed by the spatial self-attention layer in the video diffusion model.

[0048] Specifically, after the terminal extracts the main features of the reference image and the noise features of each noise image through the video diffusion model, the terminal can use the extracted main features of the reference image and the noise features of each noise image as input parameters for the spatial self-attention layer in the video diffusion model. After processing by the spatial self-attention layer, the terminal outputs attention features. That is, the terminal can use the spatial self-attention layer in the video diffusion model to concatenate the extracted main features of the reference image with the noise features of each noise image to obtain a preset number of concatenated features. Then, it performs self-attention processing on the preset number of concatenated features to obtain a preset number of attention features. Furthermore, the terminal can use the spatial self-attention layer in the video diffusion model to split the obtained preset number of attention features to obtain preset number of attention noise features and attention main features. The terminal then calculates the average value of the preset number of attention main features to obtain the target attention main features. Finally, the target attention main features are concatenated with the preset number of attention noise features to obtain the updated attention features for the preset number of frames.

[0049] For example, suppose the reference image of the target object (female celebrity A) obtained by the terminal is as follows: Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 The description text shown (Text: A person playing an acoustic guitar) can be used by the terminal to call pre-trained programs such as... Figure 4 The diffusion model shown takes the acquired reference image (portrait of actress A) and descriptive information (text) as input parameters and feeds them into the pre-trained diffusion model to achieve the desired effect. Figure 4 The sub-model in the diffusion model shown, namely the convolutional layer (Conv layer) in the video diffusion model (VDM), is derived from the stitched image z. t After extracting the mixed image features f, the terminal can use the mixed image features f as input parameters for the next layer in the Video Diffusion Model (VDM). In other words, the terminal can use the mixed image features f as... Figure 4 The input parameters of the spatial self-attention layer (SA layer) in the video diffusion model (VDM) shown are the parameters by which the terminal can separate the main features f of the reference image from the mixed image features f through the spatial self-attention layer in the VDM. r Noise features f of each noisy image z And extract the main features f of the reference image r Noise features f of noisy images with a preset number of frames respectively z The splicing is performed to obtain the splicing features X of a preset number of frames, and self-attention processing is performed on the splicing features X of the preset number of frames to obtain the attention features X* of the preset number of frames.

[0050] Furthermore, the terminal can use the spatial self-attention layer in the video diffusion model to decompose the attention features X* of the preset number of frames, thereby obtaining the attention noise features f of the preset number of frames. z 'and attention subject characteristics f r ', and focus on the subject features f of the preset number of frames. r 'Calculate the average value to determine the target attention subject feature f' rr ', then focus attention on the main features f rr 'Attention noise features f of the preset number of frames respectively z 'Concatenate the data to obtain the updated attention features f* with a preset number of frames.'

[0051] Step 210: Generate a target video that matches the description information and targets the target object, based on the attention features and the encoding features of the description information.

[0052] The encoded features of the descriptive information refer to the features obtained after encoding the descriptive information through a text encoder. For example, the text encoder in this application can be as follows: Figure 4 The CLIP encoder shown.

[0053] Specifically, after obtaining attention features, the terminal can, for example, Figure 4 The sub-model in the diffusion model shown, namely the spatial self-attention layer of the Video Diffusion Model (VDM), splits the attention features of a preset number of frames into attention noise features and attention subject features. It then calculates the average value of the attention subject features to obtain the target attention subject features. These target attention subject features are then concatenated with the attention noise features of the preset number of frames to obtain the updated attention features for the preset number of frames. Furthermore, the terminal can use the updated attention features of the preset number of frames as input parameters to the next layer (CA layer) of the VDM. That is, the terminal processes the updated attention features of the preset number of frames and the encoded features of the descriptive information through the VDM, outputting a target video that matches the descriptive information and represents the target object.

[0054] For example, suppose the reference image of the target object (female celebrity A) obtained by the terminal is as follows: Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 The description text shown (Text: A person playing an acoustic guitar) can be used by the terminal to call pre-trained programs such as... Figure 4 The diffusion model shown takes the acquired reference image (portrait of actress A) and descriptive information (text) as input parameters and feeds them into the pre-trained diffusion model. After processing as follows... Figure 4 The sub-model in the diffusion model shown is the spatial self-attention layer of the Video Diffusion Model (VDM), which processes and outputs updated attention features f* for a preset number of frames. Furthermore, the terminal can use the updated attention features f* for the preset number of frames as input parameters for the next layer of the VDM. That is, the terminal processes the updated attention features f* for the preset number of frames and the encoded features of the descriptive text ("A person playing an acoustic guitar") through the VDM, outputting a target video of the target object, female celebrity A, that matches the descriptive text ("A person playing an acoustic guitar").

[0055] In this embodiment, a reference image of the target object and descriptive information for generating a video are obtained, and the main features of the reference image and the noise features of each noise image are extracted by a video diffusion model. The video diffusion model is trained based on sample reference images of different categories of objects. Furthermore, the main features are concatenated with the noise features of each noise image to obtain corresponding concatenated features, and the concatenated features are subjected to self-attention processing to obtain attention features. Based on the attention features and the encoded features of the descriptive information, a target video of the target object that conforms to the descriptive information is generated. Since the main features of the reference image and the noise features of each noise image are directly extracted by the video diffusion model, and the video diffusion model in this application is trained based on sample reference images of different categories of objects, the spliced ​​features obtained by concatenating the main features with the noise features of each noise image contain reference information of reference images of different categories of objects. The attention features obtained by performing self-attention processing on the spliced ​​features have been injected with the main reference features of different categories of objects closely related to the content of the generated target video. Based on the attention features and the encoding features of the description information, target videos of different types of target objects that conform to the description information can be generated accurately and efficiently, thereby effectively improving the diversity of generated customized videos. Even after changing the target object (reference image), there is no need to readjust the model parameters and retrain the model. It can ensure that the generated target video has better fidelity of main features while maintaining the diversity of the generated target video.

[0056] In one embodiment, the method further includes:

[0057] The reference image and initial noise are encoded separately by the encoder in the diffusion model to obtain the encoded reference image and the noise image with a preset number of frames.

[0058] The coded reference image is combined with a noise image of a preset number of frames to form a stitched image;

[0059] The step of extracting the main features of the reference image and the noise features of each noisy image using a video diffusion model includes:

[0060] The video diffusion model is used to extract the mixed image features from the stitched images; the video diffusion model is a sub-model in the diffusion model; from the mixed image features, the main features of the reference image and the noise features of each noisy image are separated.

[0061] Here, an encoder refers to an encoder used to transform an image from pixel space to latent space. For example, the encoder in this application could be as follows: Figure 4 The variational encoder shown is the VAE Encoder.

[0062] Initialization noise refers to the initialization noise value automatically generated during the initialization of the diffusion model, which can be a random number.

[0063] A stitched image refers to an image obtained by combining multiple noisy frames with a single coded reference image, resulting in a stitched image consisting of multiple noisy frames plus a single coded reference image (one frame). For example, the stitched image in this application could be as follows: Figure 4 The z shown is obtained by performing a C operation on multiple noisy images z0 and a coded reference image r0. t .

[0064] Hybrid image features refer to the mixed features extracted from stitched images. It can be understood that the hybrid image features f in this application may include noise features f corresponding to the noise portion of the video to be generated. z and the main features f corresponding to the reference image portion r .

[0065] Specifically, let's take a scenario of customized video generation as an example. Assume the reference image of the target object (female celebrity A) input by user A, obtained by the terminal, is as follows: Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 After displaying the descriptive text "Text (A person playing anacoustic guitar)" as shown, the terminal can invoke pre-trained programs such as... Figure 4 The diffusion model shown takes the acquired reference image (portrait of actress A) and descriptive information (text) as input parameters and feeds them into the pre-trained diffusion model to achieve the desired effect. Figure 4 The encoder in the diffusion model shown, i.e., the VAE Encoder, encodes the reference image and the initial noise respectively, obtaining an encoded reference image r0 and a noise image z0 with a preset number of frames. Furthermore, the terminal can use the diffusion model to combine the encoded reference image r0 and the noise image z0 with a preset number of frames to form a stitched image z. t And through the convolutional layer (Conv layer) in the video diffusion model (VDM), a sub-model in the diffusion model, from the stitched image z t The mixed image features f are extracted, and the main features f of the reference image are separated from the mixed image features f by the spatial self-attention layer (SA layer) in the video diffusion model VDM. r Noise features f of each noisy image z This allows us to obtain the main features f of the reference image. r Noise features f of each noisy image zThis allows for the generation of high-quality, smooth-moving customized videos while maintaining the main characteristics of various characters (including people and objects), and also effectively improves the efficiency and diversity of customized video generation.

[0066] In one embodiment, the step of encoding the reference image and initial noise respectively using an encoder in the diffusion model to obtain an encoded reference image and a noise image with a preset number of frames includes:

[0067] The initial noise is encoded by the first variational encoder in the diffusion model to obtain a noise image with a preset number of frames;

[0068] The reference image is encoded by the second variational encoder in the diffusion model to obtain the encoded reference image.

[0069] In this application, the first variational encoder and the second variational encoder can be the same encoder, i.e., both are VAE Encoders.

[0070] Specifically, let's take a scenario of customized video generation as an example. Assume that the reference image A of the target object (female celebrity A) input by user A, obtained by the terminal, is as follows: Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 After displaying the descriptive text TextA (A person playing anacoustic guitar), the terminal can invoke pre-trained programs such as... Figure 4 The diffusion model shown takes the acquired reference image (portrait of actress A) and descriptive information (text) as input parameters and feeds them into the pre-trained diffusion model to achieve the desired effect. Figure 4 The VAE Encoder in the diffusion model shown encodes the initial noise to obtain a noisy image z0 with a preset number of frames, and encodes the reference image A to obtain an encoded reference image r0. This allows for the generation of high-quality, smooth-moving customized videos while better preserving the main features of various characters (including people and objects), and also effectively improves the efficiency of customized video generation.

[0071] In one embodiment, the method further includes:

[0072] Noise is added to each noisy image to obtain a set number of noisy images;

[0073] The step of assembling a stitched image from the encoded reference image and the noise image of a preset number of frames includes:

[0074] The coded reference image is combined with a pre-defined number of noisy images to form a stitched image.

[0075] Specifically, suppose the reference image A of the target object (female celebrity A) input by user A, obtained by the terminal, is as follows: Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 After displaying the descriptive text TextA (A person playing an acoustic guitar), the terminal can call a pre-trained program such as... Figure 4 The diffusion model shown takes the obtained reference image A (the portrait of actress A) and descriptive information, i.e., descriptive text TextA, as input parameters and feeds them into the pre-trained diffusion model to achieve the desired effect. Figure 4 The encoder in the diffusion model shown, i.e., the VAE Encoder, encodes the reference image and the initial noise respectively, obtaining an encoded reference image r0 and a noise image z0 with a preset number of frames. Furthermore, the terminal can use the diffusion model to add noise to the noise image z0 with a preset number of frames, obtaining a noisy image z with a preset number of frames. t Then, the encoded reference image r0 is compared with the noise-added image z of a preset number of frames. t Composition of stitched image z t This allows for the generation of high-quality, smooth-moving customized videos while better preserving the main characteristics of various characters (including people and objects), and also effectively improves the efficiency of generating customized videos.

[0076] In one embodiment, the step of concatenating the main features with the noise features of each noise image to obtain the corresponding concatenated features includes:

[0077] By using the spatial self-attention layer in the video diffusion model, the main features are concatenated with the noise features of a preset number of noisy images to obtain the concatenated features of the preset number of frames.

[0078] The process of performing self-attention processing on the spliced ​​features to obtain attention features includes:

[0079] By using a spatial self-attention layer, the splicing features of a preset number of frames are processed with self-attention to obtain the attention features of the preset number of frames.

[0080] The spatial self-attention layer is used to model the relationships between pixels within a single frame. For example, the spatial self-attention layer in this application can be as follows: Figure 4 The sub-model in the diffusion model shown is the second layer (SA layer) in the Video Diffusion Model (VDM).

[0081] Specifically, suppose the reference image A of the target object (female celebrity A) obtained by the terminal is: (e.g., ...) Figure 4 The image of the specific person shown (the portrait of actress A), and the descriptive information used to generate the video are: as follows Figure 4 The description text TextA shown (A person playing an acoustic guitar) can be used by the terminal to call pre-trained text such as... Figure 4 The diffusion model shown takes the obtained reference image A (the portrait of actress A) and descriptive information, i.e., descriptive text TextA, as input parameters and feeds them into the pre-trained diffusion model to achieve the desired effect. Figure 4 The sub-model in the diffusion model shown, namely the convolutional layer (Conv layer) in the video diffusion model (VDM), is derived from the stitched image z. t After extracting the mixed image features f, the terminal can use the mixed image features f as input parameters for the next layer in the Video Diffusion Model (VDM). In other words, the terminal can use the mixed image features f as... Figure 4 The input parameters of the spatial self-attention layer (SA layer) in the video diffusion model (VDM) shown are the parameters by which the terminal can separate the main features f of the reference image from the mixed image features f through the spatial self-attention layer in the VDM. r Noise features f of each noisy image z And extract the main features f of the reference image r Noise features f of noisy images with a preset number of frames respectively z The splicing process is performed to obtain splicing features X for a preset number of frames. Self-attention processing is then applied to these splicing features X to obtain attention features X* for a preset number of frames. This allows for the generation of high-quality, smooth-moving customized videos while better preserving the main features of various characters (including people and objects), and also effectively improves the efficiency of customized video generation.

[0082] In one embodiment, the method further includes:

[0083] Each attention feature is decomposed to obtain attention noise features and attention subject features for a preset number of frames;

[0084] The target attention subject features are obtained by averaging the attention subject features of a preset number of frames.

[0085] The target attention feature is concatenated with each attention noise feature to obtain the updated attention feature after a preset number of frames.

[0086] The step of generating a target video of the target object that conforms to the description information based on the attention features and the encoding features of the description information includes:

[0087] Based on the updated attention features and the encoded features of the descriptive information, a target video that matches the descriptive information is generated.

[0088] In this application, the attention features of each frame can be decomposed into attention noise features and attention subject features. For example, the attention features of each frame can be as follows: Figure 4 X* as shown in the figure.

[0089] Specifically, the terminal can use the spatial self-attention layer in the video diffusion model to split the attention features X* of each frame of the preset number of frames, thereby obtaining the attention noise features f of the preset number of frames. z Attention subject features f and preset frame number r ', and focus on the subject features f of the preset number of frames. r 'Calculate the average value to obtain the target attention subject feature f' rr ', then focus attention on the main features f rr 'Attention noise features f of the preset number of frames respectively z The data is then concatenated to obtain the updated attention features f* for a preset number of frames. Furthermore, the terminal can use the updated attention features f* for the preset number of frames as input parameters for the next layer (CA layer) of the Video Diffusion Model (VDM). That is, the terminal processes the updated attention features f* for the preset number of frames and the encoded features of the descriptive text TextA (A person playing an acoustic guitar) through the Video Diffusion Model (VDM), outputting a target video of the target object, female celebrity A, that matches the descriptive text TextA (A person playing an acoustic guitar). This allows for the generation of high-quality, smooth-moving customized videos while better preserving the main features of various roles (including people and objects), and also effectively improves the efficiency of generating customized videos.

[0090] In one embodiment, the step of generating a target video that matches the description information based on the updated attention features and the encoded features of the description information includes:

[0091] By using the cross-attention layer in the video diffusion model, the updated attention features and the encoded features of the descriptive information are processed to obtain the latent space features of the number of targets.

[0092] The latent space features are filtered to obtain latent space features for a preset number of frames;

[0093] The latent space features of a preset number of frames are converted into pixel space features of a preset number of frames using the decoder in the diffusion model.

[0094] Based on the pixel space features of a preset number of frames, a target video that matches the description information is generated for the target object.

[0095] Here, filtering refers to filtering the number of frames. For example, if the latent space features of the target number are obtained as the latent space features of N+1 frames of images, then the latent space features of the last reference frame need to be removed to obtain the latent space features of the filtered N frames of noisy images.

[0096] Specifically, after the terminal obtains the updated attention features f* for a preset number of frames, the terminal can, as follows: Figure 4 The cross-attention layer (CA layer) in the video diffusion model shown processes the updated attention features and the encoded features of the descriptive information to obtain latent space features for the number of targets. The terminal can filter the latent space features for the number of targets (N+1), i.e., remove the latent space features of the last reference frame, to obtain latent space features for the preset number of frames (N). Furthermore, the terminal can use the decoder in the diffusion model to convert the latent space features for the preset number of frames (N) into pixel space features for the preset number of frames (N), and based on the pixel space features for the preset number of frames, generate a target video containing N video frames that matches the descriptive information and contains the target object. This allows for the generation of high-quality, smooth, customized videos while better preserving the main features of various roles (including people and objects), and also effectively improves the efficiency of generating customized videos.

[0097] In one embodiment, the method further includes:

[0098] Acquire training sample videos, sample reference images, and sample description information;

[0099] The initial diffusion model is trained using training sample videos, sample reference images, and sample description information as training data to obtain the diffusion model.

[0100] The sample reference images in this application may include images of different categories (types). For example, the sample reference images in this application include, but are not limited to: reference images of people, reference images of objects, and image sets of various objects combined.

[0101] Specifically, such as Figure 5The diagram illustrates the training process of the diffusion model. During the model training phase, the terminal acquires training sample videos, sample reference images, and sample description information. These are then input into the initial diffusion model for training. The terminal uses the VAE Encoder within the initial diffusion model to encode the reference image and the training sample video, obtaining an encoded reference image r0 and a preset number of sample video images z0. Further, the terminal uses the initial diffusion model to combine the encoded reference image r0 with the preset number of sample video images z0 to form a stitched image z. t and stitch the image z t As input parameters for the video diffusion model (VDM), a sub-model within the diffusion model, the inputs are as follows: Figure 5 The trained diffusion model can be obtained by training the video diffusion model (VDM) shown in the diagram. Therefore, through a visual interface, specific training data can be quickly and efficiently input into the diffusion model, providing a customized diffusion model for video generation. This simplifies the tedious model training process and effectively improves both the accuracy of generated videos and the training efficiency of the diffusion model.

[0102] In one embodiment, the step of training an initial diffusion model using training sample videos, sample reference images, and sample description information as training data to obtain a diffusion model includes:

[0103] The training sample video and the sample reference image are encoded separately to obtain the encoded video image and the encoded sample reference image;

[0104] Add first noise to each encoded video image to obtain a noisy video image with a preset number of frames;

[0105] Add a second noise to the coded sample reference image to obtain a noisy reference image;

[0106] The noisy reference image and the noisy video images of a preset number of frames are combined to form a stitched sample image;

[0107] The stitched sample images are used as input parameters to train the initial diffusion model, resulting in the trained diffusion model.

[0108] The first noise and the second noise are used to distinguish the noise added to different encoded images. For example, the first noise in this application refers to the noise added to each encoded video image, and the second noise in this application refers to the cue noise added to the encoded sample reference image. That is, the cue noise is a slight noise used to enable the model to accurately distinguish the reference information and the generated video content, thereby improving the quality of customized video generation.

[0109] Specifically, such as Figure 5 The diagram illustrates the training process of the diffusion model. During the model training phase, the terminal acquires training sample videos, sample reference images, and sample description information. These are then input into the initial diffusion model for training. The terminal uses the VAE Encoder in the initial diffusion model to encode the reference image and the training sample video, obtaining an encoded sample reference image r0 and encoded video images z0 with a preset number of frames. Further, the terminal can add a first noise N to each encoded video image using the initial diffusion model, obtaining a noisy video image z0 with a preset number of frames. t And by adding a second noise, namely the warning noise α*N, to the coded sample reference image r0, we obtain the noisy reference image r. t Furthermore, the terminal uses methods such as... Figure 5 The C operation shown will add noise to the video image z at a preset number of frames. t With the noisy reference image r t Composition of stitched sample images z t ', and stitch the image z t As input parameters for the Video Diffusion Model (VDM), a sub-model within the diffusion model, the input is as follows: Figure 5 The trained diffusion model can be obtained by training the video diffusion model (VDM) shown in the diagram. This allows for the generation of high-quality, smooth-moving customized videos while better preserving the main features of various characters (including people and objects), and also effectively improves the efficiency of customized video generation.

[0110] In one embodiment, the step of training an initial diffusion model using stitched sample images as input parameters to obtain a trained diffusion model includes:

[0111] The stitched sample image is used as input parameter to train the initial diffusion model. Training stops when the target loss value determined based on the first noise and the second noise meets the preset loss condition, and the trained diffusion model is obtained.

[0112] Specifically, such as Figure 5 The diagram illustrates the training process of the diffusion model. Specifically, during the model training phase, the terminal processes the data through methods such as... Figure 5 The C operation shown will add noise to the video image z at a preset number of frames. t With the noisy reference image r t Composition of stitched sample images z t ', and stitch the image z t As input parameters for the Video Diffusion Model (VDM), a sub-model within the diffusion model, the input is as follows: Figure 5The video diffusion model (VDM) shown is trained until the target loss value L, determined based on the first noise N and the second noise (i.e., the alert noise α*N), meets the preset loss condition, and then the training stops, thus obtaining the trained diffusion model.

[0113] It is understood that the reminder noise in the embodiments of this application The calculation method can be shown in the following formula:

[0114]

[0115] Where t' = α∙t, α is a manually set hyperparameter. To prevent additional noise from significantly damaging the reference information, the value of α is set to a small value (e.g., 0.01) to ensure that the cue noise is a slight noise.

[0116] In this embodiment, high-quality, smooth-moving customized videos can be generated while maintaining the main characteristics of various characters (including people and objects) well, and the efficiency of generating customized videos can also be effectively improved.

[0117] In one embodiment, the step of determining the target loss value based on the first noise and the second noise includes:

[0118] The first loss value is determined based on the encoded video image, the first noise, the sample description information, and a preset number of video frames output by the initial diffusion model.

[0119] The second loss value is determined based on the coded sample reference image, the first noise, the sample description information, and the reference frame output by the initial diffusion model; wherein the reference frame is determined based on the second noise, the sample description information, and the time step.

[0120] Determine the sum between the first loss value and the second loss value, and use the sum as the target loss value.

[0121] Specifically, due to the actual input z in the technical solution provided in this application t Compared to the original video diffusion model, it has one more frame (i.e., a reference image frame) as its input parameters, therefore the output... An extra frame was added. Therefore, a simple and intuitive approach is to remove the output corresponding to the reference information r and only calculate the loss for the remaining frames. This method encourages the model to focus on learning customized video generation with a specified subject. However, observations of the final results show that without supervision of the reference information, the model struggles to accurately identify the reference information r as an image without added noise, leading to instability in the generated results. To address this issue, the training method provided in this application introduces a guidance information recognition loss to supervise the reference information, enabling the model to accurately distinguish between the reference information and the generated video content, thereby improving the quality of the customized generated video. Specifically, during training, at time step t, a reminder noise is added to the reference information r. Therefore, when calculating the loss function, the same loss as the diffusion model needs to be calculated for the reference information r. That is, the terminal can determine the first loss value based on the encoded video image, the first noise, the sample description information, and a preset number of video frames output by the initial diffusion model. Among them, the first loss value The calculation method can be shown in the following formula:

[0122]

[0123] in, ϵ represents the input text information, ε represents the added noise, and t represents the time step of the current diffusion model. For diffusion models, This represents the output of the diffusion model.

[0124] Furthermore, the terminal can determine the second loss value based on the coded sample reference image, the first noise, the sample description information, and the reference frame output by the initial diffusion model. The reference frame is determined based on the second noise, sample description information, and time step; the second loss value... The calculation method can be shown in the following formula:

[0125]

[0126] in, ϵ represents the input text information, ε represents the added noise, and t represents the time step of the current diffusion model. For diffusion models, This represents the output of the diffusion model.

[0127] Furthermore, the terminal can determine the sum between the first loss value and the second loss value, and use this sum as the target loss value, that is, the terminal will... As an auxiliary optimization objective, it is combined with the primary objective to guide model training, where the target loss value... The calculation method can be shown in the following formula:

[0128]

[0129] Here, β is a hyperparameter. To avoid interfering with the optimization of the main custom video generation task, β is set to a relatively small value (e.g., 0.1).

[0130] In this embodiment, high-quality, smooth-moving customized videos can be generated while maintaining the main characteristics of various characters (including people and objects) well, and the efficiency of generating customized videos can also be effectively improved.

[0131] This application also provides an application scenario in which the above-described video generation method is applied. Specifically, the video generation method is applied in this scenario as follows:

[0132] During the interaction between the user and the multimedia information platform, the aforementioned video generation method can be used. When the user (the object of operation) wants to generate a personalized video containing a specific object and conforming to the text description, the user can open the multimedia application on the terminal by triggering an operation and enter the main page of the multimedia application by selecting an operation. That is, the user can log in to the multimedia application by triggering an operation. Furthermore, the user can initiate a video generation request by triggering an operation on the main page displayed by the multimedia application. The terminal responds to the video generation request triggered by the user and sends the video generation request to the background server of the multimedia application. This allows the background server to obtain the reference image of the target object input by the user and the descriptive information used to generate the video. It also extracts the main features of the reference image and the noise features of each noise image through a video diffusion model. Further, the background server concatenates the main features with the noise features of each noise image to obtain the corresponding concatenated features, and performs self-attention processing on the concatenated features to obtain attention features. Based on the attention features and the encoded features of the descriptive information, the background server generates a target video of the target object that conforms to the descriptive information and returns the target video to the terminal so that the terminal can preview or store the target video. Therefore, when different users interact with the multimedia information platform, the video generation method provided in this application can generate high-quality, smooth-moving customized videos while maintaining the main characteristics of various roles (including people and objects), and can also effectively improve the efficiency of generating customized videos.

[0133] The method provided in this application can be applied to various video generation scenarios. The following description uses a scenario of user interaction with a multimedia information platform as an example to illustrate the video generation method provided in this application.

[0134] Among them, Diffusion: diffusion model.

[0135] Video Diffusion Model (VDM): Video Diffusion Model.

[0136] Text-to-Video (T2V) model: A model that can generate a video that matches the user's description based on a text prompt given by the user.

[0137] Zero-shot customized video generation: For users, any reference image can be input, and even if the reference image does not appear in the training samples, the video generation model can still be used to generate customized videos without further fine-tuning of the model.

[0138] Traditionally, single-role customized video generation techniques fall into two categories based on whether the video diffusion model needs to be retrained when changing different reference objects. The first type of method fine-tunes some parameters of the video diffusion model to allow it to learn the concept of new reference objects, thus achieving customized video generation. However, this type of method requires users to re-fine-tune the model for different reference objects, significantly increasing computational costs and causing considerable inconvenience. The second type of method, benefiting from work such as zero-shot customized image generation techniques, employs a heuristic approach, using external models and corresponding modules to extract and inject reference object features.

[0139] Problems with traditional technical solutions:

[0140] Traditional single-role customized video generation techniques can be divided into two categories based on whether the video diffusion model needs to be retrained when changing different reference objects. The first category involves fine-tuning some parameters of the video diffusion model to allow it to learn the concept of new reference objects, thus achieving customized video generation. However, this type of method requires users to re-fine-tune the model for different reference objects, significantly increasing computational costs and causing considerable inconvenience. The second category, benefiting from zero-shot customized image generation techniques, employs heuristic methods, using external models and corresponding modules to extract and inject reference object features. The drawback of traditional methods that require retraining the model when changing reference objects is that they impose significant training costs on users, limiting the development and practical application of customized video generation technology. Traditional zero-shot customized video generation methods assume that the video diffusion model itself lacks the ability to extract reference object features, relying on external models to extract and inject subject features, often neglecting the inherent capabilities of the video diffusion model. For example, some methods, such as Animate Anyone and X-portrait, use an additional ReferenceNet for feature extraction and directly add the subject features to the features in the video diffusion model before injection. However, these methods introduce a large number of additional training parameters, and the pixel-by-pixel injection method greatly limits the diversity of generated videos. Other methods, such as IP-Adapter, PhotoMaker, VideoBooth, and ID-Animator, use the pre-trained cross-modal alignment model CLIP as a feature extractor and inject subject features through a cross-attention layer. However, limited by the fact that the pre-trained cross-modal alignment model itself focuses on semantic-level feature understanding, these methods only use coarse-grained semantic-level features generated from the pre-trained extractor and cannot capture the appearance details of the reference object. Therefore, these carefully designed heuristic methods have failed to achieve satisfactory results in customized video generation.

[0141] The technical solution provided in this application can solve these problems simultaneously through the design of an innovative and complete process:

[0142] In this application, based on observations of video diffusion models, several potential capabilities of video diffusion models were discovered. For reference object feature extraction, since a noise-free input reference image can be considered a special case where the video diffusion model's time step is 0, the pre-trained video diffusion model is already able to extract features from it without additional training. For reference object feature injection, unlike existing methods that rely on cross-attention modules for feature injection, it was found that spatial self-attention in the video diffusion model primarily models the relationships between different pixels within a frame, making it more suitable for injecting subject reference features closely related to the generated content. Furthermore, due to the adaptive nature of spatial self-attention, it can selectively interact with these features, which helps prevent overfitting and promotes the diversity of generated videos. Therefore, this application proposes to achieve high-quality zero-shot customized video generation by leveraging the video diffusion model itself. Specifically, this application designs a single-role customized video generation method applicable to text-video diffusion models. Given a pre-trained text-video diffusion model and a reference image of the target object, the goal is for the model to learn the appearance and attributes of the object and generate a video with the reference object based on a user-provided text description. Simultaneously, the aim is to obtain a model that does not require fine-tuning parameters after changing a specified object, thereby facilitating users to easily generate videos of any object they wish to generate. Unlike simple image-to-video tasks that require maintaining complete similarity between the generated video and the reference image, the training method provided in this application aims to enable the model to learn only the concepts and features of the subject in the reference image and generate diverse videos based on text prompts.

[0143] To achieve customized video generation, two steps are required: reference object feature extraction and information injection. For reference object feature extraction, the reference image is directly input into the video diffusion model, utilizing its inherent feature extraction process to extract the appearance and attribute features of the reference object. This results in fine-grained features with minimal domain difference compared to the pre-trained video diffusion model, reducing the difficulty for the pre-trained model to understand them. For the information injection step, a bidirectional interaction mechanism between the reference object features and the generated content is established using the spatial self-attention mechanism within the diffusion model. This ensures better subject fidelity in the generated video while maintaining its diversity.

[0144] On the product side, the technical solution provided in this application is applicable to single-role zero-sample customized video generation. By providing a single reference image of a specified item, the T2V model can extract the item information from the image and generate a video containing the specified item that matches the text description based on the user's input text prompts. This has great value in applications such as customized video generation.

[0145] On the technical side, the model proposed in this application is based on, for example... Figure 5 The process framework shown is illustrated. That is, the proposed solution design can be based on the AnimateDiff model.

[0146] 1.1 Reference Item Feature Extraction

[0147] To enable the model to learn the appearance and attributes of a subject, its features are first extracted. Unlike previous methods, this is achieved using the existing network structure of VDM. For example... Figure 5 As shown, given a video x, it is encoded into the latent space using a VAE and noise is added to obtain... Given a reference image R for a specified subject, the reference image R is first encoded using a VAE to obtain r, without adding noise. Then, the encoded reference image latent space r is compared with z. t By splicing the frames together, we obtain... This serves as the actual input to the model. Next, convolutional layers in VDM are used as feature extractors to extract features from zt', obtaining the input to the spatial self-attention layer. Then, the feature f is separated to obtain the noise part corresponding to the video to be generated. and the part corresponding to the reference information At this point, the feature extraction of the specified subject has been completed.

[0148] 1.2 Customized Information Injection Module

[0149] After extracting the characteristics of the specified item, the next step is to inject these characteristics into the VDM. Each frame in Before computing spatial self-attention, it is converted into h×w tokens. and Concatenation transforms the input of the spatial self-attention layer in each frame into 2×h×w tokens, which are then represented as X. Finally, spatial self-attention is used to fuse the information to obtain X* (i.e., X'):

[0150]

[0151] Where X' represents the output attention feature, and Q, K, and V represent the query, key, and value matrices in the Attention mechanism, respectively. Specifically, Q = XW Q K=XW K V=XW V W Q W K and W VThis is the corresponding projection matrix, where d is the dimension of the key feature. After calculating the attention, the output attention feature X' is separated to obtain f. r 'and f z '. Due to f r 'It was repeated F times, and the average of the F corresponding results was taken as the final f'. r Finally, the obtained f r 'and f z 'Concatenate the results to obtain the updated f' (f*), and then input it into subsequent model layers for further processing.

[0152] 1.3 Mechanism for Differentiating Reference Information from Generated Content

[0153] Because the actual input z in the technical solution provided in this application t Compared to the original video diffusion model, it has one more frame (i.e., a reference image frame) as its input parameters, therefore the output... An extra frame was also added. Therefore, a simple and intuitive approach is to remove the output corresponding to the reference information r and only calculate the loss for the remaining frames. This method encourages the model to focus on learning customized video generation with a specified subject. However, observations of the final results show that without supervision of the reference information, the model struggles to accurately identify that the reference information r is an image without added noise, leading to instability in the generated results. To address this issue, the training method provided in this application introduces a guidance information recognition loss to supervise the reference information, enabling the model to accurately distinguish between the reference information and the generated video content, thereby improving the quality of the customized generated video. Specifically, during training, at time step t, a reminder noise is added to the reference information r. The calculation method can be shown in the following formula:

[0154]

[0155] Where t' = α∙t, α is a manually set hyperparameter. To prevent additional noise from significantly damaging the reference information, the value of α is set to a small value (e.g., 0.01) to ensure that the cue noise is a slight noise.

[0156] When calculating the loss function, the same loss as in the diffusion model needs to be calculated for the reference information r, where the loss value... The calculation method can be shown in the following formula:

[0157]

[0158] in, ϵ represents the input text information, ε represents the added noise, and t represents the time step of the current diffusion model. For diffusion models, This represents the output of the diffusion model.

[0159] Among them, the loss value The calculation method can be shown in the following formula:

[0160]

[0161] in, ϵ represents the input text information, ε represents the added noise, and t represents the time step of the current diffusion model. For diffusion models, This represents the output of the diffusion model.

[0162] Furthermore, the terminal can determine the sum between the first loss value and the second loss value, and use this sum as the target loss value, that is, the terminal will... As an auxiliary optimization objective, it is combined with the primary objective to guide model training, where the target loss value... The calculation method can be shown in the following formula:

[0163]

[0164] Here, β is a hyperparameter. To avoid interfering with the optimization of the main custom video generation task, β is set to a relatively small value (e.g., 0.1).

[0165] The beneficial effects of the technical solution in this application include:

[0166] The effectiveness of the method was verified through experiments. The effectiveness of the method provided in this application was validated on data from customized portrait generation and customized item tasks using the AnimateDiff SD1.5 model. First, a comparison of customized video generation based on objective metrics was conducted, such as... Figure 6 The image shown is a comparison diagram illustrating the effects of customized character video generation methods. Figure 7 The image shows a comparison of the effects of another customized character video generation method. PhotoMaker's result is based on the SDXL version of AnimateDiff. The method provided in this application outperforms traditional methods in all metrics compared to the same base model. That is, the video generation method provided in this application can generate high-quality, smooth-moving customized videos while better preserving the main features of various characters (including people and objects), and also effectively improves the efficiency of customized video generation. Figure 8 The image shown is a schematic diagram illustrating a video sample of a specified person generated using the method provided in this application. Figure 9 The image shown is a schematic diagram of a video sample of a specified item generated using the method provided in this application.

[0167] It is understandable that the model provided in this application uses AnimateDiff as its base model, which is weaker than the existing DiT architecture's base model. A more powerful base model can be used to further improve the results of customized plan generation. Furthermore, due to the limited size of existing academic datasets, experiments were conducted separately on customized character video generation and customized item video generation. However, if a dataset containing all items could be merged, a model capable of generating general customized items simultaneously could be obtained through a single training iteration.

[0168] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0169] Based on the same inventive concept, this application also provides a video generation apparatus for implementing the video generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video generation apparatus embodiments provided below can be found in the limitations of the video generation method described above, and will not be repeated here.

[0170] In one embodiment, such as Figure 10 As shown, a video generation device is provided, including: an acquisition module 1002, an extraction module 1004, a stitching module 1006, a processing module 1008, and a generation module 1010, wherein:

[0171] The acquisition module 1002 is used to acquire a reference image of the target object and descriptive information for generating the video.

[0172] The extraction module 1004 is used to extract the main features of the reference image and the noise features of each noisy image through a video diffusion model; the video diffusion model is trained based on sample reference images of different categories of objects.

[0173] The stitching module 1006 is used to stitch the main feature with the noise features of each of the noise images to obtain the corresponding stitched features.

[0174] The processing module 1008 is used to perform self-attention processing on the splicing features to obtain attention features.

[0175] The generation module 1010 is used to generate a target video of the target object that conforms to the description information based on the attention features and the encoding features of the description information.

[0176] In one embodiment, the apparatus further includes: an encoding module, configured to encode the reference image and initial noise respectively using an encoder in a diffusion model to obtain an encoded reference image and a noise image of a preset number of frames; a stitching module, further configured to combine the encoded reference image and the noise image of the preset number of frames into a stitched image; an extraction module, further configured to extract mixed image features from the stitched image using a video diffusion model; the video diffusion model being a sub-model in the diffusion model; and separating the main features of the reference image and the noise features of each of the noise images from the mixed image features.

[0177] In one embodiment, the encoding module is further configured to encode the initialization noise using a first variational encoder in the diffusion model to obtain the noise image with a preset number of frames; and to encode the reference image using a second variational encoder in the diffusion model to obtain the encoded reference image.

[0178] In one embodiment, the processing module is further configured to add noise to each of the noise images to obtain a set number of noise-added images; the stitching module is further configured to combine the encoded reference image and the set number of noise-added images to form a stitched image.

[0179] In one embodiment, the stitching module is further configured to stitch the subject features with the noise features of the noise image of the preset number of frames through the spatial self-attention layer in the video diffusion model to obtain the stitched features of the preset number of frames; the processing module is further configured to perform self-attention processing on the stitched features of the preset number of frames through the spatial self-attention layer to obtain the attention features of the preset number of frames.

[0180] In one embodiment, the apparatus further includes: a splitting module, configured to split each of the attention features separately to obtain attention noise features and attention subject features for the preset number of frames; a calculation module, configured to calculate the average value of the attention subject features for the preset number of frames to obtain target attention subject features; a stitching module, further configured to stitch the target attention subject features with each of the attention noise features to obtain updated attention features for the preset number of frames; and a generation module, further configured to generate a target video of the target object that conforms to the description information based on the updated attention features and the encoding features of the description information.

[0181] In one embodiment, the processing module is further configured to process the updated attention features and the encoded features of the description information through a cross-attention layer in the video diffusion model to obtain latent space features of the target number; the apparatus further includes: a filtering module for filtering the latent space features to obtain the latent space features of the preset number of frames; a conversion module for converting the latent space features of the preset number of frames into pixel space features of the preset number of frames through a decoder in the diffusion model; and a generation module for generating a target video of the target object that conforms to the description information based on the pixel space features of the preset number of frames.

[0182] In one embodiment, the acquisition module is further configured to acquire training sample videos, sample reference images, and sample description information; the apparatus further includes: a training module, configured to use the training sample videos, the sample reference images, and the sample description information as training data to train the initial diffusion model to obtain the diffusion model.

[0183] In one embodiment, the apparatus further includes: an encoding module, configured to encode the training sample video and the sample reference image respectively to obtain an encoded video image and an encoded sample reference image; an adding module, configured to add first noise to each of the encoded video images to obtain a noisy video image with a preset number of frames; add second noise to the encoded sample reference image to obtain a noisy reference image; a stitching module is further configured to combine the noisy reference image and the noisy video image with a preset number of frames to form a stitched sample image; and a training module is further configured to use the stitched sample image as input parameters to train an initial diffusion model to obtain the trained diffusion model.

[0184] In one embodiment, the training module is further configured to use the stitched sample image as input parameters to train the initial diffusion model until the target loss value determined based on the first noise and the second noise meets the preset loss condition, thereby obtaining the trained diffusion model.

[0185] In one embodiment, the apparatus further includes: a determining module, configured to determine a first loss value based on the encoded video image, the first noise, the sample description information, and a preset number of video frames output by the initial diffusion model; determine a second loss value based on the encoded sample reference image, the first noise, the sample description information, and a reference frame output by the initial diffusion model; wherein the reference frame is determined based on the second noise, the sample description information, and a time step; determine the sum between the first loss value and the second loss value, and use the sum as the target loss value.

[0186] Each module in the aforementioned video generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0187] In one embodiment, a computer device is provided, which may be a terminal or a server. In this embodiment, the computer device is described as a terminal, and its internal structure diagram is as follows. Figure 11 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a video generation method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0188] Those skilled in the art will understand that Figure 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0189] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0190] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0191] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0193] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0194] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video generation method, characterized in that, The method includes: Obtain a reference image of the target object and descriptive information for generating the video; The main features of the reference image and the noise features of each noisy image are extracted using a video diffusion model; the video diffusion model is trained based on sample reference images of different categories of objects. The main feature is concatenated with the noise features of each of the noise images to obtain the corresponding concatenated features; Self-attention processing is applied to the splicing features to obtain attention features; Based on the attention features and the encoding features of the description information, a target video of the target object that conforms to the description information is generated.

2. The method according to claim 1, characterized in that, The method further includes: The reference image and the initial noise are encoded by the encoder in the diffusion model to obtain the encoded reference image and the noise image of a preset number of frames. The encoded reference image and the noise image of a preset number of frames are combined to form a stitched image; The step of extracting the main features of the reference image and the noise features of each noisy image using a video diffusion model includes: The video diffusion model is used to extract the hybrid image features from the stitched image; the video diffusion model is a sub-model in the diffusion model; from the hybrid image features, the main features of the reference image and the noise features of each of the noise images are separated.

3. The method according to claim 2, characterized in that, The step of encoding the reference image and initial noise respectively using an encoder in a diffusion model to obtain an encoded reference image and a noise image of a preset number of frames includes: The initial noise is encoded by the first variational encoder in the diffusion model to obtain the noise image of a preset number of frames; The reference image is encoded by the second variational encoder in the diffusion model to obtain the encoded reference image.

4. The method according to claim 2, characterized in that, The method further includes: The noise images are subjected to noise addition processing to obtain a set number of noise-added images; The step of assembling a stitched image from the encoded reference image and the noise image of a preset number of frames includes: The encoded reference image and the noise-added image with a preset number of frames are combined to form a stitched image.

5. The method according to claim 2, characterized in that, The step of concatenating the main features with the noise features of each of the noise images to obtain corresponding concatenated features includes: The main features are concatenated with the noise features of the noise image at the preset number of frames through the spatial self-attention layer in the video diffusion model to obtain the concatenated features at the preset number of frames. The process of performing self-attention processing on the spliced ​​features to obtain attention features includes: The spatial self-attention layer performs self-attention processing on the splicing features of the preset number of frames to obtain the attention features of the preset number of frames.

6. The method according to claim 5, characterized in that, The method further includes: Each attention feature is then split into attention noise features and attention subject features for the preset number of frames. The target attention subject features are obtained by averaging the attention subject features of the preset number of frames. The target attention subject feature is concatenated with each of the attention noise features to obtain the updated attention features for the preset number of frames. The step of generating a target video of the target object that conforms to the description information based on the attention features and the encoding features of the description information includes: Based on the updated attention features and the encoded features of the description information, a target video of the target object that conforms to the description information is generated.

7. The method according to claim 6, characterized in that, The step of generating a target video of the target object that conforms to the description information based on the updated attention features and the encoded features of the description information includes: By using the cross-attention layer in the video diffusion model, the updated attention features and the encoded features of the descriptive information are processed to obtain the latent space features of the target quantity; The latent space features are filtered to obtain the latent space features for the preset number of frames; The latent space features of the preset number of frames are converted into pixel space features of the preset number of frames using the decoder in the diffusion model. Based on the pixel space features of the preset frame number, a target video of the target object that conforms to the description information is generated.

8. The method according to claim 1, characterized in that, The method further includes: Acquire training sample videos, sample reference images, and sample description information; The initial diffusion model is trained using the training sample video, the sample reference image, and the sample description information as training data to obtain the diffusion model.

9. The method according to claim 8, characterized in that, The step of training the initial diffusion model using the training sample video, the sample reference image, and the sample description information as training data to obtain the diffusion model includes: The training sample video and the sample reference image are encoded respectively to obtain encoded video images and encoded sample reference images; Add first noise to each of the encoded video images to obtain a noisy video image with a preset number of frames; A second noise is added to the encoded sample reference image to obtain a noisy reference image; The noisy reference image and the noisy video image of a preset number of frames are combined to form a stitched sample image; The stitched sample image is used as input parameters to train the initial diffusion model, resulting in the trained diffusion model.

10. The method according to claim 9, characterized in that, The step of using the stitched sample image as input parameters to train the initial diffusion model to obtain the trained diffusion model includes: The stitched sample image is used as input parameter to train the initial diffusion model until the target loss value determined based on the first noise and the second noise meets the preset loss condition, and the training is stopped to obtain the trained diffusion model.

11. The method according to claim 10, characterized in that, The target loss value determined based on the first noise and the second noise includes: A first loss value is determined based on the encoded video image, the first noise, the sample description information, and a preset number of video frames output by the initial diffusion model. A second loss value is determined based on the coded sample reference image, the first noise, the sample description information, and the reference frame output by the initial diffusion model; wherein the reference frame is determined based on the second noise, the sample description information, and the time step. Determine the sum between the first loss value and the second loss value, and use the sum as the target loss value.

12. A video generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire a reference image of the target object and descriptive information for generating the video; An extraction module is used to extract the main features of the reference image and the noise features of each noisy image using a video diffusion model; the video diffusion model is trained based on sample reference images of different categories of objects. The stitching module is used to stitch the main feature with the noise features of each of the noise images to obtain the corresponding stitched features; The processing module is used to perform self-attention processing on the spliced ​​features to obtain attention features; The generation module is used to generate a target video of the target object that conforms to the description information based on the attention features and the encoding features of the description information.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images