Video generation method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a video generation method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a reference image and a reference video; inputting the reference video into a video motion feature extraction model to obtain a first video motion feature; the first video motion feature is used for representing the difference between the spatial-temporal feature of the reference video and the spatial feature of the reference video; inputting the reference image and the first video motion feature into a pre-trained video generation model to obtain a target video; the target video comprises a main body object in the reference image and a picture moving according to a motion mode of the main body object of the reference video. According to the invention, the personalized video required by the user can be generated based on the pure motion mode features.

Description

Technical Field

[0001] The present disclosure relates to the field of video generation, and in particular to a video generation method, device, electronic device, storage medium, and program product. Background Art

[0002] Personalized video generation technology can generate personalized videos required by users based on static images of the required subject and reference videos containing target motion patterns.

[0003] Traditional personalized video generation techniques can leverage video diffusion models to decompose the personalized video generation task into two stages: subject learning and motion learning. Subject learning leverages text inversion and identity adapter fine-tuning to accurately capture the subject's detailed appearance from a reference image. Motion learning fine-tunes the motion adapter based on a reference video to model the motion pattern of the personalized video.

[0004] However, the motion learning part of traditional personalized video generation technology can only select random frames as decoupling conditions of the subject's appearance, and cannot learn pure motion pattern features. This leads to the problem that traditional personalized video generation technology cannot generate the personalized videos required by users based on pure motion pattern features. Summary of the Invention

[0005] The present disclosure provides a video generation method, apparatus, electronic device, storage medium, and program product to at least address the problem in related art of being unable to generate the desired personalized video based on pure motion pattern features. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a video generation method is provided, including:

[0007] Obtain reference images and reference videos;

[0008] Inputting the reference video into a video motion feature extraction model to obtain a first video motion feature; the first video motion feature is used to characterize the difference between the spatiotemporal feature of the reference video and the spatial feature of the reference video;

[0009] The reference image and the first video motion features are input into a pre-trained video generation model to obtain a target video; the target video includes the main object in the reference image and a picture of the main object moving according to the motion pattern of the reference video.

[0010] In one possible implementation, the video motion feature extraction model includes a first encoder and a second encoder, and inputting the reference video into the video motion feature extraction model to obtain the first video motion feature includes:

[0011] Inputting each frame of the reference video into the first encoder to obtain spatial features of each frame of the reference video; the spatial features are used to represent spatial information of the corresponding video image;

[0012] Inputting the video data of the reference video into the second encoder to obtain the spatiotemporal features of the reference video; the spatiotemporal features are used to represent the spatial information and temporal information of the reference video;

[0013] The first video motion feature is determined according to a difference between the spatiotemporal feature of the reference video and the spatial feature corresponding to each frame of the video image in the reference video.

[0014] In a possible implementation, determining the first video motion feature based on a difference between the spatiotemporal feature of the reference video and a spatial feature corresponding to each frame of the video image in the reference video includes:

[0015] Inputting the spatial features corresponding to each frame of the video image into the feature predictor to obtain the predicted features corresponding to each frame of the video image; the predicted features are used to predict the spatial information corresponding to each frame of the video image in the spatiotemporal features;

[0016] The difference between the spatiotemporal features of the reference video and the predicted features corresponding to each frame of the video image is determined to obtain the first video motion features.

[0017] In one possible implementation, inputting the reference image and the first video motion feature into a pre-trained video generation model to obtain a target video includes:

[0018] Acquiring image coding features of the reference image;

[0019] Inputting the image coding feature and the first video motion feature into the pre-trained video generation model, so as to inject the first video motion feature into the pre-trained video generation model as a control condition;

[0020] The target video is generated by processing the image coding features according to the control conditions through the pre-trained video generation model.

[0021] In one possible implementation, the method further includes:

[0022] Get sample images and sample videos;

[0023] Inputting the sample video into the video motion feature extraction model to obtain a second video motion feature; the second video motion feature is used to characterize the difference between the spatiotemporal feature of the sample video and the spatial feature of the sample video;

[0024] Inputting the sample image and the second video motion features into a video generation model to be trained to obtain a predicted video; the predicted video includes a scene of the main object in the sample image moving according to the motion pattern of the main object in the sample video;

[0025] Determining a model loss of the video generation model to be trained based on a difference between the sample video and the predicted video;

[0026] According to the model loss, the video generation model to be trained is trained until the model loss of the trained video generation model is less than a preset condition, thereby obtaining the pre-trained video generation model.

[0027] In one possible implementation, determining the model loss of the to-be-trained video generation model according to the difference between the sample video and the predicted video includes:

[0028] Inputting the predicted video into the video motion feature extraction model to obtain a third video motion feature of the sample video; the third video motion feature is used to characterize the difference between the spatiotemporal feature of the predicted video and the spatial feature of the predicted video;

[0029] The feature difference information is converted into a model loss of the video generation model through a preset loss function; the feature difference information represents the difference between the third video motion feature and the second video motion feature.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided a video generating apparatus, including:

[0031] A first acquisition unit is configured to acquire a reference image and a reference video;

[0032] A first motion learning unit is configured to input the reference video into a video motion feature extraction model to obtain a first video motion feature; the first video motion feature is used to characterize the difference between the spatiotemporal feature of the reference video and the spatial feature of the reference video;

[0033] The first generation unit is configured to input the reference image and the first video motion features into a pre-trained video generation model to obtain a target video; the target video includes the main object in the reference image and a picture of the main object moving according to the motion pattern of the main object in the reference video.

[0034] In one possible implementation, the video motion feature extraction model includes a first encoder and a second encoder, and the first motion learning unit is specifically configured to input each frame of the video image of the reference video into the first encoder respectively to obtain the spatial features of each frame of the video image in the reference video; the spatial features are used to characterize the spatial information of the corresponding video image; the video data of the reference video is input into the second encoder to obtain the spatiotemporal features of the reference video; the spatiotemporal features are used to characterize the spatial information and time information of the reference video; and the first video motion features are determined based on the difference between the spatiotemporal features of the reference video and the spatial features corresponding to each frame of the video image in the reference video.

[0035] In one possible implementation, the first motion learning unit is specifically configured to input the spatial features corresponding to each frame of the video image into the feature predictor to obtain the predicted features corresponding to each frame of the video image; the predicted features are used to predict the spatial information corresponding to each frame of the video image in the spatiotemporal features; determine the difference between the spatiotemporal features of the reference video and the predicted features corresponding to each frame of the video image to obtain the first video motion features.

[0036] In one possible implementation, the first generation unit is specifically configured to obtain the image coding features of the reference image; input the image coding features and the first video motion features into the pre-trained video generation model to inject the first video motion features into the pre-trained video generation model as a control condition; and process the image coding features according to the control conditions through the pre-trained video generation model to generate the target video.

[0037] In one possible implementation, the apparatus further includes:

[0038] A second acquiring unit is configured to acquire sample images and sample videos;

[0039] A second motion learning unit is configured to input the sample video into the video motion feature extraction model to obtain a second video motion feature; the second video motion feature is used to characterize the difference between the spatiotemporal feature of the sample video and the spatial feature of the sample video;

[0040] a second generating unit configured to input the sample image and the second video motion feature into a video generation model to be trained to obtain a predicted video; the predicted video includes a scene of the main object in the sample image moving according to the motion pattern of the main object in the sample video;

[0041] a loss acquisition unit, configured to determine a model loss of the video generation model to be trained based on a difference between the sample video and the predicted video;

[0042] The model training unit is configured to train the video generation model to be trained according to the model loss until the model loss of the trained video generation model is less than a preset condition, thereby obtaining the pre-trained video generation model.

[0043] In one possible implementation, the loss acquisition unit is specifically configured to input the predicted video into the video motion feature extraction model to obtain a third video motion feature of the sample video; the third video motion feature is used to characterize the difference between the spatiotemporal feature of the predicted video and the spatial feature of the predicted video;

[0044] The feature difference information is converted into a model loss of the video generation model through a preset loss function; the feature difference information represents the difference between the third video motion feature and the second video motion feature.

[0045] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the video generation method as described in the first aspect or any possible implementation of the first aspect.

[0046] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video generation method described in any one of the embodiments of the first aspect.

[0047] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which includes instructions. When the instructions are executed by a processor of an electronic device, the electronic device can execute the video generation method described in any one of the embodiments of the first aspect.

[0048] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: by obtaining a reference image and a reference video; then inputting the reference video into a video motion feature extraction model to obtain a first video motion feature, wherein the first video motion feature is used to characterize the difference between the spatiotemporal features of the reference video and the spatial features of the reference video; finally, inputting the reference image and the first video motion feature into a pre-trained video generation model to obtain a target video, wherein the target video includes the main object in the reference image and a picture moving according to the motion pattern of the main object of the reference video. In this way, the pure motion pattern corresponding to the reference video can be obtained according to the difference between the spatiotemporal features of the reference video and the spatial features of the reference video, thereby realizing the generation of personalized videos that better meet user needs using pure motion pattern features.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0051] Figure 1 The figure is a diagram showing an application environment of a video generation method according to an exemplary embodiment.

[0052] Figure 2 The figure is a flowchart of a video generating method according to an exemplary embodiment.

[0053] Figure 3 The figure is a block diagram of a video generating apparatus according to an exemplary embodiment.

[0054] Figure 4 It is a schematic diagram of a framework of a video generation method according to an exemplary embodiment.

[0055] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment.

[0056] Figure 6 is a block diagram of another electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0058] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0059] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0060] The video generation method provided by the present disclosure can be applied to Figure 1 In the application environment shown. The electronic device 110 obtains a reference image and a reference video; then, the electronic device 110 inputs the reference video into a video motion feature extraction model to obtain a first video motion feature, wherein the first video motion feature is used to characterize the difference between the spatiotemporal features of the reference video and the spatial features of the reference video; finally, the electronic device 110 inputs the reference image and the first video motion feature into a pre-trained video generation model to obtain a target video; wherein the target video includes the main object in the reference image, and a picture of the main object moving according to the motion pattern of the reference video. In actual applications, the electronic device 110 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and can also be implemented using an independent server or a server cluster consisting of multiple servers.

[0061] Figure 2 is a flow chart of a video generation method according to an exemplary embodiment. Figure 2 As shown, the video generation method is used to Figure 1 The electronic device 110 includes the following steps.

[0062] In step S210 , a reference image and a reference video are acquired.

[0063] The reference image refers to an image that can be used to indicate the main object of the target video required by the user.

[0064] The reference video refers to a video that can be used to indicate the motion pattern of the main object of the target video required by the user.

[0065] In a specific implementation, when a user needs to obtain a personalized target video based on a reference image and a reference video, the user can input the reference image and the reference video into an electronic device, and the electronic device then obtains the reference image and the reference video.

[0066] In step S220, the reference video is input into a video motion feature extraction model to obtain a first video motion feature.

[0067] The first video motion feature can be used to characterize the difference between the spatiotemporal feature of the reference video and the spatial feature of the reference video. In practical applications, the first video motion feature can also be named as a first visual feature difference.

[0068] Among them, the video motion feature extraction model can be used to learn the pure motion pattern of the input video based on the difference between the temporal and spatial features of the input video, thereby achieving the decoupling of the video space subject appearance and the temporal space motion pattern of the input video.

[0069] In a specific implementation, the electronic device can input the reference video into a video motion feature extraction model, and the video motion feature extraction model can obtain a first video motion feature that can characterize the pure motion pattern of the reference video based on the difference between the spatiotemporal features and the spatial features of the reference video.

[0070] In step S230, the reference image and the first video motion feature are input into a pre-trained video generation model to obtain a target video.

[0071] The target video includes the main object in the reference image and a picture that moves according to the motion pattern of the main object in the reference video.

[0072] The video generation model can be used to generate a corresponding target video based on an input reference image and video motion features. In practical applications, the video motion features can be feature data containing pure motion patterns, obtained by decoupling the spatial appearance of the subject from the temporal spatial motion pattern of the input reference video. In practical applications, the video generation model can be a video diffusion model.

[0073] In the specific implementation, the video generation model can determine the main object of the target video based on the input reference image, and determine the motion pattern corresponding to the main object of the target video based on the input video motion features, so that the personalized target video required by the user can be generated based on the reference image and reference video input by the user.

[0074] Specifically, based on the reference video input by the user, the electronic device can use the video motion feature extraction model to extract the first video motion feature corresponding to the reference video, and then input the first video motion feature and the reference image input by the user into the pre-trained video generation model, and generate the personalized target video required by the user through the pre-trained video generation model.

[0075] In the above-mentioned video generation method, a reference image and a reference video are obtained; the reference video is then input into a video motion feature extraction model to obtain a first video motion feature, wherein the first video motion feature is used to characterize the difference between the spatiotemporal features of the reference video and the spatial features of the reference video; finally, the reference image and the first video motion feature are input into a pre-trained video generation model to obtain a target video, which includes the main object in the reference image and a picture of the main object moving according to the motion pattern of the reference video. In this way, the pure motion pattern corresponding to the reference video can be obtained based on the difference between the spatiotemporal features of the reference video and the spatial features of the reference video, and then the pure motion pattern features can be used to generate personalized videos that better meet user needs.

[0076] In an exemplary embodiment, a video motion feature extraction model includes a first encoder and a second encoder. A reference video is input into the video motion feature extraction model to obtain a first video motion feature, including: inputting each frame of the reference video into the first encoder respectively to obtain the spatial features of each frame of the reference video; inputting the video data of the reference video into the second encoder to obtain the spatiotemporal features of the reference video; and determining the first video motion feature based on the difference between the spatiotemporal features of the reference video and the spatial features corresponding to each frame of the reference video.

[0077] The spatial features can be used to represent the spatial information of the corresponding video image.

[0078] Among them, spatiotemporal features can be used to characterize the spatial information and temporal information of the reference video.

[0079] In a specific implementation, the electronic device can input each frame of the reference video into a first encoder to obtain the spatial features of each frame of the reference video. The electronic device can then input the video data of the reference video into a second encoder to obtain the spatiotemporal features of the reference video. The electronic device can then determine the first video motion features based on the difference between the spatiotemporal features of the reference video and the spatial features corresponding to each frame of the reference video. In practical applications, the first encoder and / or the second encoder can be a visual encoder. In some examples, the first encoder can be named a per-frame spatial encoder, for example, and the second encoder can be named a spatiotemporal encoder, for example.

[0080] The technical solution of this embodiment is to extract the spatial features corresponding to each frame of the reference video through a first encoder, and to extract the overall spatiotemporal features of the reference video through a second encoder, and then obtain the first video motion features based on the difference between the spatiotemporal features of the reference video and the spatial features corresponding to each frame of the reference video, thereby achieving effective extraction of pure motion patterns in the reference video.

[0081] In an exemplary embodiment, a first video motion feature is determined based on the difference between the spatiotemporal features of a reference video and the spatial features corresponding to each frame of video image in the reference video, including: inputting the spatial features corresponding to each frame of video image into a feature predictor to obtain a predicted feature corresponding to each frame of video image; and determining the difference between the spatiotemporal features of the reference video and the predicted features corresponding to each frame of video image to obtain the first video motion feature.

[0082] Among them, the prediction features can be used to predict the spatial information corresponding to each frame of video image in the spatiotemporal features.

[0083] In a specific implementation, the electronic device inputs the spatial features corresponding to each video frame into a feature predictor to obtain the predicted features corresponding to each video frame. The electronic device then determines the difference between the spatiotemporal features of the reference video and the predicted features corresponding to each video frame to obtain the first video motion features. In practical applications, the feature predictor can also be named a per-frame feature predictor, for example.

[0084] The technical solution of this embodiment obtains the predicted features corresponding to each frame of video image through a feature predictor, and then determines the first video motion features corresponding to the reference video based on the difference between the spatiotemporal features of the reference video and the predicted features corresponding to each frame of video image, thereby achieving the decoupling of the video space subject appearance and the temporal space motion pattern of the reference video, and then effectively extracting the pure motion pattern in the reference video.

[0085] In an exemplary embodiment, a reference image and a first video motion feature are input into a pre-trained video generation model to obtain a target video, including: obtaining image coding features of the reference image; inputting the image coding features and the first video motion feature into the pre-trained video generation model to inject the first video motion feature into the pre-trained video generation model as a control condition; and processing the image coding features according to the control condition by the pre-trained video generation model to generate the target video.

[0086] In a specific implementation, the electronic device obtains the image coding features of the reference image; then, the electronic device inputs the image coding features and the first video motion features into a pre-trained video generation model to inject the first video motion features into the pre-trained video generation model as a control condition; and then, the pre-trained video generation model processes the image coding features according to the control conditions to generate the target video required by the user.

[0087] Specifically, the electronic device can first convert the input reference image into a feature vector through the encoder of a pre-trained video generation model (such as a video diffusion model) to obtain image coding features. Then, a random noise video with parameters such as frame rate and resolution that meet the specifications of the target video is created as the initial state of the target video. Then, based on the noise video, the electronic device gradually performs denoising, so that each denoising step incorporates the image coding features, so that the main object of the generated target video matches the main object of the reference image. Because the control conditions of the pre-trained video generation model include the first video motion feature extracted from the reference video, the motion pattern corresponding to the main object of the target video generated by the model will match the motion pattern corresponding to the main object of the reference video. For example, if the reference image is a landscape, the generated target video will dynamically change around the landscape, and this dynamic change matches the motion pattern corresponding to the main object of the reference video input by the user.

[0088] The technical solution of this embodiment is to inject the first video motion feature extracted from the reference video as a control condition into a pre-trained video generation model, and use the pre-trained video generation model to generate a target video according to the image coding features corresponding to the reference image, so that the motion pattern of the generated target video is more consistent with the motion pattern of the reference video input by the user, thereby enabling the video generation model to better meet the user's personalized video customization needs.

[0089] In an exemplary embodiment, the video generation method also includes: obtaining a sample image and a sample video; inputting the sample video into a video motion feature extraction model to obtain a second video motion feature; inputting the sample image and the second video motion feature into a video generation model to be trained to obtain a predicted video; determining the model loss of the video generation model to be trained based on the difference between the sample video and the predicted video; training the video generation model to be trained based on the model loss until the model loss of the trained video generation model is less than a preset condition, thereby obtaining a pre-trained video generation model.

[0090] The sample image refers to an image that can be used to indicate the main object of the predicted video.

[0091] The sample video refers to a video that can be used to indicate the motion pattern of the main object of the predicted video.

[0092] The second video motion feature can be used to characterize the difference between the spatiotemporal feature of the sample video and the spatial feature of the sample video. In practical applications, the second video motion feature can also be named as a second visual feature difference.

[0093] The predicted video may include a main object in the sample image and a picture in which the main object moves according to the motion pattern of the main object in the sample video.

[0094] In a specific implementation, the electronic device obtains a sample image and a sample video; then, the electronic device inputs the sample video into a video motion feature extraction model to obtain a second video motion feature; then, the sample image and the second video motion feature are input into a video generation model to be trained to obtain a predicted video; then, based on the difference between the sample video and the predicted video, the model loss of the video generation model to be trained is determined; then, based on the model loss, the video generation model to be trained is trained until the model loss of the trained video generation model is less than a preset condition, thereby obtaining a pre-trained video generation model. In practical applications, the video generation model to be trained can be a video diffusion model to be trained. In practical applications, the extraction process of the second video motion feature is similar to the extraction process of the first video motion feature in the above embodiment, and will not be repeated here.

[0095] The technical solution of this embodiment determines the model loss of the video generation model to be trained by comparing the difference between the sample video and the predicted video. Based on the model loss, the video generation model to be trained is trained to obtain a pre-trained video generation model. This approach effectively trains a video generation model capable of generating personalized videos required by users.

[0096] In an exemplary embodiment, the model loss of the video generation model to be trained is determined based on the difference between the sample video and the predicted video, including: inputting the predicted video into a video motion feature extraction model to obtain a third video motion feature of the sample video; and converting the feature difference information into the model loss of the video generation model through a preset loss function.

[0097] The third video motion feature can be used to characterize the difference between the spatiotemporal feature of the predicted video and the spatial feature of the predicted video. In practical applications, the third video motion feature can also be named as a third visual feature difference.

[0098] The feature difference information may represent the difference between the third video motion feature and the second video motion feature.

[0099] In a specific implementation, the electronic device can input the predicted video into a video motion feature extraction model to obtain a third video motion feature of the sample video. The electronic device then converts the feature difference information into a model loss for the video generation model using a preset loss function. In actual applications, the extraction process for the third video motion feature is similar to the extraction process for the first and second video motion features in the above embodiment and will not be further described here.

[0100] The technical solution of this embodiment can convert the feature difference information corresponding to the third video motion feature and the second video motion feature into the model loss of the video generation model through a preset loss function, so that a video generation model that can generate a video that is more in line with the user's personalized needs can be obtained based on the model loss training.

[0101] To facilitate understanding by those skilled in the art, Figure 3 A processing flow chart of a video generation method is provided as an example. Figure 3 As shown, taking the video generation model as the video diffusion model as an example, the video motion feature extraction model extracts the pure video motion features of the reference video by means of difference, and then injects the video motion features as control conditions into the video generation model, so that the video diffusion model outputs a generated video that meets the user's personalized needs.

[0102] In practical applications, such as Figure 3 As shown, the video motion feature extraction model can use each frame spatial encoder to extract the spatial features of each frame of the reference video for the input reference video ( Figure 3t is the time) to obtain the spatial feature set V of each frame of the video image, and at the same time use the spatiotemporal encoder to extract the spatiotemporal features z of the reference video as a whole; then, the per-frame feature predictor P of the video motion feature extraction model can use the spatial encoder to extract per-frame representations from each frame of V, and independently calculate the best approximation of the temporal aggregation representation z for each frame; then, the predicted features h output by the per-frame feature predictor P are subtracted from z using the removal operator to form the appearance bottleneck layer of the video motion feature extraction model, and the action-rich control latent variable δV is obtained through the appearance bottleneck layer, that is, the video motion features (also called visual feature differences) that represent the pure motion pattern of the reference video are obtained.

[0103] Furthermore, the electronic device can input the reference video and the generated video into the video motion feature extraction model at the same time, calculate the video motion features corresponding to the reference video and the generated video respectively, and then obtain the corresponding feature loss (Feature Loss) based on the video motion features corresponding to the reference video and the generated video. Based on the feature loss, the video motion features corresponding to the reference video and the generated video can be approximated, thereby making the motion pattern of the generated video more similar to the motion pattern of the reference video.

[0104] It can be understood that the video generation method disclosed in the above-mentioned embodiments can effectively decouple the spatial appearance of the video from the temporal motion pattern characteristics of the video, allowing the video generation model to learn a purer representation of the video motion pattern, thereby making the motion pattern of the user-customized target video more similar to the motion pattern of the reference video. In addition, the video generation method disclosed in the present invention directly understands the motion pattern based on the reference video, so that when the user customizes the video, no other triggering conditions (such as text input, etc.) are required. As a result, the video generation method disclosed in the present invention can customize any motion pattern according to user needs, effectively improving the versatility of the method.

[0105] It should be understood that although Figure 2 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0106] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, please refer to the description of other method embodiments.

[0107] Figure 4 FIG. 1 is a block diagram of a video generation device according to an exemplary embodiment. Figure 4 , the apparatus 400 comprises:

[0108] A first acquiring unit 410 is configured to acquire a reference image and a reference video;

[0109] The first motion learning unit 420 is configured to input the reference video into a video motion feature extraction model to obtain a first video motion feature; the first video motion feature is used to represent the difference between the spatiotemporal feature of the reference video and the spatial feature of the reference video;

[0110] The first generation unit 430 is configured to input the reference image and the first video motion features into a pre-trained video generation model to obtain a target video; the target video includes the main object in the reference image and a picture of the main object moving according to the motion pattern of the main object in the reference video.

[0111] In an exemplary embodiment, the video motion feature extraction model includes a first encoder and a second encoder, and the first motion learning unit 420 is specifically configured to input each frame of the video image of the reference video into the first encoder respectively to obtain the spatial features of each frame of the video image in the reference video; the spatial features are used to characterize the spatial information of the corresponding video image; the video data of the reference video is input into the second encoder to obtain the spatiotemporal features of the reference video; the spatiotemporal features are used to characterize the spatial information and time information of the reference video; and the first video motion features are determined based on the difference between the spatiotemporal features of the reference video and the spatial features corresponding to each frame of the video image in the reference video.

[0112] In an exemplary embodiment, the first motion learning unit 420 is specifically configured to input the spatial features corresponding to each frame of the video image into the feature predictor to obtain the predicted features corresponding to each frame of the video image; the predicted features are used to predict the spatial information corresponding to each frame of the video image in the spatiotemporal features; determine the difference between the spatiotemporal features of the reference video and the predicted features corresponding to each frame of the video image to obtain the first video motion features.

[0113] In an exemplary embodiment, the first generation unit 430 is specifically configured to obtain the image coding features of the reference image; input the image coding features and the first video motion features into the pre-trained video generation model to inject the first video motion features into the pre-trained video generation model as a control condition; and process the image coding features according to the control conditions through the pre-trained video generation model to generate the target video.

[0114] In an exemplary embodiment, the apparatus 400 further includes:

[0115] A second acquiring unit is configured to acquire sample images and sample videos;

[0116] A second motion learning unit is configured to input the sample video into the video motion feature extraction model to obtain a second video motion feature; the second video motion feature is used to characterize the difference between the spatiotemporal feature of the sample video and the spatial feature of the sample video;

[0117] a second generating unit configured to input the sample image and the second video motion feature into a video generation model to be trained to obtain a predicted video; the predicted video includes a scene of the main object in the sample image moving according to the motion pattern of the main object in the sample video;

[0118] a loss acquisition unit, configured to determine a model loss of the video generation model to be trained based on a difference between the sample video and the predicted video;

[0119] The model training unit is configured to train the video generation model to be trained according to the model loss until the model loss of the trained video generation model is less than a preset condition, thereby obtaining the pre-trained video generation model.

[0120] In an exemplary embodiment, the loss acquisition unit is specifically configured to input the predicted video into the video motion feature extraction model to obtain a third video motion feature of the sample video; the third video motion feature is used to characterize the difference between the spatiotemporal feature of the predicted video and the spatial feature of the predicted video;

[0121] The feature difference information is converted into a model loss of the video generation model through a preset loss function; the feature difference information represents the difference between the third video motion feature and the second video motion feature.

[0122] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0123] Figure 5 1 is a block diagram illustrating an electronic device 500 for executing the video generation method described in each of the above method embodiments, according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0124] Reference Figure 5 The electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0125] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0126] The memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0127] The power supply assembly 506 provides power to the various components of the electronic device 500. The power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0128] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the electronic device 500 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0129] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0130] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0131] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor assembly 514 can also detect changes in the position of the electronic device 500 or components of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the device 500, and temperature changes of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0132] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0133] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0134] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by the processor 520 of the electronic device 500 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0135] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by the processor 520 of the electronic device 500 to implement the above method.

[0136] Figure 6 1 is a block diagram of an electronic device 600 for executing the video generation method in each of the above method embodiments according to an exemplary embodiment. For example, the electronic device 600 may be a server. Figure 6 The electronic device 600 includes a processing component 620, which further includes one or more processors, and a memory resource represented by a memory 622 for storing instructions executable by the processing component 620, such as an application. The application stored in the memory 622 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 620 is configured to execute the instructions to perform the above method.

[0137] The electronic device 600 may further include a power supply component 624 configured to perform power management of the electronic device 600, a wired or wireless network interface 626 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 628. The electronic device 600 may operate based on an operating system stored in the memory 622, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0138] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 622 including instructions. The instructions can be executed by a processor of the electronic device 600 to perform the above method. The storage medium can be a computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0139] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by a processor of the electronic device 600 to implement the above method.

[0140] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. can also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0141] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0142] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that: The method comprises: Obtain reference images and reference videos; Inputting the reference video into a video motion feature extraction model to obtain a first video motion feature; the first video motion feature is used to characterize the difference between the spatiotemporal feature of the reference video and the spatial feature of the reference video; The reference image and the first video motion features are input into a pre-trained video generation model to obtain a target video; the target video includes the main object in the reference image and a picture of the main object moving according to the motion pattern of the reference video.

2. The video generation method according to claim 1, wherein: The video motion feature extraction model includes a first encoder and a second encoder, and inputting the reference video into the video motion feature extraction model to obtain the first video motion feature includes: Inputting each frame of the reference video into the first encoder to obtain spatial features of each frame of the reference video; the spatial features are used to represent spatial information of the corresponding video image; Inputting the video data of the reference video into the second encoder to obtain the spatiotemporal features of the reference video; the spatiotemporal features are used to represent the spatial information and temporal information of the reference video; The first video motion feature is determined according to a difference between the spatiotemporal feature of the reference video and the spatial feature corresponding to each frame of the video image in the reference video.

3. The video generation method according to claim 2, characterized in that The determining the first video motion feature according to a difference between the spatiotemporal feature of the reference video and the spatial feature corresponding to each frame of the video image in the reference video includes: Inputting the spatial features corresponding to each frame of the video image into a feature predictor to obtain predicted features corresponding to each frame of the video image; the predicted features are used to predict the spatial information corresponding to each frame of the video image in the spatiotemporal features; The difference between the spatiotemporal features of the reference video and the predicted features corresponding to each frame of the video image is determined to obtain the first video motion features.

4. The video generation method according to claim 1, wherein: The step of inputting the reference image and the first video motion feature into a pre-trained video generation model to obtain a target video includes: Acquiring image coding features of the reference image; Inputting the image coding feature and the first video motion feature into the pre-trained video generation model, so as to inject the first video motion feature into the pre-trained video generation model as a control condition; The target video is generated by processing the image coding features according to the control conditions through the pre-trained video generation model.

5. The video generation method according to claim 1, wherein: The method further comprises: Get sample images and sample videos; Inputting the sample video into the video motion feature extraction model to obtain a second video motion feature; the second video motion feature is used to characterize the difference between the spatiotemporal feature of the sample video and the spatial feature of the sample video; Inputting the sample image and the second video motion features into a video generation model to be trained to obtain a predicted video; the predicted video includes a scene of the main object in the sample image moving according to the motion pattern of the main object in the sample video; Determining a model loss of the video generation model to be trained based on a difference between the sample video and the predicted video; According to the model loss, the video generation model to be trained is trained until the model loss of the trained video generation model is less than a preset condition, thereby obtaining the pre-trained video generation model.

6. The video generation method according to claim 5, characterized in that The determining, based on the difference between the sample video and the predicted video, the model loss of the video generation model to be trained, includes: Inputting the predicted video into the video motion feature extraction model to obtain a third video motion feature of the sample video; the third video motion feature is used to characterize the difference between the spatiotemporal feature of the predicted video and the spatial feature of the predicted video; The feature difference information is converted into a model loss of the video generation model through a preset loss function; the feature difference information represents the difference between the third video motion feature and the second video motion feature.

7. A video generating device, characterized in that: include: A first acquisition unit is configured to acquire a reference image and a reference video; A first motion learning unit is configured to input the reference video into a video motion feature extraction model to obtain a first video motion feature; the first video motion feature is used to characterize the difference between the spatiotemporal feature of the reference video and the spatial feature of the reference video; A first generating unit is configured to input the reference image and the first video motion feature into a pre-trained video generation model to obtain a target video; The target video includes a main object in the reference image and a picture in which the main object moves according to a motion pattern of the main object in the reference video.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video generating method according to any one of claims 1 to 6.

10. A computer program product comprising instructions, characterized in that: When the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the video generating method according to any one of claims 1 to 6.

Images