Graphics video model training method and device, equipment and storage medium

Abstract

One or more embodiments of the invention provide a graph-to-video model training method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first sample video, and extracting an image frame from the first sample video; acquiring multiple types of exercise intensity data of the first sample video obtained by performing exercise intensity evaluation on the first sample video by a trained exercise estimation model; inputting the image frames and the multiple types of exercise intensity data into a preset image-generated video model, and performing calculation by the image-generated video model based on the image frames and the multiple types of exercise intensity data to generate a corresponding video; and determining training loss based on the generated video, and determining to complete training for the map-to-video model after completing adjustment of model parameters for the map-to-video model according to the loss.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a training method, apparatus, device and storage medium for an image-generated video model. Background Art

[0002] Image to Video refers to the use of artificial intelligence technology to convert static images into dynamic video content. It combines the latest advances in image processing, computer vision, and deep learning, making it possible to create short videos with coherent movements from a single image. This technology is usually based on a deep learning model, which learns the patterns in the image and infers the appropriate movement to make the image move.

[0003] Image-generated video technology makes the originally static images vivid and interesting by automatically adding elements such as animation and transition, so it is widely used in entertainment, advertising marketing, and personal creation. For example, it can be used to create animations, game cutscenes, or movie special effects, or enable companies to quickly create product display videos to increase the appeal of content, or allow users to easily convert personal photos into interesting video clips for social media sharing, etc. In practical applications, how to realize image-generated video and ensure the quality of the generated video has become a problem of great concern. Summary of the invention

[0004] One or more embodiments of the present application provide the following technical solutions:

[0005] The present application provides a method for training a graph-generated video model, the method comprising:

[0006] Acquire a first sample video, and extract image frames from the first sample video;

[0007] Acquire multiple types of motion intensity data of the first sample video by performing motion intensity evaluation on the first sample video using the trained motion estimation model;

[0008] Inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a corresponding video;

[0009] A training loss is determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

[0010] The present application also provides a training device for an image-generated video model, the device comprising:

[0011] A first acquisition module, acquiring a first sample video and extracting image frames from the first sample video;

[0012] A second acquisition module is configured to acquire multiple types of motion intensity data of the first sample video obtained by performing motion intensity evaluation on the first sample video using a trained motion estimation model;

[0013] A generating module, inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a corresponding video;

[0014] The training module determines the training loss based on the generated video, and after completing the adjustment of the model parameters of the image-generated video model according to the loss, determines that the training of the image-generated video model is completed.

[0015] The present application also provides an electronic device, comprising:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] The processor implements the steps of any of the above methods by running the executable instructions.

[0019] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0020] In the above technical scheme, first, a sample video can be obtained and image frames can be extracted from the sample video. The motion intensity of the sample video can be evaluated by a trained motion estimation model to obtain multiple types of motion intensity data of the first sample video. Then, the image frame and the multiple types of motion intensity data can be input into a preset image-generated video model, and the image-generated video model performs calculations based on the image frame and the multiple types of motion intensity data to generate a corresponding video. Finally, the training loss can be determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

[0021] By adopting the above method, on the one hand, different types of motion in the video can be distinguished, and the decoupled multiple types of motion intensity data can be injected into the image-generated video model for training, thereby improving the learning effect of the image-generated video model based on the decoupled motion intensity data and ensuring the accuracy of the motion-related knowledge learned by the image-generated video model; on the other hand, using an additional motion estimation model to evaluate the motion intensity of the video can ensure the accuracy of the various types of motion intensity data of the evaluated video, helping the image-generated video model to better understand and generate motion in the video. In this way, when using the trained image-generated video model for image-generated video, the motion intensity of the generated video can be better controlled, and a higher-quality video can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The following is a description of the accompanying drawings required for describing the exemplary embodiments, wherein:

[0023] Figure 1 is a schematic diagram of an image-generated video system shown in an exemplary embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of a training and use process of a graph-generated video model shown in an exemplary embodiment of the present application;

[0025] Figure 3 is a flowchart of a method for training an image-generated video model shown in an exemplary embodiment of the present application;

[0026] Figure 4 is a flow chart of a method for training and using a motion estimation model shown in an exemplary embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a structure of a device shown in an exemplary embodiment of the present application;

[0028] Figure 6 It is a block diagram of a training device for an image-generated video model shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. On the contrary, they are only examples consistent with some aspects of one or more embodiments of the present application.

[0030] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0031] Image-generated video refers to the use of artificial intelligence technology to convert static images into dynamic video content. It combines the latest advances in image processing, computer vision, and deep learning, making it possible to create short videos with coherent movements from a single image. This technology is usually based on a deep learning model, which learns the patterns in the image and infers the appropriate movement to make the image move.

[0032] In the field of image-generated videos, motion modeling capability is an important factor affecting the quality of generated videos. Motion modeling capability refers to the ability to understand and generate natural and smooth object or scene motion in the process of converting static images into dynamic videos. It involves learning the laws of object movement in image sequences and how to predict or generate new frames based on these laws, thereby creating coherent and realistic video content.

[0033] Specifically, motion modeling capabilities usually include motion estimation, motion prediction, deformation and animation, scene consistency, and spatiotemporal consistency. Among them, motion estimation refers to identifying the position changes of objects between two or more frames. Motion prediction refers to predicting the content of the next picture based on the existing frames of images, which requires the model to understand the movement trend of the object. Deformation and animation refer to the fact that for non-rigid objects (such as faces and animals), it is also necessary to consider the changes in their shape and make static pictures "alive" through appropriate algorithms. Scene consistency refers to ensuring that not only the individual elements of the generated video look natural, but also the entire scene maintains consistency and logic. Spatiotemporal consistency refers to ensuring that the movements on the timeline of the generated video are smooth and the relationship between objects in space is reasonable.

[0034] In the current field of image-generated videos, the lack of motion modeling capabilities is still a key issue that needs to be solved. The lack of motion modeling capabilities is mainly reflected in the following aspects:

[0035] Motion intensity control: Motion intensity usually refers to the speed of movement, the significance of movement, etc. Existing image-generated video technology has limitations in controlling the motion intensity of objects or scenes in the generated video. Ideally, it is hoped that natural, smooth and expected animation effects will be automatically generated based on the input content. However, current methods often find it difficult to accurately adjust the degree of such movement, resulting in the generated video being either too stiff and lacking in dynamics, or too exaggerated in motion and not in line with the physical laws of the real world.

[0036] Motion score calculation: In order to evaluate and optimize the motion performance in the video, a "motion score" is usually defined to measure the intensity of the motion in the video. The current method is to infer the intensity of the motion by analyzing the differences between adjacent frames in the video. However, this method is easily affected by non-motion factors, such as changes in lighting conditions, changes in background color, or updates to the content itself. These external factors can lead to misjudgment of the actual motion situation, thus affecting the quality of the final output video.

[0037] Distraction processing: Although the frame-by-frame difference method is simple and intuitive, it faces many challenges in practical applications. For example, when the image sequence contains rapidly changing colors or patterns, these visual changes may be mistakenly interpreted as important motion features; similarly, when encountering scene switching or camera movement, it is difficult to distinguish which are viewpoint changes caused by camera movement and which are real object displacements.

[0038] Realistic Perception Match: Humans are very sensitive to the real feeling of motion in videos and can easily identify unnatural or incoherent movements. In contrast, current AI systems still have a lot of room for improvement in this regard. They may not accurately capture subtle changes in movement or cannot handle complex multi-object interaction scenes well, which limits the authenticity and attractiveness of the generated videos.

[0039] In the prior art, for a model for image-generated videos (which may be referred to as an image-generated video model), it can use a first sample video to learn the motion in the video during the model training process, and apply the learned motion-related knowledge to the motion intensity control of the generated video during the subsequent use of the model.

[0040] However, on the one hand, the motion in the video is more complicated in practical applications. For example, the motion in a video includes both object motion and camera motion. The image-generated video model cannot decouple different types of motion during the learning process, which will lead to low accuracy of motion-related knowledge learned by the image-generated video model. On the other hand, the error in the way the motion score of the video is calculated based on the difference between frames will also affect the image-generated video model's understanding and generation of motion in the video. This makes the image-generated video model have poor control over the motion intensity of the generated video, making it difficult to guarantee the quality of the generated video, and it is also easy to produce motion mixing.

[0041] One or more embodiments of the present application provide a method for training an image-generated video model. In this technical solution, a sample video can be first obtained and an image frame can be extracted from the sample video. A motion estimation model that has been trained can also be used to evaluate the motion intensity of the sample video to obtain multiple types of motion intensity data for the first sample video. The image frame and the multiple types of motion intensity data can then be input into a preset image-generated video model, which performs calculations based on the image frame and the multiple types of motion intensity data to generate a corresponding video. Finally, the training loss can be determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

[0042] By adopting the above method, on the one hand, different types of motion in the video can be distinguished, and the decoupled multiple types of motion intensity data can be injected into the image-generated video model for training, thereby improving the learning effect of the image-generated video model based on the decoupled motion intensity data and ensuring the accuracy of the motion-related knowledge learned by the image-generated video model; on the other hand, using an additional motion estimation model to evaluate the motion intensity of the video can ensure the accuracy of the various types of motion intensity data of the evaluated video, helping the image-generated video model to better understand and generate motion in the video. In this way, when using the trained image-generated video model for image-generated video, the motion intensity of the generated video can be better controlled, and a higher-quality video can be generated.

[0043] Please refer to Figure 1 , Figure 1 It is a schematic diagram of an image-generated video system shown in an exemplary embodiment of the present application.

[0044] like Figure 1 As shown, the above-mentioned image-generated video system may include a server and at least one client accessing the server through any type of wired or wireless network.

[0045] Among them, the above-mentioned server can correspond to a server including an independent physical host, or it can be a server cluster composed of multiple independent physical hosts; or it can correspond to a virtual server, cloud server, etc. carried by a host cluster.

[0046] The above-mentioned client may correspond to terminal devices such as smart phones, tablet computers, laptops, desktop computers, PCs (Personal Computers), PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), smart car devices or game consoles.

[0047] Users can use the image-generated video service provided by the image-generated video system through the client; the client and the server can implement user-oriented image-generated video services through data interaction between each other.

[0048] For example, the client can output a corresponding user interface to the user, so that the user can perform operations such as inputting images for generating videos, inputting description texts and various types of motion intensity data (e.g., scores reflecting motion intensity) for assisting the image-generated video, and selecting image-generated video services in the user interface, so as to use the image-generated video services provided by the image-generated video system. The client can send the images, description texts, and various types of motion intensity data input by the user to the server, which takes the description texts and various types of motion intensity data as given conditions, generates a video based on the image under the guidance of the conditions, and outputs the generated video to the user, that is, returns the generated video to the client, and the client displays the generated video to the user through the user interface for the user to view, thereby realizing a user-oriented image-generated video service.

[0049] Specifically, a picture-generated video model may be deployed on the server, and the picture-generated video system may be based on the picture-generated video model, and the picture-generated video model may execute specific steps in the picture-generated video task.

[0050] Correspondingly, the server may also be equipped with a motion estimation model for training the image-generated video model, and the motion estimation model is used to evaluate various types of motion intensity data of the video. When training the image-generated video model, the image-generated video model may be trained based on a sample video, a description text corresponding to the sample video, and various types of motion intensity data of the sample video evaluated by the motion estimation model.

[0051] Alternatively, a sample video library may also be deployed on the server, and each record in the sample video library includes a sample video, a description text corresponding to the sample video, and various types of motion intensity data of the sample video evaluated by the motion estimation model. When training the image-generated video model, the image-generated video model may be trained based on the corresponding sample videos, description texts, and various types of motion intensity data in the sample database.

[0052] In addition, other functional components or functional subsystems may be deployed on the above-mentioned server, such as feature extraction components, database retrieval components, prompt text generation components, etc. These components or subsystems may work together with the image-generated video model deployed on the server to jointly realize the image-generated video service.

[0053] It should be noted that the training of the above-mentioned image-generated video model can be completed directly on the above-mentioned server; or, the training of the above-mentioned image-generated video model can be completed on other electronic devices with certain computing capabilities, and the trained image-generated video model can be deployed on the above-mentioned server; this application does not impose any special restrictions on this. The above-mentioned motion estimation model and the image-generated video model can be deployed on the same electronic device or on different electronic devices, and this application does not impose any special restrictions on this.

[0054] Please refer to Figure 2 , Figure 2 It is a schematic diagram of the training and use process of a graph-generated video model shown in an exemplary embodiment of the present application.

[0055] like Figure 2 As shown, when training the image-generated video model, a video as a sample (which may be referred to as a first sample video) may be first obtained, and image frames may be extracted from the first sample video.

[0056] The trained motion estimation model can be used to evaluate the motion intensity of the first sample video to obtain various types of motion intensity data of the first sample video. In this case, various types of motion intensity data of the first sample video output by the trained motion estimation model can be obtained.

[0057] When the image frame is extracted from the first sample video and multiple types of motion intensity data of the first sample video are obtained, the image frame and the multiple types of motion intensity data can be input into the image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a video corresponding to the image frame and the multiple types of motion intensity data. That is, the generated video is a video generated based on the image frame, and the motion intensity in the video matches the multiple types of motion intensity data.

[0058] When a video generated by the above-mentioned image-generated video model is obtained, the training loss can be determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

[0059] After completing the training of the above-mentioned image-generated video model, the trained image-generated video model can be used to perform image-generated video tasks. When using the trained image-generated video model, an image (which can be called a target image) and preset motion intensity data for generating a video can be obtained, and the target image and the motion intensity data are input into the trained image-generated video model, and the image-generated video model performs calculations based on the target image and the motion intensity data to generate a corresponding video.

[0060] Alternatively, when training the image-generated video model, text describing the motion in the first sample video (referred to as sample text) may also be obtained. Accordingly, the image frame, the sample text, and the multiple types of motion intensity data may be input into the image-generated video model, and the image-generated video model may perform calculations based on the image frame, the sample text, and the multiple types of motion intensity data to generate a corresponding video.

[0061] When using the trained image-generated video model, an image for generating a video (which may be called a target image), a text for describing the motion in the video to be generated at this time (which may be called a target text), and preset motion intensity data may be obtained. The target image, the target text, and the motion intensity data are input into the trained image-generated video model, and the image-generated video model performs calculations based on the target image, the target text, and the motion intensity data to generate a corresponding video.

[0062] Please refer to Figure 3 , Figure 3 It is a flowchart of a method for training an image-generated video model shown as an exemplary embodiment of the present application.

[0063] like Figure 3 As shown, the training method of the above-mentioned image-generated video model may include the following steps:

[0064] Step 302: Obtain a first sample video, and extract image frames from the first sample video.

[0065] In this embodiment, a graph-generated video model may be pre-set, wherein the graph-generated video model may be a generative model based on deep learning, such as a variational autoencoder (VAE), a generative adversarial network (GANs), a diffusion model, and the like.

[0066] The generative model aims to learn the distribution of data so that it can generate new data that is similar to the training data. Therefore, based on the input image, the generative model can generate an image sequence containing multiple images similar to the image, which can be regarded as a video.

[0067] In some embodiments, the above-mentioned image-derived video model may be a diffusion model.

[0068] The diffusion model is a generative model whose basic idea is to transform data into noise through a step-by-step denoising process, and then learn how to reverse this process to recover the original data from pure noise. The working principle of the diffusion model includes a forward process and a reverse process; in the forward process, starting from the original data, Gaussian noise is gradually added, and each step makes the data closer to completely random noise; in the reverse process, the goal is to start from pure noise and gradually recover the original data through a series of denoising steps. Diffusion models can be used to generate new data, usually by extracting data from a noise distribution and transforming it into new data that looks like it came from the original data set through a reverse process.

[0069] When training the above-mentioned image-generated video model, a video as a sample (which may be referred to as a first sample video) may be first obtained, and image frames may be extracted from the first sample video.

[0070] It should be noted that when extracting image frames from the above-mentioned first sample video, one or more image frames may be randomly extracted, or image frames may be extracted according to certain extraction rules (for example, extracting every N frames), and this application does not impose any special restrictions on this.

[0071] Step 304: obtaining multiple types of motion intensity data of the first sample video by performing motion intensity evaluation on the first sample video using the trained motion estimation model.

[0072] In this embodiment, the trained motion estimation model can be used to perform motion intensity evaluation on the first sample video to obtain various types of motion intensity data of the first sample video. In this case, various types of motion intensity data of the first sample video output by the trained motion estimation model can be obtained.

[0073] In practical applications, the motion in the video can generally include object motion and scene motion. For example, in a road video captured by a fixed camera, the scene does not move, but cars, pedestrians, etc. as objects will move; in a road video captured by a mobile camera, not only cars, pedestrians, etc. as objects will move, but the scene will also move. Based on this, in some embodiments, the motion intensity data can be divided into two types: object motion intensity data and lens motion intensity data. Of course, more other types of motion intensity data can also be divided according to actual conditions and needs, and this application does not impose any special restrictions on this.

[0074] In the above case, the multiple types of motion intensity data of the first sample video may include the object motion intensity data and the lens motion intensity data of the first sample video. That is, after the motion estimation model performs motion intensity estimation on the first sample video, it may distinguish the object motion intensity and the lens motion intensity of the first sample video and output the object motion intensity data and the lens motion intensity data of the first sample video.

[0075] In some embodiments, in order to reduce the complexity of data processing to a certain extent, the above-mentioned motion intensity data can specifically be a score reflecting the motion intensity. For example, the above-mentioned object motion intensity data can be an object motion intensity score in the range of [0,10], which focuses on the motion of the subject in the video, such as the motion of a car, the motion of a windmill, etc. The above-mentioned lens motion intensity data can be a lens motion intensity score in the range of [0,10], which focuses on the lens movement when shooting the video, such as left and right panning, push-pull zooming, etc.

[0076] In some embodiments, the first sample video is input into the trained motion estimation model, and the motion estimation model performs motion intensity evaluation on the first sample video to obtain multiple types of motion intensity data of the first sample video. This can be performed online or offline.

[0077] Specifically, when the first sample video is obtained, the first sample video can be input into the trained motion estimation model in real time, and the motion estimation model evaluates the motion intensity of the first sample video to obtain multiple types of motion intensity data of the first sample video.

[0078] Alternatively, in order to speed up the acquisition of various types of motion intensity data of the above-mentioned first sample video, thereby improving the training efficiency of the image-generated video model, multiple sample videos including the first sample video can be pre-input into the above-mentioned trained motion estimation model, and the motion estimation model performs motion intensity evaluation on each sample video to obtain various types of motion intensity data of each sample video; and each sample video and its various types of motion intensity data can be stored together to facilitate the acquisition of various types of motion intensity data of each sample video.

[0079] Step 306: Input the image frame and the multiple types of motion intensity data into a preset image-derived video model, and the image-derived video model performs calculations based on the image frame and the multiple types of motion intensity data to generate corresponding videos.

[0080] In this embodiment, when the image frame is extracted from the first sample video and multiple types of motion intensity data of the first sample video are obtained, the image frame and the multiple types of motion intensity data can be input into the image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a video corresponding to the image frame and the multiple types of motion intensity data. That is, the generated video is a video generated based on the image frame, and the motion intensity in the video matches the multiple types of motion intensity data.

[0081] In some embodiments, when the image frame and the multiple types of motion intensity data are input into the image-derived video model, and the image-derived video model performs calculations based on the image frame and the multiple types of motion intensity data to generate a corresponding video, the image frame and the multiple types of motion intensity data can be input into the image-derived video model, and the image-derived video model generates a feature vector corresponding to the image frame, and generates feature vectors corresponding to the multiple types of motion intensity data. Subsequently, the image-derived video model can splice these feature vectors, and further calculate based on the spliced ​​feature vectors to generate a video corresponding to the image frame and the multiple types of motion intensity data.

[0082] In practical applications, the model structure of the above-mentioned image-generated video model may include a linear layer, and the linear layer performs the feature extraction task. That is, after the above-mentioned image frame and the above-mentioned multiple types of motion intensity data are input into the image-generated video model, the linear layer in the image-generated video model may first map the image frame into a corresponding feature vector, and map the multiple types of motion intensity data into corresponding feature vectors respectively.

[0083] It should be noted that when the above-mentioned image-generated video model is a diffusion model, since an integer T needs to be randomly generated when training the diffusion model, as one of the training conditions, the above-mentioned feature vector can be added to the feature vector corresponding to the integer T, and further calculation can be performed based on the added feature vector.

[0084] By splicing the feature vectors corresponding to multiple types of motion intensity data onto other feature vectors, and having the graph-based video model perform further calculations based on the spliced ​​feature vectors to generate videos, different types of motion intensity data can be better decoupled, further improving the learning effect of the graph-based video model based on the decoupled motion intensity data.

[0085] Step 308: Determine a training loss based on the generated video, and after adjusting the model parameters of the image-generated video model according to the loss, determine that the training of the image-generated video model is completed.

[0086] In this embodiment, when a video generated by the above-mentioned image-generated video model is obtained, the training loss can be determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

[0087] It should be noted that the selection and calculation method of the above training loss will vary depending on the specific task objectives, model architecture, and the type of data used. For the above-mentioned image-generated video model, the following training losses can be selected:

[0088] Reconstruction Loss: Use reconstruction loss to measure the difference between the generated video frame and the target real video frame. For example, you can calculate the sum of the squares of the pixel value differences between the generated frame and the target frame (i.e., mean square error), or calculate the sum of the absolute values ​​of the pixel value differences between the generated frame and the target frame (i.e., absolute error), or use a pre-trained deep network (such as VGG-16) to extract features and compare the differences between the generated frame and the target frame in these feature spaces (i.e., perceptual loss).

[0089] Motion Consistency Loss: In order to ensure the motion coherence between the generated video frames, motion consistency loss may be introduced. This can be achieved by calculating the optical flow field between adjacent frames or directly comparing the differences between frames. For example, the L1 or L2 norm can be used to measure whether the changes between the previous and next two frames are in line with expectations.

[0090] Temporal Smoothness Loss: Encourages the generated video to be smoother in the temporal dimension and avoid sudden changes.

[0091] Structural Similarity Index (SSIM) loss: used to evaluate the structural similarity between two images.

[0092] Of course, you can also combine multiple loss functions to comprehensively optimize model performance. For example, a complete loss function may be a weighted sum of the above losses, where the weight coefficient of each loss term can be adjusted according to actual conditions and needs.

[0093] In practical applications, the model parameters of the above-mentioned image-generated video model can be adjusted automatically according to the above-mentioned loss based on the preset model parameter adjustment rules.

[0094] Alternatively, the loss may be output to the user manually, and the user may adjust the model parameters of the image-generated video model according to the loss.

[0095] It should be noted that after completing the training of the above-mentioned image-generated video model, the trained image-generated video model can be used to perform image-generated video tasks.

[0096] Specifically, in some embodiments, when using the trained image-generated video model, an image (which may be referred to as a target image) and preset motion intensity data for generating a video may be obtained, and the target image and the motion intensity data may be input into the trained image-generated video model, and the image-generated video model may perform calculations based on the target image and the motion intensity data to generate a corresponding video.

[0097] In practical applications, in order to make the generated video better meet the needs of users and improve the user experience of using the image-generated video service, users can be allowed to control the content, style and even emotional tone of the generated video through text description. In this case, the image used to generate the video, the description text used to describe the movement in the video to be generated, and various types of motion intensity data can be input into the above-mentioned image-generated video model, and the image-generated video model uses the description text and the motion intensity data as given conditions, and generates a video based on the image under the guidance of the conditions.

[0098] Specifically, in some embodiments, when training the above-mentioned image-generated video model, text (which may be referred to as sample text) for describing the motion in the above-mentioned first sample video may also be obtained. Accordingly, when the above-mentioned image frame and the above-mentioned multiple types of motion intensity data are input into the image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a corresponding video, the image frame, the sample text and the multiple types of motion intensity data may be input into the image-generated video model, and the image-generated video model performs calculation based on the image frame, the sample text and the multiple types of motion intensity data to generate a corresponding video.

[0099] In some embodiments, when using the trained image-generated video model described above, an image for generating a video (which may be referred to as a target image), a text for describing the motion in the video that needs to be generated at this time (which may be referred to as a target text), and preset motion intensity data may be obtained, and the target image, the target text, and the motion intensity data may be input into the trained image-generated video model, and the image-generated video model may perform calculations based on the target image, the target text, and the motion intensity data to generate a corresponding video.

[0100] In the above technical scheme, first, a sample video can be obtained and image frames can be extracted from the sample video. The motion intensity of the sample video can be evaluated by a trained motion estimation model to obtain multiple types of motion intensity data of the first sample video. Then, the image frame and the multiple types of motion intensity data can be input into a preset image-generated video model, and the image-generated video model performs calculations based on the image frame and the multiple types of motion intensity data to generate a corresponding video. Finally, the training loss can be determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

[0101] By adopting the above method, on the one hand, different types of motion in the video can be distinguished, and the decoupled multiple types of motion intensity data can be injected into the image-generated video model for training, thereby improving the learning effect of the image-generated video model based on the decoupled motion intensity data and ensuring the accuracy of the motion-related knowledge learned by the image-generated video model; on the other hand, using an additional motion estimation model to evaluate the motion intensity of the video can ensure the accuracy of the various types of motion intensity data of the evaluated video, helping the image-generated video model to better understand and generate motion in the video. In this way, when using the trained image-generated video model for image-generated video, the motion intensity of the generated video can be better controlled, and a higher-quality video can be generated.

[0102] Please refer to Figure 4 , Figure 4 It is a flowchart of a method for training and using a motion estimation model shown in an exemplary embodiment of the present application.

[0103] like Figure 4 As shown, the training and use method of the motion estimation model may include the following steps:

[0104] Step 402: Acquire a second sample video; wherein the second sample video is labeled with the multiple types of motion intensity data.

[0105] Step 404: Based on the second sample video, perform supervised training on the motion estimation model.

[0106] In this embodiment, in order to use the above motion estimation model to evaluate various types of motion intensity data of a video, it is also necessary to first train the motion estimation model.

[0107] Specifically, first, a video as a sample (referred to as a second sample video) may be obtained. For any second sample video, the second sample video is annotated with multiple types of motion intensity data of the second sample video as multiple labels of the second sample video.

[0108] Subsequently, supervised training may be performed on the motion estimation model based on the second sample video with labels.

[0109] In some embodiments, in order to improve the model effect of the above-mentioned motion estimation model, a multi-layer dynamic convolution structure (Multi-layer Dynamic Convolution Structure) and a multi-layer perceptron (Multi-layerPerceptron, MLP) can be used; and the multi-layer dynamic convolution structure can be pre-trained on the ActivityNet dataset first, and then the above-mentioned second sample data can be used to train the motion estimation model as a whole.

[0110] The ActivityNet dataset contains a large number of YouTube videos covering a variety of daily activities, sports events, entertainment activities, etc., and each video is finely labeled, including the start time, end time and corresponding activity category of the action. By pre-training the above multi-layer dynamic convolutional structure, the subsequent training speed of the above motion estimation model can be accelerated and the training efficiency of the above motion estimation model can be improved.

[0111] Step 406: Input the first sample video into the trained motion estimation model, and use the motion estimation model to evaluate the motion intensity of the first sample video to obtain multiple types of motion intensity data of the first sample video.

[0112] In this embodiment, after the training of the motion estimation model is completed, the trained motion estimation model can be used to perform the motion intensity assessment task. That is, the first sample video can be input into the trained motion estimation model, and the motion estimation model performs motion intensity assessment on the first sample video to obtain the above-mentioned multiple types of motion intensity data of the first sample video.

[0113] Specifically, in some embodiments, the first sample video can be input into the trained motion estimation model, and the motion estimation model can identify the motion trajectory in the first sample video, and perform motion intensity evaluation on the first sample video based on the motion trajectory to obtain multiple types of motion intensity data of the first sample video.

[0114] For example, if the motion trajectory of an object is identified in the above video, the object motion intensity data of the video can be evaluated based on the speed and intensity of the changes in the motion trajectory; and for the motion trajectory of a background point identified in the above video, the lens motion intensity data of the video can be evaluated based on the speed and intensity of the changes in the motion trajectory.

[0115] In practical applications, the constructed motion estimation model may also use other methods to evaluate the motion intensity of the video, and this application does not impose any special restrictions on this.

[0116] Corresponding to the embodiments of the aforementioned method for training an image-generated video model, the present application also provides embodiments of an apparatus for training an image-generated video model.

[0117] Please refer to Figure 5 , Figure 5 It is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508 and a non-volatile memory 510, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0118] Please refer to Figure 6 , Figure 6 It is a block diagram of a training device for an image-generated video model shown in an exemplary embodiment of the present application.

[0119] The training device of the above-mentioned image-generated video model can be applied to Figure 5 The device shown is used to implement the technical solution of this application.

[0120] The training device of the image-generated video model may include:

[0121] A first acquisition module 602 acquires a first sample video and extracts image frames from the first sample video;

[0122] A second acquisition module 604 is configured to acquire multiple types of motion intensity data of the first sample video obtained by performing motion intensity evaluation on the first sample video using the trained motion estimation model;

[0123] A generating module 606, inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a corresponding video;

[0124] The training module 608 determines a training loss based on the generated video, and after adjusting the model parameters of the image-generated video model according to the loss, determines that the training of the image-generated video model is completed.

[0125] In some embodiments, the first sample video is input into the trained motion estimation model, and the motion estimation model performs motion intensity evaluation on the first sample video to obtain multiple types of motion intensity data of the first sample video, including:

[0126] The first sample video is input into the trained motion estimation model, the motion estimation model identifies the motion trajectory in the first sample video, and performs motion intensity evaluation on the first sample video according to the motion trajectory to obtain multiple types of motion intensity data of the first sample video.

[0127] In some embodiments, the motion estimation model includes a multi-layer dynamic convolutional structure and a multi-layer perceptron.

[0128] In some embodiments, the apparatus further comprises:

[0129] A third acquisition module acquires a second sample video; wherein the second sample video is annotated with the multiple types of motion intensity data;

[0130] The second training module performs supervised training on the motion estimation model based on the second sample video.

[0131] In some embodiments, the inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and the image-generated video model performing calculations based on the image frame and the multiple types of motion intensity data to generate a corresponding video, includes:

[0132] The image frame and the multiple types of motion intensity data are input into a preset image-generated video model, and the image-generated video model generates a feature vector corresponding to the image frame, and generates feature vectors corresponding to the multiple types of motion intensity data respectively, and splices the feature vectors, and further calculates based on the spliced ​​feature vectors to generate a corresponding video.

[0133] In some embodiments, the multiple types of motion intensity data include object motion intensity data and lens motion intensity data.

[0134] In some embodiments, the image-derived video model is a diffusion model.

[0135] In some embodiments, the apparatus further comprises:

[0136] A fourth acquisition module, acquiring a target image to be generated and preset motion intensity data;

[0137] The generation module is also used to input the target image and the motion intensity data into the trained image-generated video model, and the image-generated video model performs calculations based on the image and the motion intensity data to generate a corresponding video.

[0138] In some embodiments, the apparatus further comprises:

[0139] A fifth acquisition module, acquiring a sample text for describing the motion in the first sample video;

[0140] The step of inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and having the image-generated video model perform calculations based on the image frame and the multiple types of motion intensity data to generate corresponding videos, comprises:

[0141] The image frame, the sample text and the multiple types of motion intensity data are input into a preset image-generated video model, and the image-generated video model performs calculations based on the image frame, the sample text and the multiple types of motion intensity data to generate a corresponding video.

[0142] In some embodiments, the apparatus further comprises:

[0143] A sixth acquisition module, which acquires a target image to be generated, a target text for describing the motion in the generated video, and preset motion intensity data;

[0144] The generation module is also used to input the target image, the target text and the motion intensity data into the trained image-derived video model, and the image-derived video model performs calculations based on the target image, the target text and the motion intensity data to generate a corresponding video.

[0145] For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0146] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0147] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0148] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0149] Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0150] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0151] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" refers to and includes any or all possible combinations of one or more associated listed items.

[0153] The description of the terms "one embodiment", "some embodiments", "example", "specific example" or "an implementation method" etc. used in one or more embodiments of the present application means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic description of these terms is not necessarily for the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without contradicting each other.

[0154] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0155] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0156] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A method for training a graph-generated video model, the method comprising: Acquire a first sample video, and extract image frames from the first sample video; Acquire multiple types of motion intensity data of the first sample video by performing motion intensity evaluation on the first sample video using the trained motion estimation model; Inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a corresponding video; A training loss is determined based on the generated video, and after the model parameters of the image-generated video model are adjusted according to the loss, it is determined that the training of the image-generated video model is completed.

2. According to the method of claim 1, the first sample video is input into the trained motion estimation model, and the motion estimation model performs motion intensity evaluation on the first sample video to obtain multiple types of motion intensity data of the first sample video, including: The first sample video is input into the trained motion estimation model, the motion estimation model identifies the motion trajectory in the first sample video, and performs motion intensity evaluation on the first sample video according to the motion trajectory to obtain multiple types of motion intensity data of the first sample video.

3. According to the method of claim 1, the motion estimation model includes a multi-layer dynamic convolution structure and a multi-layer perceptron.

4. The method according to claim 1, further comprising: Acquire a second sample video; wherein the second sample video is annotated with the multiple types of motion intensity data; Based on the second sample video, supervised training is performed on the motion estimation model.

5. The method according to claim 1, wherein the inputting the image frame and the multiple types of motion intensity data into a preset image-derived video model, and the image-derived video model performing calculations based on the image frame and the multiple types of motion intensity data to generate a corresponding video, comprises: The image frame and the multiple types of motion intensity data are input into a preset image-generated video model, and the image-generated video model generates a feature vector corresponding to the image frame, and generates feature vectors corresponding to the multiple types of motion intensity data respectively, and splices the feature vectors, and further calculates based on the spliced ​​feature vectors to generate a corresponding video. 6 . The method according to claim 1 , wherein the multiple types of motion intensity data include object motion intensity data and lens motion intensity data.

7. According to the method of claim 1, the image-derived video model is a diffusion model.

8. The method according to claim 1, further comprising: Acquire a target image to be generated and preset motion intensity data; The target image and the motion intensity data are input into the trained image-derived video model, and the image-derived video model performs calculations based on the image and the motion intensity data to generate a corresponding video.

9. The method according to claim 1, further comprising: Obtaining sample text for describing motion in the first sample video; The step of inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and having the image-generated video model perform calculations based on the image frame and the multiple types of motion intensity data to generate corresponding videos, comprises: The image frame, the sample text and the multiple types of motion intensity data are input into a preset image-generated video model, and the image-generated video model performs calculations based on the image frame, the sample text and the multiple types of motion intensity data to generate a corresponding video.

10. The method according to claim 9, further comprising: Acquire a target image to be generated, a target text for describing the motion in the generated video, and preset motion intensity data; The target image, the target text and the motion intensity data are input into the trained image-derived video model, and the image-derived video model performs calculations based on the target image, the target text and the motion intensity data to generate a corresponding video.

11. A training device for a graph-generated video model, the device comprising: A first acquisition module, acquiring a first sample video and extracting image frames from the first sample video; A second acquisition module is configured to acquire multiple types of motion intensity data of the first sample video obtained by performing motion intensity evaluation on the first sample video using a trained motion estimation model; A generating module, inputting the image frame and the multiple types of motion intensity data into a preset image-generated video model, and the image-generated video model performs calculation based on the image frame and the multiple types of motion intensity data to generate a corresponding video; The training module determines the training loss based on the generated video, and after completing the adjustment of the model parameters of the image-generated video model according to the loss, determines that the training of the image-generated video model is completed.

12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images