Training method of video generation model and video generation method

Abstract

The invention provides a training method of a video generation model and a video generation method, and the training method can comprise the steps: marking a plurality of marked video frames in a sample video, and obtaining a marked video; inputting the marked video and the sample behavior description information of the digital object in the sample video into a diffusion model to obtain a target sample video; wherein the marked video frame is kept unchanged in the training process of the diffusion model; and if the target sample video is matched with the sample video, determining the diffusion model as a video generation model. The video generation model trained by the training method can generate videos with coherent behaviors and more accurate information.

Description

Technical Field

[0001] The present application relates to the field of video production, and more specifically, to a video generation model training method and a video generation method. Background Art

[0002] Digital objects usually refer to virtual objects created through computer technology and digital media, which can be three-dimensional models with physical behavior, expressions and voice.

[0003] In some application scenarios, the dynamic behavior of digital objects can be presented through videos, for example, presenting the coherent process of a digital person changing from smiling to crying.

[0004] In the related art, videos can be generated through diffusion models. However, during the training process of the diffusion model, each sample video frame will be subjected to noise addition and noise reduction processing. As a result, when the diffusion model actually generates a video of a digital object, it can only perform simple constraints on the posture or appearance of the digital object through partial information in the sample video frame, resulting in poor consistency in the behavior of the digital object in the generated video and low accuracy. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a video generation model training method and a video generation method, so as to generate a video with coherent behavior and relatively accurate information through the video generation model.

[0006] In the first aspect, the embodiment of the present application provides a training method for a video generation model, the method comprising: marking a plurality of marked video frames in a sample video to obtain a marked video; inputting the marked video and the sample behavior description information of the digital object in the sample video into a diffusion model to obtain a target sample video; the diffusion model comprises a noise adding module and a noise reduction module, the noise adding module is used to perform noise adding processing on the marked video, and the noise reduction module is used to perform noise reduction processing on the marked video after noise adding to obtain a target sample video with coherent behavior; wherein the marked video frame remains unchanged during the training process of the diffusion model; if the target sample video matches the sample video, the diffusion model is determined as a video generation model. In this way, the marked video frame can remain unchanged during the training process of the diffusion model, so that the diffusion model can use the real and clear discontinuous frames as the diffusion basis, learn the continuity features between the discontinuous frames, so that the trained video generation model can generate a video with coherent behavior and relatively accurate information.

[0007] Optionally, marking a plurality of marked video frames in the sample video to obtain the marked video includes: randomly marking a preset proportion of marked video frames in the sample video to obtain the marked video. In this way, by marking the video frames in the sample video in a random manner, the credibility of the marked video is improved to a certain extent, making the video generation model more suitable for application in actual scenarios.

[0008] Optionally, randomly marking a preset ratio of marked video frames in the sample video to obtain the marked video includes: dividing the sample video into a preset number of sample video segments; randomly marking multiple marked video frames from multiple sample video segments to obtain the marked video; wherein the number of marked video frames in the first and last sample video segments is greater than the number of marked video frames in other sample video segments. In this way, by controlling the number of marked video frames of video frames at different positions, the video generation model can generate a more accurate and coherent video given the first and last conditional frames, so that the video generation model can better meet actual needs to a certain extent.

[0009] Optionally, performing noise processing on the marked video by the noise adding module includes: performing noise processing on each sample video frame in the marked video by the noise adding module; and eliminating the noise adding result of each marked video frame in the marked video. In this way, on the one hand, the marked video frame can be kept unchanged during the training process of the diffusion model, and on the other hand, the diffusion model can use a unified noise adding method to process all sample video frames, making the noise adding process more consistent and easier to manage.

[0010] Optionally, the noise reduction module performs noise reduction processing on the noisy marked video, including: performing noise reduction processing on each sample video frame in the marked video by the noise reduction module; and eliminating the noise reduction result of each marked video frame in the marked video. In this way, on the one hand, the marked video frame can be kept unchanged during the training process of the diffusion model, and on the other hand, the diffusion model can use a unified noise reduction method to process all sample video frames, making the noise reduction process more consistent and easier to manage.

[0011] In the second aspect, the embodiment of the present application provides a video generation method, which includes: obtaining conditional frames and behavior description information in the video to be generated; the conditional frames are used to present the target behavior of the digital object; the target video is generated according to the conditional frames and the behavior description information by the noise reduction module in the video generation model; the video generation model is trained by the method described in the first aspect. In this way, since only a small number of conditional frames are used, the target video is generated in combination with the behavior description information, thereby reducing the production time, reducing the production complexity, and saving the production cost.

[0012] In the third aspect, the embodiment of the present application provides a training device for a video generation model, the device comprising: a marking module, used to mark multiple marked video frames in a sample video to obtain a marked video; an input module, used to input the marked video and the sample behavior description information of the digital object in the sample video into a diffusion model to obtain a target sample video; the diffusion model comprises a noise addition module and a noise reduction module, the noise addition module is used to perform noise addition processing on the marked video, and the noise reduction module is used to perform noise reduction processing on the marked video after noise addition to obtain a target sample video with coherent behavior; wherein the marked video frame remains unchanged during the training process of the diffusion model; a determination module is used to determine the diffusion model as a video generation model if the target sample video matches the sample video. In this way, the marked video frame can remain unchanged during the training process of the diffusion model, so that the diffusion model can use the real and clear discontinuous frames as the diffusion basis, learn the continuity features between the discontinuous frames, so that the trained video generation model can generate a video with coherent behavior and relatively accurate information.

[0013] In a fourth aspect, an embodiment of the present application provides a video generation device, which includes: an acquisition module for acquiring conditional frames and behavior description information in a video to be generated; the conditional frames are used to present the target behavior of a digital object; a generation module for generating a target video according to the conditional frames and the behavior description information through a noise reduction module in a video generation model; the video generation model is trained by the method described in the first aspect. In this way, since only a small number of conditional frames are used, the target video is generated in combination with the behavior description information, thereby reducing production time, reducing production complexity, and saving production costs.

[0014] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect or the second aspect are executed.

[0015] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method provided in the first aspect or the second aspect are performed.

[0016] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method described in the first aspect or the second aspect is executed.

[0017] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a method for training a video generation model provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a video generation method provided in an embodiment of the present application;

[0021] Figure 3 A structural block diagram of a video generation model training device provided in an embodiment of the present application;

[0022] Figure 4 A structural block diagram of a video generation device provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of an electronic device for executing a training method for a video generation model or a video generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0025] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0026] It should be noted that the embodiments in this application or the technical features in the embodiments may be combined if there is no conflict.

[0027] In the related art, there is a problem that when a video of a digital object is generated using a diffusion model, the behavior of the digital object in the video is poorly coherent and has low accuracy; in order to improve this situation, the present application provides a method for training a video generation model; further, the method selects some frames in a sample video not to participate in the noise addition and noise reduction processes of the diffusion model, so that the diffusion model can use non-continuous frames as the basis for diffusion and learn the continuity features between non-continuous frames, so as to obtain a video generation model that can generate videos with coherent behavior and relatively accurate information.

[0028] In some application scenarios, the training method of the video generation model can be applied to terminal devices such as smartphones and computers; in other application scenarios, it can also be applied to server devices such as servers, server clusters or cloud platforms. For the convenience of writing, this application is described below using the application to the server as an example.

[0029] The defects existing in the solutions in the above-mentioned related technologies are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed in the embodiments of the present invention below for the above-mentioned problems should all be the contributions made by the inventor to the present invention during the process of the present invention.

[0030] Please refer to Figure 1 , which shows a flow chart of a method for training a video generation model provided by an embodiment of the present application. Figure 1 As shown, the training method of the video generation model includes the following steps 101 to 103.

[0031] Step 101, marking a plurality of marked video frames in a sample video to obtain a marked video;

[0032] The digital objects mentioned above can be regarded as digital images corresponding to people, animals, etc. The digital objects can present different postures or appearances.

[0033] In some application scenarios, the server may, for example, pre-traverse each video frame in the sample video, and then select and mark the video frames that meet the preset requirements, thereby obtaining a marked video. In these application scenarios, the preset requirements may, for example, be video frames that perform preset actions or make preset expressions.

[0034] In these application scenarios, the server may, for example, mark an identifier in the screened video frame to obtain a marked video. The identifier may, for example, be a number, a letter, or a combination thereof.

[0035] Furthermore, the server may also use marking tools such as CVAT (Computer Vision Annotation Tool, an open source tool for image and video data annotation), LabelImg (a graphic image annotation tool), etc. to mark the filtered video frames, thereby obtaining a marked video including visually marked frames.

[0036] Step 102, inputting the sample behavior description information of the digital objects in the labeled video and the sample video into a diffusion model to obtain a target sample video; the diffusion model includes a noise adding module and a noise reducing module, the noise adding module is used to perform noise adding processing on the labeled video, and the noise reducing module is used to perform noise reducing processing on the labeled video after noise adding, to obtain a target sample video with coherent behavior; wherein the labeled video frame remains unchanged during the training process of the diffusion model;

[0037] The sample behavior description information may include, for example, positive description information such as "blinking", "shaking head", "smiling", etc., and may also include negative description information such as "no obvious changes in hands", "not blinking", etc.

[0038] The working process of the above diffusion model may include a diffusion process (ie, a noise adding process: adding noise to a sample video) and a reverse diffusion process (ie, a noise reduction process: restoring original data from a noisy sample video).

[0039] In some application scenarios, the server can pre-set the timestep of the diffusion model, and then determine the number of times the video frames in the marked video are processed based on the timestep. When the timestep increases by 1 step, a noise addition operation is performed, and when the timestep decreases by 1 step, a noise reduction operation is performed. For example, if it takes 1000 steps to add noise to the video frames in the marked video to white noise, then timestep can be set to 1000, and then the diffusion model can gradually perform 1000 noise addition operations on each video frame. In addition, the diffusion model also needs 1000 steps to perform noise reduction on the noisy video frames to gradually restore them to their original state. At this time, 1000 noise reduction operations can be performed on each video frame.

[0040] Furthermore, since the denoising module can restore the noisy video frame to its original state, and the noisy video frame is blurred, the denoising process can be regarded as a process of filling in the missing information in the video frame.

[0041] In addition, since the labeled video frames remain unchanged during the training process of the diffusion model, it can be regarded as that the diffusion model does not perform noise addition and noise reduction processing on the labeled video frames, so they always remain in their original state.

[0042] Therefore, the denoising module of the diffusion model can fill in the missing video frame information between frames based on the marked video frames that maintain the original state and combine the sample behavior description information, thereby achieving the effect of filling the gaps between frames and obtaining the target sample video with coherent behavior.

[0043] Furthermore, since the marked video frames always remain in their original state, the diffusion model fills the gaps between frames with real and clear video frames, thereby improving the accuracy of the target sample video.

[0044] Step 103: If the target sample video matches the sample video, the diffusion model is determined as a video generation model.

[0045] In some application scenarios, the server can, for example, determine the average pixel value error of each video frame in the target sample video and the average pixel value error of each video frame in the sample video. If the difference between these two average pixel value errors is less than a preset error threshold, it can be determined that the target sample video matches the sample video.

[0046] In some other application scenarios, the server may also determine the number of training times, and if the number of training times is greater than a preset threshold, it may also be determined that the target sample video matches the sample video.

[0047] Furthermore, if the two match, it can be determined that the diffusion model has converged, and thus the converged diffusion model can be determined as the video generation model.

[0048] In this implementation, the marked video frames can remain unchanged during the training process of the diffusion model, so that the diffusion model can use real and clear discontinuous frames as the diffusion basis and learn the continuity features between discontinuous frames, so that the trained video generation model can generate videos with coherent behavior and relatively accurate information.

[0049] In some application scenarios, before the above step 101, the server may, for example, pre-process the sample video. Specifically, if the size of the sample video is 512×512×100, the server may perform dimensionality reduction processing on it through a variational encoder to obtain a 32×32×100 dimensionality reduction video, and then cut it into video blocks that match the diffusion model input (for example, cut into 8×8×5 video blocks and 4×4×20 video blocks), and then each block may be straightened to obtain a 16×20 token vector, and then for each video block, its position information and time information are concatenated with the corresponding token vector to obtain a 16×22 long vector.

[0050] Then, the server can concatenate the long vector, sample behavior description information, and time step into a vector, and then combine it with the label information of the labeled video frame as the input of the diffusion model. In this way, to a certain extent, the diffusion model can focus more on learning the local features of the video frame and improve its accuracy.

[0051] In some optional implementations, the step 101 of marking a plurality of marked video frames in the sample video to obtain the marked video includes: randomly marking a preset proportion of marked video frames in the sample video to obtain the marked video.

[0052] The above preset ratio is also the preset ratio of the marked video frames to all the video frames in the sample video, which may be 50%, 40% or the like, for example.

[0053] In this implementation, the server marks the video frames in the sample video in a random manner, which improves the credibility of the marked video to a certain extent, making the video generation model more suitable for application in actual scenarios.

[0054] In some optional implementations, the above-mentioned randomly marking a preset proportion of marked video frames in the sample video to obtain the marked video includes: first dividing the sample video into a preset number of sample video segments; then, randomly marking multiple marked video frames from multiple sample video segments to obtain the marked video; wherein the number of marked video frames in the first and last sample video segments is respectively greater than the number of marked video frames in other sample video segments.

[0055] The above-mentioned preset number may include, for example, 3, 5, etc.

[0056] It should be noted that, since the starting frame and the ending frame of the video to be generated are often given when the video generation model is actually applied, the sample video can be divided into multiple sample video segments, and then more marked video frames are randomly marked in the first and last two sample video segments, and fewer marked video frames are randomly marked in other sample video segments.

[0057] For example, if the preset ratio is 50%, the server can divide the sample video into 5 sample video segments (i.e., the first segment, the front segment, the middle segment, the back segment, and the last segment), and then randomly mark multiple marked video frames in each segment according to the ratio of [15% (i.e., %75x20%), 5% (i.e., 25%x20%), 10% (i.e., 50%x20%), 5% (i.e., 25%x20%), 15% (i.e., 75%x20%)].

[0058] In this implementation, by controlling the number of marked video frames at different positions, the video generation model can generate a more accurate and coherent video given the first and last two conditional frames, which makes the video generation model more in line with actual needs to a certain extent.

[0059] In some optional implementations, performing noise processing on the marked video by the noise adding module includes: performing noise processing on each sample video frame in the marked video by the noise adding module; and eliminating the noise adding result of each marked video frame in the marked video.

[0060] That is, the server can directly perform noise addition processing on all sample video frames in the marked video, and then eliminate the noise addition results of the marked video frames.

[0061] In some application scenarios, for example, the noise addition result of the marked video frame may be multiplied by 0 to achieve a cancellation effect.

[0062] In this implementation, all sample video frames are first denoised and then the denoised results of the marked video frames are eliminated. In this way, the marked video frames can be kept unchanged during the training of the diffusion model, and the diffusion model can use a unified denoising method to process all sample video frames, making the denoising process more consistent and easier to manage.

[0063] In some optional implementations, the denoising module performs denoising on the noisy marked video, including: performing denoising on each sample video frame in the marked video by the denoising module; and eliminating the denoising result of each marked video frame in the marked video.

[0064] Similar to the above-mentioned noise adding process, the server can directly perform noise reduction processing on all sample video frames in the marked video, and then eliminate the noise reduction results of the marked video frames.

[0065] In some application scenarios, for example, the noise reduction result of the marked video frame may be multiplied by 0 to achieve a noise elimination effect.

[0066] In this implementation, all sample video frames are first subjected to denoising, and then the denoising results of the marked video frames are eliminated. In this way, on the one hand, the marked video frames can remain unchanged during the training process of the diffusion model, and on the other hand, the diffusion model can use a unified denoising method to process all sample video frames, making the denoising process more consistent and easier to manage.

[0067] It should be noted that the server may also set a judgment mechanism of the diffusion model during the noise addition or noise reduction process. Specifically, when the diffusion model is adding noise or reducing noise, for example, it may determine whether the current video frame is a marked video frame. If so, it performs noise addition or noise reduction processing on it; if not, it skips the video frame so as not to perform noise addition or noise reduction processing on it. In this way, the computing power of the diffusion model can be saved to a certain extent, and its processing efficiency can be improved.

[0068] See also Figure 2 , which shows a flow chart of a video generation method provided by an embodiment of the present application. Figure 2 As shown, the video generation method includes the following steps 201 to 202.

[0069] Step 201, obtaining conditional frames and behavior description information in the video to be generated; the conditional frames are used to present the target behavior of the digital object.

[0070] The above-mentioned conditional frames are video frames whose original appearance remains unchanged during the video generation process, and the number of the conditional frames may be one frame or multiple frames.

[0071] The target behaviors mentioned above may include, for example, “winking”, “smiling”, etc.

[0072] Step 202, generating a target video according to the conditional frame and the behavior description information by using a noise reduction module in a video generation model; the video generation model generates a target video by Figure 1 It can be obtained by training in any implementation method in the shown embodiments.

[0073] It should be noted that in the training process of the video generation model, the noise addition module and the noise reduction module are required, but in the actual application of the video generation model, only the noise reduction module is required to fill the gaps between the conditional frames and obtain a target video with coherent behavior. For example, if the target behaviors in the two conditional frames are "blinking" and "smiling" respectively, the target video can present a coherent process of blinking first and then smiling.

[0074] Furthermore, the process of generating the target video based on the conditional frames and behavior description information by the video generation model and the technical effects achieved are similar to the implementation process and technical effects achieved by the noise reduction module of the diffusion model in the previous article, and will not be repeated here.

[0075] In some application scenarios, if only one conditional frame is given, the corresponding target video can be generated in combination with the behavior description information. For example, if the behavior description information is "blink and then throw your head back and laugh", it can be split into three actions: "blink", "raise your head", and "laugh". Among them, if the digital object in the conditional frame is in a closed-eye state, a blinking short video including 16-32 video frames can be generated according to the conditional frame and the behavior description information "blink", and then the last frame can be used as the starting frame of the next head-up short video, and then the starting frame and the behavior description information "raise your head" can be combined to generate a head-up short video including 16-32 video frames, and then the last frame can be used as the starting frame of the next laughing short video, and then the starting frame and the behavior description information "laugh" can be combined to generate a laughing short video including 16-32 video frames, and then the three short videos can be spliced ​​to obtain a target video that meets the behavior description information "blink and then throw your head back and laugh".

[0076] In some application scenarios, if multiple conditional frames are given, each conditional frame can be used as the starting frame or the ending frame, and after being arranged and combined with other frames, the video generation model is combined with the behavior description information to generate rich coherent actions. It should be noted that due to the existence of multiple conditional frames, the lack of behavior description information will not affect the coherence or accuracy of the target video, so the current behavior description information can be empty.

[0077] In some application scenarios, if multiple conditional frames and behavior description information between any two conditional frames are given, a more complex target video can be generated through a video generation model.

[0078] In addition, in related technologies, for example, Unreal Engine or Unity (game development engine) can be used to construct animation content of a proposed digital image through artificial modeling, thereby generating a coherent video. In addition, after playing with real objects, a coherent video can be generated by combining multiple behaviors of the role-playing. However, there are problems such as long video production cycle, high production complexity, and high labor cost.

[0079] In this implementation, the video generation model can generate a target video with strong coherence and high accuracy based on the conditional frames and behavior description information that remain unchanged in their original appearance. In this way, since only a small number of conditional frames are used and the target video is generated in combination with the behavior description information, the production time is shortened, the production complexity is reduced, and the production cost is saved.

[0080] Those skilled in the art will appreciate that, in the above method of a specific embodiment, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0081] Please refer to Figure 3 , which shows a structural block diagram of a video generation model training device provided in an embodiment of the present application. The video generation model training device can be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1 The method embodiment involves various steps.

[0082] Optionally, the training device of the video generation model includes a marking module 301, an input module 302 and a determination module 303. The marking module 301 is used to mark multiple marked video frames in the sample video to obtain a marked video; the input module 302 is used to input the marked video and the sample behavior description information of the digital object in the sample video into the diffusion model to obtain a target sample video; the diffusion model includes a noise addition module and a noise reduction module, the noise addition module is used to perform noise addition processing on the marked video, and the noise reduction module is used to perform noise reduction processing on the marked video after noise addition to obtain a target sample video with coherent behavior; wherein the marked video frame remains unchanged during the training process of the diffusion model; the determination module 303 is used to determine the diffusion model as the video generation model if the target sample video matches the sample video.

[0083] Optionally, the marking module 301 is further used to randomly mark a preset proportion of marked video frames in the sample video to obtain the marked video.

[0084] Optionally, the marking module 301 is further used to: divide the sample video into a preset number of sample video segments; randomly mark multiple marked video frames from multiple sample video segments to obtain the marked video; wherein the number of marked video frames in the first and last sample video segments is respectively greater than the number of marked video frames in other sample video segments.

[0085] Optionally, the input module 302 is further used for: performing noise addition processing on each sample video frame in the marked video through the noise addition module; and eliminating the noise addition result of each marked video frame in the marked video.

[0086] Optionally, the input module 302 is further used to: perform noise reduction processing on each sample video frame in the marked video through the noise reduction module; and eliminate the noise reduction result of each marked video frame in the marked video.

[0087] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0088] Please refer to Figure 4 , which shows a structural block diagram of a video generation device provided in an embodiment of the present application. The video generation device may be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 2 The method embodiment corresponds to and can be executed Figure 2 The method embodiment involves various steps.

[0089] Optionally, the video generation device includes an acquisition module 401 and a generation module 402. The acquisition module 401 is used to acquire conditional frames and behavior description information in the video to be generated; the conditional frames are used to present the target behavior of the digital object; the generation module 402 is used to generate the target video according to the conditional frames and the behavior description information through a noise reduction module in a video generation model; the video generation model is trained by the method described in the first aspect.

[0090] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0091] Please refer to Figure 5 , Figure 5 A structural diagram of an electronic device for executing a training method for a video generation model or a video generation method provided in an embodiment of the present application, the electronic device may include: at least one processor 501, such as a CPU, at least one communication interface 502, at least one memory 503 and at least one communication bus 504. Among them, the communication bus 504 is used to realize direct connection and communication between these components. Among them, the communication interface 502 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 503 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 503 can optionally also be at least one storage device located away from the aforementioned processor. Computer-readable instructions are stored in the memory 503. When the computer-readable instructions are executed by the processor 501, the electronic device can execute the above Figure 1 or Figure 2 The method process is shown.

[0092] Understandably, Figure 5 The structure shown is for illustration only, and the electronic device may also include Figure 5 More or fewer components as shown, or with Figure 5 Different configurations shown. Figure 5Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0093] The present application embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following can be performed: Figure 1 or Figure 2 Any implementation method in the method embodiment shown.

[0094] An embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments. For example, the method may include: marking a plurality of marked video frames in a sample video to obtain a marked video; inputting the marked video and sample behavior description information of digital objects in the sample video into a diffusion model to obtain a target sample video; the diffusion model includes a noise addition module and a noise reduction module, the noise addition module is used to perform noise addition processing on the marked video, and the noise reduction module is used to perform noise reduction processing on the marked video after noise addition to obtain a target sample video with coherent behavior; wherein the marked video frames remain unchanged during the training process of the diffusion model; if the target sample video matches the sample video, the diffusion model is determined as a video generation model.

[0095] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0096] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0097] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0098] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0099] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a video generation model, characterized in that: include: Marking a plurality of marked video frames in the sample video to obtain a marked video; Inputting the sample behavior description information of the digital objects in the labeled video and the sample video into a diffusion model to obtain a target sample video; the diffusion model includes a noise adding module and a noise reducing module, the noise adding module is used to perform noise adding processing on the labeled video, and the noise reducing module is used to perform noise reducing processing on the labeled video after noise adding, to obtain a target sample video with coherent behavior; wherein the labeled video frame remains unchanged during the training process of the diffusion model; If the target sample video matches the sample video, the diffusion model is determined as a video generation model.

2. The method according to claim 1, characterized in that: The step of marking a plurality of marked video frames in the sample video to obtain the marked video includes: A preset ratio of marked video frames is randomly marked in the sample video to obtain the marked video.

3. The method according to claim 2, characterized in that Randomly marking a preset ratio of marked video frames in the sample video to obtain the marked video, including: Dividing the sample video into a preset number of sample video segments; A plurality of marked video frames are randomly marked from a plurality of sample video segments respectively to obtain the marked video; wherein the number of marked video frames in the first and last two sample video segments is respectively greater than the number of marked video frames in other sample video segments.

4. The method according to any one of claims 1 to 3, characterized in that: The step of performing noise adding processing on the marked video by the noise adding module includes: Performing noise addition processing on each sample video frame in the marked video by the noise addition module; and The noise added result of each marked video frame in the marked video is eliminated.

5. The method according to any one of claims 1 to 3, characterized in that: The noise reduction module performs noise reduction processing on the marked video after the noise is added, including: Performing noise reduction processing on each sample video frame in the marked video by the noise reduction module; and The denoising result of each marked video frame in the marked video is eliminated.

6. A video generation method, characterized in that: include: Obtain conditional frames and behavior description information in the video to be generated; The conditional frame is used to present the target behavior of the digital object; Generate a target video according to the conditional frame and the behavior description information through a noise reduction module in a video generation model; The video generation model is trained by the method described in any one of claims 1-5.

7. A training device for a video generation model, characterized in that: include: A marking module, used for marking a plurality of marked video frames in a sample video to obtain a marked video; An input module is used to input the sample behavior description information of the digital objects in the labeled video and the sample video into a diffusion model to obtain a target sample video; the diffusion model includes a noise addition module and a noise reduction module, the noise addition module is used to perform noise addition processing on the labeled video, and the noise reduction module is used to perform noise reduction processing on the labeled video after noise addition to obtain a target sample video with coherent behavior; wherein the labeled video frame remains unchanged during the training process of the diffusion model; A determination module is used to determine the diffusion model as a video generation model if the target sample video matches the sample video.

8. A video generating device, characterized in that: include: An acquisition module is used to acquire conditional frames and behavior description information in the video to be generated; The conditional frame is used to present the target behavior of the digital object; A generation module, used to generate a target video according to the conditional frame and the behavior description information through a noise reduction module in a video generation model; The video generation model is trained by the method described in any one of claims 1-5.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 is performed.

Images