Video generation method and related equipment thereof

Abstract

The invention relates to a video generation method and related equipment thereof, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of receiving a target duration, a target frame rate and key frame description information of a to-be-generated video input by a user based on a key frame picture input component provided with an animation dynamic entry command code; generating a plurality of static frame files based on the target duration, the target frame rate and the key frame description information; and synthesizing the plurality of static frame files into a video file according to the target frame rate. According to the embodiment of the invention, the video is automatically generated with low code engineering quantity, high efficiency and low cost, the video generation efficiency is improved, and the video generation effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video generation method and related equipment. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by its inclusion in this section.

[0003] With the development of artificial intelligence and natural language processing technology, natural language processing tools driven by artificial intelligence have been able to communicate by understanding and learning human language, and complete tasks such as writing emails, copywriting, translation, and image generation. However, the relevant technology can only realize the generation of static images based on language descriptions, and has not yet been able to achieve large-scale frame-by-frame video generation based on artificial intelligence; although the small-scale deployed AI (artificial intelligence) environment can generate videos based on language descriptions, there are technical problems such as large code engineering, high operating costs, low video generation efficiency, and poor video generation effects. Summary of the invention

[0004] A video generation method and related equipment provided by an embodiment of the present invention at least solve the problems of large code engineering workload, high operating cost, low video generation efficiency and poor generation effect in the video intelligent generation process existing in the related art.

[0005] In order to solve the above technical problem, one aspect of an embodiment of the present invention provides a video generation method, including:

[0006] Based on the key frame picture input component provided with the animation dynamic entry command code, receiving the target duration, target frame rate and key frame description information of the video to be generated input by the user;

[0007] Generate multiple static frame files based on target duration, target frame rate and key frame description information;

[0008] Composite multiple still frame files into a video file based on a target frame rate.

[0009] In some embodiments, the step of generating multiple static frame files based on the target duration, the target frame rate and the key frame description information includes:

[0010] Determine the target total number of frames based on the target duration and the target frame rate, generate a static key frame file based on the key frame description information, and determine the number of transition frames between adjacent key frames based on the target total number of frames;

[0011] Static transition frame files between adjacent key frames are generated according to the static key frame files and the number of transition frames, and the static key frame files and the static transition frame files are arranged in time sequence to obtain multiple static frame files.

[0012] In some of the embodiments, the step of generating a static key frame file according to the key frame description information includes:

[0013] The key frame description information is parsed, and the image generation model is called to generate a static key frame file according to the parsing results.

[0014] In some of these embodiments, the method further comprises:

[0015] Adjust the initial key frame number indicated by the animation dynamic entry command according to the target duration and target frame rate of the video to be generated input by the user;

[0016] Based on the adjusted key frame numbers, key frame description information of the video to be generated inputted by the user is received.

[0017] In some embodiments, the step of generating a plurality of static frame files based on the target duration, the target frame rate and the key frame description information further includes:

[0018] Determine a target total number of frames based on the target duration and the target frame rate;

[0019] Generate multiple static frame files in batches according to the target total number of frames and key frame description information.

[0020] In some of these embodiments, the method further comprises:

[0021] Receive update information of the video to be generated input by the user, and update the video file.

[0022] In some of the embodiments, before the step of inputting a component based on a key frame picture provided with an animation dynamic entry command code, the method includes:

[0023] The animation dynamic entry command code is set in the underlying code of the prompt dialog input item, and the key frame screen input component is constructed based on the underlying code.

[0024] In order to solve the above technical problem, another aspect of an embodiment of the present invention provides a video generating device, including:

[0025] A receiving module, for receiving target duration, target frame rate and key frame description information of a video to be generated input by a user based on a key frame picture input component provided with an animation dynamic entry command code;

[0026] A static frame file generation module, used to generate multiple static frame files based on target duration, target frame rate and key frame description information;

[0027] The video synthesis module is used to synthesize multiple static frame files into a video file according to a target frame rate.

[0028] In order to solve the above technical problem, another aspect of an embodiment of the present invention provides an electronic device, including: a processor, and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes any one of the above video generation methods.

[0029] In order to solve the above technical problem, another aspect of the embodiments of the present invention provides a non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute any of the above video generation methods.

[0030] The beneficial effects of the embodiments of the present invention are as follows: by receiving the target duration, target frame rate and key frame description information of the video to be generated input by the user based on the key frame picture input component provided with the animation dynamic entry command code; generating multiple static frame files based on the target duration, target frame rate and key frame description information; and synthesizing the multiple static frame files into a video file according to the target frame rate. The automatic generation of videos is realized with low code engineering, high efficiency and low cost, which reduces the cost of video generation, improves the efficiency of video generation and enhances the effect of video generation.

[0031] The details of one or more embodiments of the invention are set forth in the following drawings and description so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0033] Figure 1 It is a main flow chart of a video generation method according to an embodiment of the present invention.

[0034] Figure 2 It is a main flow chart of a video generation method according to another embodiment of the present invention.

[0035] Figure 3 It is a main framework diagram of a video generating device according to an embodiment of the present invention.

[0036] Figure 4 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0037] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0038] Diffusion Model: It works by learning the information attenuation due to noise and then uses the learned patterns to generate images.

[0039] Latent diffusion model (LDM): is a variant of the diffusion model. For example, Stable diffusion is a text-to-image generation model based on the latent diffusion model, which can generate images with corresponding descriptions based on arbitrary text input.

[0040] prompt: is a method in the JavaScript language, which is mainly used to display a prompt dialog box. In AI semantic generation, it is the generation instruction entered by the user in the form of text description.

[0041] Keyframe: A term in computer animation. The animation between keyframes can be created and added by software, which is called transition frame or intermediate frame.

[0042] Frame: The smallest unit of a single image in a video / animation, equivalent to each frame on a movie film. On the timeline of animation software, a frame is represented by a frame or a frame mark.

[0043] Frame rate: refers to the number of frames refreshed per second, which can also be understood as how many times the graphics processor can refresh per second. For video, frame rate refers to the number of still frames displayed per second.

[0044] The related technology has realized the operation of generating static images based on language description, but has not yet realized the large-scale generation of frame-by-frame videos based on artificial intelligence; and although the small-scale AI environment deployed through private cloud can generate videos based on language description, there are technical problems such as large code engineering volume, high operating costs, low video generation efficiency, and poor video generation effect.

[0045] In order to solve the above problems, the embodiment of the present invention provides a video generation method, such as Figure 1 As shown, the video generation method mainly includes:

[0046] Step S101, based on a key frame picture input component provided with an animation dynamic entry command code, receiving the target duration, target frame rate and key frame description information of the video to be generated input by the user.

[0047] Among them, the animation dynamic entry command code provided in the embodiment of the present invention is the "animation_prompt" dynamic entry command code, which can receive the target duration, target frame rate and key frame description information of the video to be generated input by the user through the key frame screen input component, and because the dynamic entry command code allows the user to dynamically modify the video frame value (including the frame rate and duration of the video), add description information and other operations. Through the above settings, based on the key frame screen input component provided with the animation dynamic entry command code, the key frame description information and video frame information input by the user are received, which helps to subsequently generate multiple static frame files according to the key frame description information, and synthesize the static frame files into a video file according to the target duration, thereby realizing the generation of videos based on text, reducing the amount of code engineering, reducing operating costs, and improving video generation efficiency.

[0048] In some embodiments, in order to meet the operating habits of different users, the corresponding key frame can also be switched to a key moment, that is, the description information of the key moment of the video to be generated by the user is received, and multiple static frame files are generated accordingly. By switching the frame position corresponding to the key frame to the key moment, not only can the different operating habits of users be met, but also the ease of use is enhanced.

[0049] Step S102: Generate multiple static frame files based on the target duration, target frame rate and key frame description information.

[0050] Based on the target duration and target frame rate, the total number of frames corresponding to the video to be generated can be determined. Combined with the key frame description information, multiple static frame files can be generated through the text generation image model, so that multiple static frame files can be subsequently synthesized into a video file. At the same time, since multiple static frame files can be generated in batches according to the key frame description information, the overall video generation efficiency is improved.

[0051] Step S103: synthesize multiple static frame files into a video file according to the target frame rate.

[0052] After all static frame files are generated, multiple static frame files can be combined into a video file according to the target frame rate pre-entered by the user to complete the automatic generation of the video. If the user does not enter a target frame rate but only enters a target duration, multiple static frame files can be combined into a video file according to the default frame rate.

[0053] In some of the embodiments, the step of generating multiple static frame files based on the target duration, target frame rate and key frame description information includes: determining the target total number of frames based on the target duration and target frame rate, generating a static key frame file according to the key frame description information, and determining the number of transition frames between adjacent key frames according to the target total number of frames; generating a static transition frame file between adjacent key frames according to the static key frame file and the number of transition frames, arranging the static key frame files and the static transition frame files in time sequence to obtain multiple static frame files.

[0054] Among them, the key frames provided by the embodiment of the present invention include at least one of the video start frame, the middle frame, and the end frame, and the middle frame can also be one or more. For example, if the target duration of the video to be generated is 5s and the target frame rate is 24fps, the corresponding total number of frames is 120 frames. The 1st frame (video start frame), the 60th frame (video middle frame), and the 120th frame (video end frame) are used as key frames. After receiving the key frame description information input by the user for the key frame, multiple static frame files are generated. It should be noted that the above values ​​are only examples and are not intended to limit the present invention. It can be understood that the number of key frames and the level of detail of the key frame description information will affect the final generated video effect.

[0055] The above-provided embodiment of the present invention is a specific implementation method for generating multiple static frame files based on key frame description information, that is, firstly generating a corresponding static key frame file based on the key frame description information, and then generating a static transition frame file based on the connection between adjacent key frames, and arranging the static key frame file and the static transition frame file in time sequence, so as to subsequently quickly synthesize multiple static frame files with time sequence characteristics into a video file, so as to further improve the video generation efficiency.

[0056] In some of the embodiments, the step of generating a static key frame file according to the key frame description information includes: parsing the key frame description information, and calling an image generation model to generate a static key frame file according to the parsing result.

[0057] In the specific implementation, the CLIP (Contrastive Language-Image Pre-training) node can be called to parse the key frame description information, and then the image generation model (such as the GAN model (adversarial generation model), DIFFUSION model (diffusion model)) can be called to generate a static frame file.

[0058] In some of the embodiments, the above method also includes: adjusting the initial key frame number indicated by the animation dynamic entry command according to the target duration and target frame rate of the video to be generated input by the user; based on the adjusted key frame number, receiving the key frame description information of the video to be generated input by the user.

[0059] The animation dynamic entry command code set in the embodiment of the present invention can meet the user's dynamic adjustment of the generated video. It should be pointed out that the longer the duration of the video to be generated, the more corresponding key frames need to be increased to improve the effect of the generated video.

[0060] In some of the embodiments, the above-mentioned step of generating multiple static frame files based on the target duration, target frame rate and key frame description information also includes: determining the target total number of frames based on the target duration and the target frame rate; and batch generating multiple static frame files according to the target total number of frames and the key frame description information.

[0061] Through the above settings, the embodiment of the present invention provides another specific implementation method for generating static frame files according to key frame description information, that is, directly generating multiple static frame files in batches according to the target total number of frames and key frame description information, further improving the video generation efficiency.

[0062] In some of the embodiments, the method further includes: receiving update information of the video to be generated input by a user, and updating the video file.

[0063] After the video file is generated, it is also possible to receive update information of the video to be generated from the user's feedback on the video file and update the video file. The above settings are helpful to improve the user experience and optimize the video generation effect.

[0064] In some of the embodiments, the user can also adjust the video length by himself, and the input node is stretched proportionally after the adjustment. According to another specific implementation of the embodiment of the present invention, the frame cutoff point can also be set, the update information of the frame cutoff point position can be modified, and the video file can be updated and adjusted accordingly to meet the diverse needs of users.

[0065] In some of the embodiments, before the step of setting a key frame screen input component based on the animation dynamic entry command code, the method includes: setting the animation dynamic entry command code in the underlying code of the input item of the display prompt dialog box, and constructing the key frame screen input component based on the underlying code.

[0066] The video generation method provided by the embodiment of the present invention is based on the decomposition of the diffusion model generation principle. In the underlying code of the prompt dialog input item (prompt input item), the "animation_prompt" dynamic entry command is added, and a key frame picture input component constructed based on the underlying code is provided to the user on the front-end interface to receive the description information corresponding to the key frames such as the start frame, middle frame, and end frame of the video picture from the user, and then multiple static frame files are generated according to the key frame description information, and finally the multiple static frame files are synthesized into a video file.

[0067] The video generation method provided by the embodiment of the present invention receives the target duration, target frame rate and key frame description information of the video to be generated input by the user through a key frame picture input component provided with an animation dynamic entry command code; generates multiple static frame files based on the target duration, target frame rate and key frame description information; and synthesizes the multiple static frame files into a video file according to the target frame rate. This method realizes automatic video generation with low code engineering, high efficiency and low cost, reduces the cost of video generation, improves the efficiency of video generation and enhances the effect of video generation.

[0068] In order to solve the above problems, the embodiment of the present invention also provides another video generation method, which is applied to a tool for generating videos based on artificial intelligence. Figure 2 As shown, the video generation method mainly includes:

[0069] Step S201, setting the animation dynamic entry command code in the underlying code of the display prompt dialog input item, and constructing the key frame picture input component based on the underlying code.

[0070] Through the above settings, the "animation_prompt" dynamic entry command is added to the underlying code of the prompt input item, and a key frame screen input component built based on the underlying code is provided to the user on the front-end interface to receive the description information corresponding to the key frames such as the start frame, middle frame, and end frame of the video screen. Subsequently, multiple static frame files can be generated according to the key frame description information, and finally the multiple static frame files are synthesized into a video file. Thus, a solution for automatically generating videos based on text (description information) is realized, and the code engineering workload is small, which reduces the cost of video generation and improves the video generation rate.

[0071] Step S202, based on the key frame picture input component provided with the animation dynamic entry command code, receives the target duration, target frame rate and key frame description information of the video to be generated input by the user.

[0072] Among them, the animation dynamic entry command code provided by the embodiment of the present invention can receive the target duration, target frame rate and key frame description information of the video to be generated input by the user through the key frame screen input component, and because the dynamic entry command code allows the user to dynamically modify the video frame value (including the frame rate and duration of the video), add description information and other operations. Through the above settings, based on the key frame screen input component provided with the animation dynamic entry command code, the key frame description information and video frame information input by the user are received, which is helpful for the subsequent generation of multiple static frame files according to the key frame description information, and the static frame files are synthesized into a video file according to the target duration, thereby realizing the generation of videos based on text, reducing the code engineering workload, reducing operating costs, and improving the efficiency of video generation.

[0073] The key frames provided in the embodiment of the present invention include at least one of the video start frame, the middle frame, and the end frame, and the middle frame can also be one or more. Taking the target duration of the video to be generated as 5s and the target frame rate as 24fps as an example, the corresponding total number of frames is 120 frames, then the 1st frame (video start frame), the 60th frame (video middle frame), and the 120th frame (video end frame) are used as key frames, and after receiving the key frame description information input by the user for the key frames, multiple static frame files are generated. The corresponding code example is:

[0074] [animation_prompt=

[0075] 1:"promptsxxxxxxxxxxxxxxx"

[0076] 60:"promptsxxxxxxxxxxxxxxx"

[0077] 120: "promptsxxxxxxxxxxxxxxx"], where "xxxxxxxxxxxxxxx" after prompts represents the details of the key frame description information. It should be noted that the above values ​​are only examples and are not intended to limit the present invention. It is understood that the number of key frames and the level of detail of the key frame description information will affect the final generated video effect.

[0078] In some embodiments, in order to meet the operating habits of different users, the corresponding key frame can also be switched to a key moment, that is, the description information of the key moment of the video to be generated by the user is received, and multiple static frame files are generated accordingly. By switching the frame position corresponding to the key frame to the key moment, not only can the different operating habits of users be met, but also the ease of use is enhanced.

[0079] In some of the embodiments, the above method also includes: adjusting the initial key frame number indicated by the animation dynamic entry command according to the target duration and target frame rate of the video to be generated input by the user; based on the adjusted key frame number, receiving the key frame description information of the video to be generated input by the user.

[0080] The animation dynamic entry command code set in the embodiment of the present invention can meet the user's dynamic adjustment of the generated video. It should be pointed out that the longer the duration of the video to be generated, the more corresponding key frames need to be increased to improve the effect of the generated video.

[0081] Step S203, determining a target total number of frames based on the target duration and the target frame rate, and determining the number of transition frames between adjacent key frames according to the target total number of frames.

[0082] Based on the target duration and target frame rate, the target total number of frames corresponding to the video to be generated can be determined, and then the number of transition frames between adjacent key frames can be determined, so that after the static key frame file is generated, the static transition frame file between adjacent key frames can continue to be generated.

[0083] Step S204, parse the key frame description information, and call the image generation model to generate a static key frame file according to the parsing result.

[0084] In the specific implementation, the CLIP (Contrastive Language-Image Pre-training) node can be called to parse the key frame description information, and then the image generation model (such as the GAN model (adversarial generation model), DIFFUSION model (diffusion model)) can be called to generate a static frame file.

[0085] Step S205, generating static transition frame files between adjacent key frames according to the static key frame files and the number of transition frames, arranging the static key frame files and the static transition frame files in time sequence to obtain a plurality of static frame files.

[0086] The above-provided embodiment of the present invention is a specific implementation method for generating multiple static frame files based on key frame description information, that is, firstly generating a corresponding static key frame file based on the key frame description information, and then generating a static transition frame file based on the connection between adjacent key frames, and arranging the static key frame file and the static transition frame file in time sequence, so as to subsequently quickly synthesize multiple static frame files with time sequence characteristics into a video file, so as to further improve the video generation efficiency.

[0087] In some of the embodiments, the above-mentioned step of generating multiple static frame files based on the target duration, target frame rate and key frame description information also includes: determining the target total number of frames based on the target duration and the target frame rate; and batch generating multiple static frame files according to the target total number of frames and the key frame description information.

[0088] Through the above settings, the embodiment of the present invention provides another specific implementation method for generating static frame files according to key frame description information, that is, directly generating multiple static frame files in batches according to the target total number of frames and key frame description information, further improving the video generation efficiency.

[0089] Step S206: synthesize the multiple static frame files into a video file according to the target frame rate.

[0090] After all static frame files are generated, you only need to synthesize multiple static frame files into a video file according to the target frame rate pre-entered by the user to complete the automatic generation of the video. If the user does not enter the target frame rate, but only enters the target duration, multiple static frame files can be synthesized into a video file according to the default frame rate. Compared with the solution of deploying an AI environment in a private cloud for video generation in the related art, the existing problems of high operating costs, high requirements on the user's code ability, and poor video generation efficiency and generation effect, the embodiment of the present invention is based on a key frame screen input component with an animation dynamic entry command code to reduce the amount of code engineering, reduce the difficulty of user operation, improve video generation efficiency, have a relatively broad application scenario, and can provide users with a good experience.

[0091] In some of the embodiments, the method further includes: receiving update information of the video to be generated input by a user, and updating the video file.

[0092] After the video file is generated, it is also possible to receive update information of the video to be generated from the user's feedback on the video file and update the video file. The above settings are helpful to improve the user experience and optimize the video generation effect.

[0093] In some of the embodiments, the user can also adjust the video length by himself, and the input node is stretched proportionally after the adjustment. According to another specific implementation of the embodiment of the present invention, the frame cutoff point can also be set, the update information of the frame cutoff point position can be modified, and the video file can be updated and adjusted accordingly to meet the diverse needs of users.

[0094] The video generation method provided by the embodiment of the present invention receives the target duration, target frame rate and key frame description information of the video to be generated input by the user through a key frame picture input component provided with an animation dynamic entry command code; generates multiple static frame files based on the target duration, target frame rate and key frame description information; and synthesizes the multiple static frame files into a video file according to the target frame rate. This method realizes automatic video generation with low code engineering, high efficiency and low cost, reduces the cost of video generation, improves the efficiency of video generation and enhances the effect of video generation.

[0095] Based on the above-mentioned video generation method provided by the embodiment of the present invention, the embodiment of the present invention also provides a video generation device, which is applied to the automatic generation system of AI video, such as Figure 3 As shown, the video generating device 300 includes:

[0096] The receiving module 301 is used to receive the target duration, target frame rate and key frame description information of the video to be generated input by the user based on the key frame picture input component provided with the animation dynamic entry command code.

[0097] Among them, the animation dynamic entry command code provided by the embodiment of the present invention can receive the target duration, target frame rate and key frame description information of the video to be generated input by the user through the key frame screen input component, and because the dynamic entry command code allows the user to dynamically modify the video frame value (including the frame rate and duration of the video), add description information and other operations. Through the above settings, based on the key frame screen input component provided with the animation dynamic entry command code, the key frame description information and video frame information input by the user are received, which is helpful for the subsequent generation of multiple static frame files according to the key frame description information, and the static frame files are synthesized into a video file according to the target duration, thereby realizing the generation of videos based on text, reducing the code engineering workload, reducing operating costs, and improving the efficiency of video generation.

[0098] In some embodiments, in order to meet the operating habits of different users, the corresponding key frame can also be switched to a key moment, that is, the description information of the key moment of the video to be generated by the user is received, and multiple static frame files are generated accordingly. By switching the frame position corresponding to the key frame to the key moment, not only can the different operating habits of users be met, but also the ease of use is enhanced.

[0099] The static frame file generating module 302 is used to generate a plurality of static frame files based on the target duration, the target frame rate and the key frame description information.

[0100] Based on the target duration and target frame rate, the total number of frames corresponding to the video to be generated can be determined. Combined with the key frame description information, multiple static frame files can be generated through the text generation image model, so that multiple static frame files can be subsequently synthesized into a video file. At the same time, since multiple static frame files can be generated in batches according to the key frame description information, the overall video generation efficiency is improved.

[0101] The video synthesis module 303 is used to synthesize multiple static frame files into a video file according to a target frame rate.

[0102] After all static frame files are generated, multiple static frame files can be combined into a video file according to the target frame rate pre-entered by the user to complete the automatic generation of the video. If the user does not enter a target frame rate but only enters a target duration, multiple static frame files can be combined into a video file according to the default frame rate.

[0103] In some of the embodiments, the static frame file generation module 302 is further used to: determine the target total number of frames based on the target duration and the target frame rate, generate a static key frame file according to the key frame description information, and determine the number of transition frames between adjacent key frames according to the target total number of frames; generate a static transition frame file between adjacent key frames according to the static key frame file and the number of transition frames, arrange the static key frame file and the static transition frame file in time sequence, and obtain multiple static frame files.

[0104] Among them, the key frames provided by the embodiment of the present invention include at least one of the video start frame, the middle frame, and the end frame, and the middle frame can also be one or more. For example, if the target duration of the video to be generated is 5s and the target frame rate is 24fps, the corresponding total number of frames is 120 frames. Then the 1st frame (video start frame), the 60th frame (video middle frame), and the 120th frame (video end frame) can be used as key frames. After receiving the key frame description information input by the user for the key frame, multiple static frame files are generated. It should be noted that the above values ​​are only examples and are not intended to limit the present invention. It can be understood that the number of key frames and the level of detail of the key frame description information will affect the final generated video effect.

[0105] The above-provided embodiment of the present invention is a specific implementation method for generating multiple static frame files based on key frame description information, that is, firstly generating corresponding static key frame files based on the key frame description information, and then generating static transition frame files based on the content connection between adjacent key frames, and arranging the static key frame files and static transition frame files in time sequence, so as to subsequently quickly synthesize multiple static frame files with time sequence characteristics into a video file, so as to further improve the video generation efficiency.

[0106] In some of the embodiments, the static frame file generation module 302 is further used to: parse the key frame description information, and call the image generation model to generate a static key frame file according to the parsing result.

[0107] In the specific implementation, the CLIP (Contrastive Language-Image Pre-training) node can be called to parse the key frame description information, and then the image generation model (such as the GAN model (adversarial generation model), DIFFUSION model (diffusion model)) can be called to generate a static frame file.

[0108] In some of the embodiments, the video generating device 300 further includes an adjustment module for: adjusting the initial key frame number indicated by the animation dynamic entry command according to the target duration and target frame rate of the video to be generated input by the user; and receiving the key frame description information of the video to be generated input by the user corresponding to the adjusted key frame number.

[0109] The animation dynamic entry command code set in the embodiment of the present invention can meet the user's dynamic adjustment of the generated video. It should be pointed out that the longer the duration of the video to be generated, the more corresponding key frames need to be increased to improve the effect of the generated video.

[0110] In some of the embodiments, the static frame file generation module 302 is further used to: determine a target total number of frames based on a target duration and a target frame rate; and batch generate multiple static frame files according to the target total number of frames and key frame description information.

[0111] Through the above settings, the embodiment of the present invention provides another specific implementation method for generating static frame files according to key frame description information, that is, directly generating multiple static frame files in batches according to the target total number of frames and key frame description information, further improving the video generation efficiency.

[0112] In some of the embodiments, the video generating device 300 further includes an updating module, which is used to receive update information of the video to be generated input by a user, and update the video file.

[0113] After the video file is generated, it is also possible to receive update information of the video to be generated from the user's feedback on the video file and update the video file. The above settings are helpful to improve the user experience and optimize the video generation effect.

[0114] In some of the embodiments, the user can also adjust the video length by himself, and the input node is stretched proportionally after the adjustment. According to another specific implementation of the embodiment of the present invention, the frame cutoff point can also be set, the update information of the frame cutoff point position can be modified, and the video file can be updated and adjusted accordingly to meet the diverse needs of users.

[0115] In some of the embodiments, the video generating device 300 further includes a key frame screen input component construction module. Before the step of providing a key frame screen input component based on an animation dynamic entry command code, the key frame screen input component module is used to: set the animation dynamic entry command code in the underlying code of the input item of the prompt dialog box, and construct the key frame screen input component based on the underlying code.

[0116] The video generation device provided by the embodiment of the present invention is based on the disassembly of the diffusion model generation principle. In the underlying code of the prompt dialog box input item (prompt input item), the "animation_prompt" dynamic entry command is added, and a key frame picture input component constructed based on the underlying code is provided to the user on the front-end interface to receive the user's description information corresponding to the key frames such as the start frame, middle frame, and end frame of the video picture, and then generate multiple static frame files according to the key frame description information, and finally synthesize the multiple static frame files into a video file.

[0117] The video generation device provided by the embodiment of the present invention receives the target duration, target frame rate and key frame description information of the video to be generated input by the user through a key frame picture input component provided with an animation dynamic entry command code; generates multiple static frame files based on the target duration, target frame rate and key frame description information; and synthesizes the multiple static frame files into a video file according to the target frame rate. This realizes the automatic generation of videos with low code engineering, high efficiency and low cost, reduces the cost of video generation, improves the efficiency of video generation, and enhances the effect of video generation.

[0118] An embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory in communication with the at least one processor. The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the electronic device executes the method of the embodiment of the present invention.

[0119] An embodiment of the present invention further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present invention.

[0120] An embodiment of the present invention further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to enable the computer to execute the method of the embodiment of the present invention.

[0121] refer to Figure 4 , a block diagram of an electronic device that can be used as a server or client of an embodiment of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0122] like Figure 4 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0123] Multiple components in the electronic device are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device, and the input unit 406 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include but is not limited to a disk, an optical disk. The communication unit 409 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0124] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0125] The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of an embodiment of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0128] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0129] The various steps described in the method implementation methods provided in the embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0130] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0131] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A video generation method, comprising: Based on the key frame picture input component provided with the animation dynamic entry command code, receiving the target duration, target frame rate and key frame description information of the video to be generated input by the user; Generate a plurality of static frame files based on the target duration, the target frame rate and the key frame description information; The plurality of static frame files are synthesized into a video file according to the target frame rate.

2. The method according to claim 1, wherein: The step of generating a plurality of static frame files based on the target duration, the target frame rate and the key frame description information comprises: Determining a target total number of frames based on the target duration and the target frame rate, generating a static key frame file according to the key frame description information, and determining the number of transition frames between adjacent key frames according to the target total number of frames; Static transition frame files between adjacent key frames are generated according to the static key frame files and the number of transition frames, and the static key frame files and the static transition frame files are arranged in time sequence to obtain the multiple static frame files.

3. The method according to claim 2, wherein: The step of generating a static key frame file according to the key frame description information comprises: The key frame description information is parsed, and an image generation model is called to generate the static key frame file according to the parsing result.

4. The method according to claim 1, wherein: The method further comprises: According to the target duration and target frame rate of the video to be generated input by the user, the initial key frame number indicated by the animation dynamic entry command is adjusted; Based on the adjusted key frame numbers, key frame description information of the video to be generated inputted by the user is received.

5. The method according to claim 1, wherein: The step of generating a plurality of static frame files based on the target duration, the target frame rate and the key frame description information further includes: Determine a target total number of frames based on the target duration and the target frame rate; A plurality of static frame files are generated in batches according to the target total number of frames and the key frame description information.

6. The method according to claim 1, wherein: The method further comprises: Receive update information of the video to be generated input by the user, and update the video file.

7. The method according to claim 1, wherein: Prior to the step of inputting a component based on a key frame image provided with an animation dynamic entry command code, the method comprises: The animation dynamic entry command code is set in the underlying code of the display prompt dialog input item, and the key frame picture input component is constructed based on the underlying code.

8. A video generating device, comprising: A receiving module, for receiving target duration, target frame rate and key frame description information of a video to be generated input by a user based on a key frame picture input component provided with an animation dynamic entry command code; A static frame file generation module, used to generate a plurality of static frame files based on the target duration, the target frame rate and the key frame description information; The video synthesis module is used to synthesize the multiple static frame files into a video file according to the target frame rate.

9. An electronic device, comprising: A processor, and a memory storing a program, wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images