Video generation method, computing device, storage medium and program product

Abstract

The embodiment of the invention provides a video generation method, computing equipment, a storage medium and a program product. Wherein in response to a video generation request, key information provided by a user is determined, and a target script can be generated by using a large model based on the key information; wherein the target script relates to at least one scene; generating a target image corresponding to any scene by using the large model in combination with a target script, generating a plurality of candidate videos corresponding to the scene by using the large model in combination with the target script and the target image, and determining a scene video matched with the target script from the plurality of candidate videos; and then generating a target video based on the respective scene video of the at least one scene. According to the technical scheme provided by the embodiment of the invention, the video generation efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video generation method, a computing device, a storage medium, and a program product. Background Art

[0002] At present, animation video generation usually relies on complex multi-stage processes and requires a lot of manual input. For example, it is necessary to manually design the script content such as stories, characters, and scenes, and manually perform character modeling and scene construction during the production stage. The above processes are relatively complicated and cannot meet the flexible and changeable animation video generation needs.

[0003] Therefore, how to simplify the generation process of animation videos and improve the efficiency of animation video generation has become a technical problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present application provide a video generation method, a computing device, a storage medium, and a program product to solve the problem of low video generation efficiency in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, comprising:

[0006] In response to the video generation request, determining key information provided by the user;

[0007] Based on the key information, a target script is generated using a large model; the target script involves at least one scene;

[0008] Using the large model in combination with the target script, a target image corresponding to any scene is generated;

[0009] Using the large model in combination with the target script and the target image, a plurality of candidate videos corresponding to the scene are generated, and a scene video matching the target script is determined from the plurality of candidate videos;

[0010] A target video is generated based on the scene video of each of the at least one scene.

[0011] In a second aspect, an embodiment of the present application provides a video generation method, including:

[0012] Obtain key information provided by users;

[0013] In response to a video generation operation triggered by the key information, generating a video generation request;

[0014] The video generation request is sent to the server; the video generation request is used to trigger the server to generate a target script based on the key information using the big model and generate a target image corresponding to any scene in combination with the target script, and to generate multiple candidate videos corresponding to the scene using the big model in combination with the target script and the target image, and to determine a scene video matching the target script from the multiple candidate videos, and to generate a target video based on the scene videos of at least one scene involved in the target script.

[0015] In a third aspect, an embodiment of the present application provides a video generation method, including:

[0016] In response to a video generation request, generating a target script using the large model; the target script involves at least one scene;

[0017] Using the large model in combination with the target script, a plurality of candidate videos corresponding to any scene are generated, and a scene video matching the target script is determined from the plurality of candidate videos;

[0018] A target video is generated based on the scene video of each of the at least one scene.

[0019] In a fourth aspect, a computing device is provided in an embodiment of the present application, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the video generation method as described in the first aspect above, the video generation method as described in the second aspect above, or the video generation method as described in the third aspect above.

[0020] In a fifth aspect, a computer storage medium is provided in an embodiment of the present application, storing a computer program, which, when executed by a computer, implements the video generation method described in the first aspect above, the video generation method described in the second aspect above, or the video generation method described in the third aspect above.

[0021] In a sixth aspect, a computer program product is provided in an embodiment of the present application, comprising a computer program / instructions, which, when executed by a computer, implements the video generation method as described in the first aspect above, the video generation method as described in the second aspect above, or the video generation method as described in the third aspect above.

[0022] In an embodiment of the present application, in response to a video generation request, key information provided by the user is determined, and a target script can be generated based on the key information using a large model; wherein the target script involves at least one scene; then the large model can be used in combination with the target script to generate a target image corresponding to any scene, and the large model can be used in combination with the target script and the target image to generate multiple candidate videos of the scene, and the scene video matching the target script is determined from the multiple candidate videos, and then the target video can be generated based on the scene video of at least one scene. In an embodiment of the present application, the user only needs to provide key information and use the large model to generate a coherent and detailed target script without multiple interactive operations, which can reduce manual creation and improve creation efficiency. The large model can be directly used in combination with the target script to automatically generate the target image of the scene and generate the scene video based on the target image, which can shorten the video generation cycle, realize the automation of video generation, improve generation efficiency, and generate multiple candidate videos with the help of the large model before performing video screening to ensure video quality.

[0023] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of an embodiment of a video generation method provided by the present application is shown;

[0026] Figure 2 A flowchart of an embodiment of a video generation method provided by the present application is shown;

[0027] Figure 3 A flowchart of an embodiment of a video generation method provided by the present application is shown;

[0028] Figure 4 A schematic diagram of scene interaction in an actual application of an embodiment of the present application is shown;

[0029] Figure 5 A schematic diagram of video generation in a practical application of an embodiment of the present application is shown;

[0030] Figure 6 A schematic structural diagram of an embodiment of a video generating device provided by the present application is shown;

[0031] Figure 7 A schematic structural diagram of an embodiment of a video generating device provided by the present application is shown;

[0032] Figure 8 A schematic diagram of the structure of an embodiment of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0034] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0035] The technical solution of the embodiment of the present application can be applied to video generation scenarios, especially animation video generation scenarios. As described in the background technology, currently, animation video generation usually relies on a complex multi-stage process, which requires a lot of manual input, such as manual design of the story, characters, scenes and other script contents, and manual character modeling and scene construction during the production stage. The above process is relatively complex and cannot meet the flexible and changeable animation video generation requirements, resulting in low video generation efficiency.

[0036] In order to improve the efficiency of video generation, the inventors thought that with the development of artificial intelligence technology, it is possible to consider using machine learning models to assist in video generation, such as using a pre-trained image generation model to generate images based on the script content, and then using the images to form a video or using a pre-trained video generation model to generate a video, etc. However, the inventors found that using machine learning models still requires a lot of manual input and correction, the degree of automation is still low, and the video generation efficiency is low. The inventors further found that the quality of videos generated directly from the script content using machine learning models is poor.

[0037] In order to improve the efficiency of video generation, the inventor has proposed the technical solution of the present application after a series of studies. In the embodiment of the present application, in response to the video generation request, the target script can be generated using a large model; wherein the target script involves at least one scene; the large model is used in combination with the target script to generate multiple candidate videos corresponding to any scene; the large model is used in combination with the target script to screen and obtain scene videos from the multiple candidate videos; based on the scene videos of at least one scene, the target video is generated. In the embodiment of the present application, the use of a large model to generate a target script can reduce manual creation and improve creation efficiency. After that, the large model is directly used in combination with the target script to generate multiple candidate videos, and the scene video is screened from the generated multiple candidate videos, which can avoid the complex process of manual correction and manual construction of roles and scenes, shorten the cycle of video generation, and improve generation efficiency. In the embodiment of the present application, the script generation, video generation, and video screening operations are automatically realized with the help of a large model, which realizes the automation of video generation, reduces manual resources, reduces video production costs, and improves video generation efficiency. In addition, in the embodiment of the present application, video screening can be performed with the help of a large model to ensure the quality and effect of the video. In addition, by using a large model combined with the target script, we can first generate a scene image of the scene, and then combine the scene image to generate a video of the scene, which can further ensure the video quality and effect.

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0039] It should be noted that the technical solution of the embodiment of the present application is applicable to a network virtual environment. The users described generally refer to "virtual users". Real users can register a user account in the server through registration to obtain a user identity in the network environment.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] The implementation details of the technical solution of the embodiment of the present application are elaborated in detail below.

[0042] Figure 1A flowchart of an embodiment of a video generation method provided in the present application. The technical solution of this embodiment can be executed by a server. In a practical application, the technical solution of the embodiment of the present application can be applied to a system architecture composed of a user end and a server. A connection can be established between the user end and the server through a network. The network can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] Among them, the user end can be implemented as a browser, APP (Application), or web application such as H5 (HyperText Markup Language5, Hypertext Markup Language Version 5) application, or light application (also known as applet, a lightweight application) or cloud application, etc. The first client or the second client can be deployed in an electronic device, and needs to rely on the device to run or some apps in the device to run, etc. The electronic device can, for example, have an electronic display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, a desktop computer, a smart speaker, a smart watch, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. Electronic devices can usually include at least one processing component and at least one storage component. Electronic devices may also include basic configurations such as network card chips, IO (input / output) buses, audio and video components, which are not limited in this application. Optionally, depending on the implementation form of the electronic device, some peripheral devices may also be included, such as a keyboard, a mouse, an input pen, a printer, etc., which is not limited in this application.

[0044] The service end may include servers that provide various services, such as a server that provides instant messaging, a server that provides audio and video communications, a proxy server that provides gateway services, etc.

[0045] It should be noted that the server can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0046] Figure 1The video generation method of the illustrated embodiment may include the following steps:

[0047] 101: In response to a video generation request, generate a target script using a large model.

[0048] The target script may involve at least one scene. A scene refers to a series of actions and dialogues that take place at a specific place and time, usually used to describe a specific event or plot segment in a story. A scene is one of the basic building blocks of the target script, and each scene has its own unique environment setting, character interactions, and actions.

[0049] Optionally, the video generation request may be triggered by a user, and the user end may sense the video generation operation triggered by the user, and then generate a video generation request and send it to the server end.

[0050] Optionally, the video generation request may include key information provided by the user. Therefore, the key information provided by the user may be determined in response to the video generation request, and the target script may be generated based on the key information using the macro model. The key information may be, for example, a short story text provided by the user, and the macro model may be used to expand the story text and generate a coherent and detailed target script. The story text may include basic elements such as background, main characters, initial events, etc.

[0051] Among them, a large model refers to a machine learning model with a large number of parameters and complex structure, which can process massive data and complete various complex tasks, such as language generation, computer vision, speech recognition, etc. It is an AI (artificial intelligence) model. The large model can be implemented using a large language model (English: Large Language Model, referred to as: LLM) or a large multimodal model (English: Large Multimodal Model, referred to as: LMM), etc. For example, a generative pre-trained model, a bidirectional encoder model, etc. can be used. This application does not limit this.

[0052] 102: Utilize the large model in combination with the target script to generate multiple candidate videos corresponding to any scene, and determine the scene video matching the target script from the multiple candidate videos.

[0053] Among them, after using the big model to generate the target script, the big model can be called again to combine with the target script to generate multiple candidate videos corresponding to each scene.

[0054] The target script generated by using the large model may include, for example, character description information, the scene setting information, and scene description information. The character description information may be used to introduce the character and its basic characteristics, such as name, personality, characteristics, appearance, background, and other information; the scene setting information may be used to introduce location, time, environment, and other information. The scene description information may be used to introduce a series of actions and dialogues of the character at a specific location and / or at a specific time, and other plot information. Of course, this is only an example of a possible structure of a script, and the present application is not limited to this.

[0055] The large model may be generated by combining the character description information, scene setting information, and scene description information in the target script to generate multiple candidate videos corresponding to each scene. Each candidate video may include animation effects such as characters and / or scenes.

[0056] Of course, the large model may also use other methods to generate multiple candidate videos corresponding to each scene, which will be described in detail in the following embodiments.

[0057] When multiple candidate scenes are generated using the big model, the big model can be called again to determine the scene video that matches the target script from the multiple candidate videos.

[0058] Optionally, a large model may be used in combination with a target script to determine a scene video from a plurality of candidate videos.

[0059] The large model can be used to determine the scene video that matches the target script from multiple candidate videos according to certain screening requirements, so that the scene video meets the setting of the target script, etc., to ensure the generation quality of the scene video. The specific implementation method of the scene video will be described in detail in the corresponding embodiments below.

[0060] 103: Generate a target video based on the scene video of at least one scene.

[0061] Optionally, the scene videos of each scene can be spliced ​​in the scene order to obtain a target video, which can present the plot of the target script with animation effects.

[0062] In an embodiment of the present application, in response to a video generation request, a target script can be generated using a large model; wherein the target script involves at least one scene; the large model is used in combination with the target script to generate multiple candidate videos corresponding to any one scene; the large model is used in combination with the target script to screen and obtain a scene video from multiple candidate videos; based on the scene video of each of at least one scene, the target video is generated. In an embodiment of the present application, the use of a large model to generate a target script can reduce manual creation and improve creation efficiency. After that, the large model is directly used in combination with the target script to generate multiple candidate videos, and the scene video is screened from the generated multiple candidate videos, which can avoid the complex process of manual correction and manual construction of roles and scenes, shorten the video generation cycle, and improve generation efficiency. In an embodiment of the present application, operations such as script generation, video generation, and video screening are automatically realized with the help of a large model, which realizes the automation of video generation, reduces manual resources, reduces video production costs, and improves video generation efficiency. In addition, in an embodiment of the present application, video screening can be performed with the help of a large model to ensure the quality and effect of the video.

[0063] In some embodiments, the method may further include: outputting a target script; updating the target script in response to a script update request for the target script; and generating a plurality of candidate videos corresponding to any one scene in response to a script confirmation request for the target script in combination with the target script.

[0064] By outputting the target script, the user can update the target script, ensure the quality of the target script, and improve the user experience. Optionally, the target script can be sent to the user end so that the user end can display the target script in the user interface. The script update request can be generated by the user end in response to the user's update operation on the target script.

[0065] In some embodiments, the method may further include: outputting a scene video; in response to a video update request for the scene video, returning to the step of using the large model in combination with the target script to generate multiple candidate videos corresponding to any scene and re-executing the step; in response to a video confirmation request for the scene video, generating a target video based on the scene video of at least one scene.

[0066] By outputting the scene video, the user can update the scene video, ensure the quality and effect of the scene video, and improve the user experience. Optionally, the scene video of each scene can be sent to the user end so that the user end can play the scene video in the user interface. The video update request can be generated by the user end in response to the user's update operation on the scene video.

[0067] As can be seen from the above description, the target script may include character description information, scene setting information, and scene description information of at least one scene. In response to the video generation request, generating the target script using the big model may include: determining key information provided by the user in response to the video generation request; and generating the target script using the big model based on the key information. In some embodiments, see Figure 2 The flowchart of the video generation method shown may include:

[0068] 201: In response to a video generation request, determine key information provided by a user.

[0069] 202: Generate target scripts using big models based on key information.

[0070] Wherein, the target script involves at least one scene.

[0071] 203: Generate a target image corresponding to any scene by combining the large model with the target script.

[0072] 204: Using the large model in combination with the target script and the target image, a plurality of candidate videos corresponding to the scene are generated, and a scene video matching the target script is determined from the plurality of candidate videos.

[0073] 205: Generate a target video based on the scene video of at least one scene.

[0074] In this embodiment, in order to further improve the quality of the target video, we can first use the big model to generate a target image corresponding to any scene based on the target script; then we can use the big model to generate multiple candidate videos corresponding to the scene based on the target script and the target image.

[0075] That is, the target image of the scene can be generated first, and then the candidate video can be generated by combining the target image. The candidate video can be generated by combining the target script and the target image, which can further ensure the quality of the video generation.

[0076] In some embodiments of the present application, the above-mentioned generation of a target scenario using a large model based on the key information may include:

[0077] Based on the key information, the script content is generated using the big model; based on the script content, the character description information and scene setting information are generated using the big model; based on the script content, the character description information and the scene setting information, the big model is used to generate scene description information for at least one scene.

[0078] In this embodiment, a large model is used to generate script content, and then generate character description information, scene setting information, and scene description information. This helps the large model to accurately understand the characters and plots, accurately identify each scene, and grasp the conversion relationship between scenes, thereby ensuring the quality of the scene video generated based on the character description information, scene setting information, and scene description information in the target script.

[0079] In some embodiments, the method may also utilize the big model to verify the scene description information of at least one scene based on the script content, character description information, and scene setting information, and update the scene description information according to the verification result.

[0080] In some embodiments, generating script content based on key information using a big model may include: generating a first prompt instruction based on the key information and a first generation requirement; inputting the first prompt instruction into the big model to instruct the big model to generate script content based on the key information according to the first generation requirement.

[0081] Among them, the first generation requirement may require the large model to generate script content with rich storylines based on key information. Optionally, the first generation requirement may also require the length of the generated script content, etc.

[0082] For example, the first generation requirement can be: Generate a story based on the key information {input} provided by the user, ensuring that the provided story is about 150 words and rich in plot details. If the story is lacking, it can be enriched by deepening the character interaction, adding dialogues (enclosed in double quotes), describing the actions in detail, and describing emotions. Make necessary adjustments to enhance the continuity and coherence of the plot while retaining the original names of the characters. In addition, consider introducing a key moment or turning point to increase engagement and narrative impact.

[0083] For example, the key information may be filled in the {input} of the first generation requirement to generate a first prompt instruction.

[0084] In some embodiments, using a big model to generate character description information and scene setting information based on script content may include: generating a second prompt instruction based on the script content and a second generation requirement; inputting the second prompt instruction into the big model to instruct the big model to generate character description information and scene setting information based on the script content according to the second generation requirement.

[0085] The second generation requirement may require the large model to extract role description information and scene setting information based on the script content. Optionally, the second generation requirement may also require the large model to generate role description information and scene setting information in a certain format.

[0086] For example, the second generation requirement can be: Generate a list of characters and scene setting information for story {output1}, and provide a detailed description of all characters and scene settings involved. Character description information: Any person or animal that appears multiple times is considered a character. Each character should include a comprehensive description, including appearance, personality traits, and any important actions or roles in the story. Scene setting information: For each scene, use the following format: 'Name (Indoor / Outdoor):', followed by a description of the key elements in a concise style and color. Clearly indicate whether each scene is indoors or outdoors."

[0087] For example, the script content can be filled in {output1} of the second generation requirement to generate a second prompt instruction.

[0088] In some embodiments, using a large model to generate scene description information for at least one scene based on script content, character description information, and scene setting information may include: generating a third prompt instruction based on the script content, character description information, scene setting information, and a third generation requirement; inputting the third prompt instruction into the large model to instruct the large model to generate scene description information for at least one scene based on the script content, character description information, and scene setting information in accordance with the third generation requirement; generating a fourth prompt instruction based on the character description information, scene setting information, scene description information for at least one scene, and a fourth generation requirement; inputting the fourth prompt instruction into the large model to instruct the large model to verify the scene description information for at least one scene based on the character description information and scene setting information in accordance with the fourth generation requirement, and update the scene description information according to the verification result.

[0089] Among them, the third generation requirement may require the big model to generate scene description information for at least one scene based on the script content, character description information and scene setting information. Optionally, the third generation requirement may require the big model to generate scene description information in a certain format.

[0090] For example, the third generation requirement may be: Generate scene description information for multiple scenes (including dialogue, action, and emotion) based on the character description information {output2a} and the scene setting information {output2b}. Describe each scene using the format of "[character description information for this scene][scene setting information for this scene]:scene description information for this scene". Select characters and settings from the provided list, and ensure that all characters and items are named in full form.

[0091] For example, the role description information may be filled in {output2a} of the third generation requirement, and the scene setting information may be filled in {output2b} of the third generation requirement to generate a third prompt instruction.

[0092] Among them, the fourth generation requirement may require the large model to verify the scene description information of at least one scene based on the role description information and the scene setting information, and update the scene description information according to the verification result.

[0093] For example, the fourth generation requirement can be: Check the generated scene description information {output3} based on the character description information {output2a} and scene setting information {output2b}. 1) Review each scene in the "Scene" section, focusing on the characters and scene setting information. If you find any characters or settings that are not listed in the "Character Description Information" or "Scene Setting Information" sections, or any terminology inconsistencies, please answer "Yes". 2) Make sure that every character that needs to appear in the scene illustration is included in the initial set of characters. If any characters are missing, please answer "Yes". If both steps are correct, please reply "No problems found". If your answer is yes. Please revise according to the following requirements: {AAAA}

[0094] For example, the role description information may be filled in {output2a} of the fourth generation requirement, the scene setting information may be filled in {output2b} of the fourth generation requirement, and the scene description information of at least one scene may be filled in {output3} to generate a fourth prompt instruction. The revision requirement may be filled in {AAAA} to update the scene description information.

[0095] Optionally, for any scene, the large model is used to generate a target image corresponding to the scene based on the target script. The large model can be used to generate a target image corresponding to the scene by calling an image generation tool based on the target script.

[0096] The above-mentioned use of the large model to generate multiple candidate videos corresponding to the scene based on the target script and the target image can be: using the large model based on the target script and the target image, calling the video generation tool to generate multiple candidate videos corresponding to the scene.

[0097] The image generation tool and the video generation tool may be, for example, a professional generation tool or a pre-trained model.

[0098] In some embodiments, using a large model in combination with a target script to generate a target image corresponding to any scene may include: for any scene, using the large model based on the target script, calling an image generation tool to generate multiple candidate images corresponding to the scene, and determining a target image that matches the target script from the multiple candidate images.

[0099] The target image corresponding to the scene is generated according to the target script. The candidate video generated based on the target script and the target image is easy to maintain the details of the target image, thereby improving the controllability and accuracy of generating the candidate video. Among them, the target image can be screened from multiple candidate images, which can ensure the quality and effect of the target image, which is conducive to ensuring the quality and effect of the candidate videos, and then ensuring the quality of the target video.

[0100] In some embodiments, the image generation tool may be an image generation model, which may be implemented using a large model, etc. The above-mentioned method of calling the image generation tool to generate multiple candidate images corresponding to the scene for any scene using the large model based on the target script may include: for any scene, generating multiple image generation instructions based on the target script using the large model; calling the image generation tool to generate multiple candidate images based on the multiple image generation instructions.

[0101] Among them, one image generation instruction can be used to instruct the image generation tool to generate a candidate image of any scene, and the large model can call the image generation tool multiple times based on multiple image generation instructions to generate multiple candidate images of any scene.

[0102] For any scene, using the big model to generate multiple image generation instructions based on the target script may include: for any scene, generating a fifth prompt instruction based on the scene description information of at least one scene and the fifth generation requirement; inputting the fifth prompt instruction into the big model to instruct the big model to generate multiple image generation instructions based on the scene description information of at least one scene according to the fifth generation requirement.

[0103] Among them, the fifth generation requirement may require the large model to generate multiple image generation instructions based on the scene description information of the scene.

[0104] For example, the fifth generation requirement can be: Generate image generation instructions for the image generation tool using the scene description information {output4}. The image generation instructions should provide a clear visual description based on the scene description information to help create the scene image. The main instructions include: 1) All characters involved should be renamed according to the following rules: {BBBB} 2) For scene descriptions, simplify the actions and interactions between characters and avoid descriptions that are too complex or difficult to draw.

[0105] For example, the scene description information of at least one scene may be filled in {output4} of the fifth generation requirement to generate a fifth prompt instruction.

[0106] Optionally, in some embodiments, since the target script may include scene description information of each scene, the above-mentioned determining a target image matching the target script from a plurality of candidate images may include: using a large model to determine a target image matching the scene description information from a plurality of candidate images;

[0107] Optionally, for any scene, a sixth prompt instruction may be generated based on the image generation instruction and the image screening requirement; the sixth prompt instruction may be input into the large model to instruct the large model to determine a target image that matches the scene description information from multiple candidate images of the scene according to the image screening requirement.

[0108] For example, the image screening requirement can be: {####} These candidate images numbered 1 to 4 are generated based on the image generation instruction {output5}. You need to answer: Which image most accurately reflects the description of the image generation instruction? Please provide a clear answer in the following format 'The answer is image x', and then explain and analyze your choice. For the selected images, you need to further check the role consistency between the scene image and the role image, and if it is inconsistent, please generate an error symbol.

[0109] For example, multiple candidate images of any scene may be filled in {####} of the image screening requirement, and an image generation instruction may be filled in {output5} of the image screening requirement to generate a sixth prompt instruction.

[0110] In some embodiments, the method may also include: outputting a target image; in response to an image update request for the target image, returning to the step of using the large model based on the target script to call the image generation tool to generate the target image corresponding to the scene and re-executing the step; in response to an image confirmation request for the target image, using the large model based on the target script and the target image to call the video generation tool to generate multiple candidate videos corresponding to the scene.

[0111] By outputting the target image, the user can update the target image, thereby ensuring the quality and effect of the target image, which in turn helps to improve the video quality of the scene video and improve the user experience.

[0112] In some embodiments, for any scene, using the big model based on the target script, calling the image generation tool to generate multiple candidate images corresponding to the scene may include: for any scene, using the big model to call the image generation tool based on the character description information to generate a character image; using the big model to call the image generation tool based on the scene setting information to generate a scene image; using the big model to call the image generation tool to combine the character image, scene image and scene description information to generate multiple candidate images corresponding to the scene.

[0113] Optionally, the above-mentioned image generation tool may be an image generation model, and the image generation model may be obtained by training in the following manner: training the image generation model based on the first sample text and the first sample image corresponding to the first sample text.

[0114] In some embodiments, for any scene, a large model may be used to call a video generation tool based on a target script and a target image to generate multiple candidate videos corresponding to the scene.

[0115] In some embodiments, using a large model in combination with a target script and a target image, calling a video generation tool to generate multiple candidate videos corresponding to a scene may include: using the large model to generate multiple video generation instructions based on the target script and the target image; calling the video generation tool to generate multiple candidate videos based on the multiple video generation instructions.

[0116] Among them, one video generation instruction can be used to instruct the video generation tool to generate a candidate video for any scene, and the large model can call the video generation tool multiple times based on multiple video generation instructions to generate multiple candidate videos for any scene.

[0117] Among them, using the big model to generate multiple video generation instructions based on the target script and the target image may include: generating a seventh prompt instruction based on the scene description information, character description information, target image and the seventh generation requirement of at least one scene; inputting the seventh prompt instruction into the big model to instruct the big model to generate multiple video generation instructions based on the scene description information, character description information and target image of at least one scene according to the seventh generation requirement.

[0118] For example, the seventh generation requirement may be: You need to generate video generation instructions based on the following information: image: {output6}; animation description: {output4}; character description: {output2a}. Your response includes: Part 1, screen description; Part 2, action description.

[0119] For example, the scene description information may be filled into {output4} of the seventh generation requirement, the character description information may be filled into {output2a} of the seventh generation requirement, and the target image may be filled into {output6} of the seventh generation requirement to generate a video generation instruction.

[0120] In some embodiments, the method may further include: generating an eighth prompt instruction based on the video generation instruction, the target image of at least one scene and the eighth generation requirement; inputting the eighth prompt instruction into the large model to instruct the large model to generate video parameter information based on the video generation instruction and the target image of at least one scene according to the eighth generation requirement.

[0121] For example, the eighth generation requirement may be: You need to predict appropriate parameter results in JSON format based on the video generation instruction {output7} and the target image {output6} of at least one scene. You need to fill in the corresponding description and parameter information in the following template according to the specific scene and requirements:

[0122] {

[0123] "description":"Description of the scene",

[0124] "option":{

[0125] "parameters":{

[0126] "motion": "Motion of scene action or character expression",

[0127] "guidanceScale":"Guidance scale for image generation",

[0128] "negativePrompt": "Content that needs to be avoided"

[0129] },

[0130] "camera":{

[0131] "zoom":"in" / "out" / null,"Zoom: inward, outward or no zoom",

[0132] "pan":"left" / "right" / null,"Pan: left, right or no pan",

[0133] "tilt":"up" / "down" / null,"Tilt: up, down or no tilt",

[0134] "rotate":"cw" / "ccw" / null:"Rotate: clockwise, counterclockwise or no rotation",

[0135] }

[0136] }

[0137] }

[0138] For example, the target image of at least one scene may be filled in {output6} of the eighth generation requirement, and the video generation instruction may be filled in {output7} of the eighth generation requirement to generate an eighth prompt instruction.

[0139] Optionally, for any scene, calling a video generation tool to generate multiple candidate videos based on multiple video generation instructions may include: for any scene, inputting multiple video generation instructions, video parameter information and a target image of the scene into the video generation tool to instruct the video generation tool to generate multiple candidate videos based on the video parameter information and the target image of the scene according to the multiple video generation instructions.

[0140] In some embodiments, determining a scene video that matches a target script from a plurality of candidate videos may include: utilizing a large model in combination with the target script and scene videos corresponding to historical scenes to determine a scene video that matches the target script from a plurality of candidate videos.

[0141] Among them, using the big model in combination with the target script and the scene video corresponding to the historical scene, determining the scene video matching the target script from multiple candidate videos may include: generating a ninth prompt instruction based on the scene description information of at least one scene, multiple candidate videos and video screening requirements; inputting the ninth prompt instruction into the big model to instruct the big model to determine the screening scene video matching the target script from multiple candidate videos according to the video screening requirements.

[0142] Optionally, since the target script includes scene description information of each scene, a scene video matching the scene description information may be specifically determined from a plurality of candidate videos.

[0143] For example, the video screening requirement may be: you need to select a better-rated video from multiple candidate videos {####} corresponding to any scene as the scene video based on the scene description information {output4}. The evaluation criterion is to see which video's display action is more consistent with the scene description information.

[0144] For example, the scene description information of at least one scene may be filled in {output4} of the video screening requirement, and the multiple candidate videos corresponding to at least one scene may be filled in {####} of the video screening requirement to generate a ninth prompt instruction.

[0145] Among them, the use of a large model combined with the target script and the scene video corresponding to the historical scene, and the screening of the scene video from multiple candidate videos can ensure the visual quality and continuity of the scene video.

[0146] Optionally, the above-mentioned video generation tool can be a video generation model, and the video generation model can be trained in the following manner: train the video generation model based on the second sample text and the second sample image corresponding to the second sample text, and the sample video corresponding to the second sample text and the second sample image.

[0147] The prompt mentioned in this article is a form of input used to prompt or guide the big model to give expected output. It is used to indicate what action the big model should take or what output it should generate when performing a specific task. The prompt is a natural language input, similar to a command or instruction, to let the big model know what it needs to do. It can be a question, a sentence, or a paragraph, which is used to provide the big model with context and parameter information of the input information, prompting or guiding the big model to give expected output.

[0148] In some embodiments, the method may further include: determining the character dialogue corresponding to any scene; and converting the character dialogue into audio data.

[0149] Generating a target video based on the scene video of at least one scene may include: generating the target video based on the scene video of at least one scene and corresponding audio data, so that the target video may include the audio data.

[0150] Among them, the character dialogue can be determined from the scene description information of any scene. After the character dialogue is converted into audio data, the scene video and corresponding audio data of at least one scene can be merged to generate a target video.

[0151] In some embodiments, converting character dialogues into audio data may include: converting character dialogues into audio data; when the audio duration of the audio data is greater than the video duration of the scene video of the scene, updating the character dialogues according to the video duration using a large model, and re-executing the operation of converting the character dialogues into audio data.

[0152] For example, when the audio duration of the audio data is longer than the video duration of the scene video, the large model can be used to shorten the character dialogue to shorten the audio duration of the character dialogue converted into audio data, so that the audio duration of the audio data matches the duration of the scene video.

[0153] In addition, when the audio duration of the audio data is less than the video duration of the scene video, the large model can be used to expand the character dialogue to increase the audio duration of the character dialogue converted into audio data so that the audio duration of the audio data matches the duration of the scene video.

[0154] Figure 3 This is a flowchart of an embodiment of a video generation method provided by the present application. The technical solution of this embodiment can be executed by a user end. The method may include the following steps:

[0155] 301: Get key information provided by the user.

[0156] As can be seen from the foregoing description, the key information may refer to a short story or an instruction.

[0157] 302: In response to the video generation operation triggered by the key information, generate a video generation request.

[0158] Optionally, the user terminal may provide a user interface, and information input prompt information may be displayed in the user interface, for example, it may be an input box for the user to input key information.

[0159] In addition, the video generation prompt information may also be displayed in the user interface, so that step 201 may be a response to a video generation operation triggered by the video generation prompt information, and a video generation request may be generated based on the key information.

[0160] 303: Send a video generation request to the server.

[0161] Among them, the video generation request is used to trigger the server to generate a target script based on key information using a large model and generate multiple candidate videos corresponding to any scene in combination with the target script, and determine the scene video matching the target script from the multiple candidate videos; and generate a target video based on the scene videos of at least one scene involved in the target script.

[0162] Optionally, the video generation request can be specifically used to trigger the server to generate a target script based on the key information using the big model and generate a target image corresponding to any scene in combination with the target script, and to generate multiple candidate videos corresponding to the scene using the big model in combination with the target script and the target image, and to determine a scene video that matches the target script from the multiple candidate videos, and to generate a target video based on the scene videos of at least one scene involved in the target script.

[0163] The specific implementation method of the server generating the target video in response to the video generation request can be found in the above Figure 1 As described in the illustrated embodiment, details will not be repeated here.

[0164] In this embodiment, the user side generates a video generation request in response to the video generation operation; sends the video generation request to the server side, wherein the video generation request is used to trigger the server side to generate a target script based on key information using a large model and generate multiple candidate videos corresponding to any scene in combination with the target script, and use the large model in combination with the target script to filter and obtain a scene video from the multiple candidate videos; and generate a target video based on the scene video of at least one scene involved in the target script. In the embodiment of the present application, the user only needs to provide key information such as a short story or instruction, that is, the target script can be generated using the large model, without the need for multiple user interactions, which can reduce manual creation and improve creation efficiency. After that, the large model is directly used to generate multiple candidate videos in combination with the target script, and the scene video is filtered from the generated multiple candidate videos, which can avoid the complex process of manual correction and manual construction of roles and scenes, shorten the video generation cycle, and improve generation efficiency. In the embodiment of the present application, the script generation, video generation, and video screening operations are automatically realized with the help of the large model, which realizes the automation of video generation, reduces manual resources, reduces video production costs, and improves video generation efficiency. In addition, in the embodiment of the present application, video screening can be performed with the help of the large model to ensure the quality and effect of the video.

[0165] In some embodiments, the method may further include: obtaining video prompt information of the target video from the server, so as to output the video prompt information. The user end may display the video prompt information of the target video on a user interface.

[0166] The video prompt information may be, for example, a thumbnail diagram corresponding to the target video, so that in response to a trigger operation on the video prompt information, the target video may be obtained from the server and played.

[0167] In addition, the video prompt information may also include the storage address of the target video, so that the user terminal can download the target video from the storage address.

[0168] After the server generates the target video, it can save the target video and generate a storage address and feed it back to the user. Optionally, the server can save the target video to a network storage system, etc., which is not limited in this application.

[0169] In some embodiments, the method may also include: displaying a target script and script confirmation prompt information in a user interface; generating a script confirmation request in response to a trigger operation for the script confirmation prompt information, and sending the script confirmation request to the server; so that the server can respond to the script confirmation request and generate a target image corresponding to any scene based on the target script using a large model.

[0170] In some embodiments, the method may also include: displaying a scene video and video confirmation prompt information corresponding to any scene in a user interface; generating a video confirmation request in response to a trigger operation for the video confirmation prompt information, and sending the video confirmation request to the server; so that the server can respond to the video confirmation request and generate a target video based on the scene video of at least one scene.

[0171] In some embodiments, the method may also include: displaying script update prompt information for the target script in a user interface; generating a script update request in response to a trigger operation for the script update prompt information, and sending the script update request to a server; so that the server can respond to the script update request and re-execute the generation of the target script using the large model.

[0172] In some embodiments, the method may also include: displaying video update prompt information for the scene video of any scene in the user interface; generating a video update request in response to a trigger operation for the video update prompt information, and sending the video update request to the server; so that the server can respond to the video update request and return to the step of using the large model combined with the target script to generate multiple candidate videos corresponding to any scene and re-execute it.

[0173] In addition, as can be seen from the previous description, the server can first use the large model to generate a target image of any scene based on the target script, and then generate multiple candidate videos based on the target image.

[0174] In some embodiments, the method may further include: displaying image update prompt information for a target image of any scene in a user interface; generating an image update request in response to a trigger operation for the image update prompt information, and sending the image update request to a server; so that the server can respond to the image update request and reuse the large model to generate the target image.

[0175] In addition, image confirmation prompt information for the target image can also be displayed, thereby generating an image confirmation request in response to a trigger operation for the image confirmation prompt information, and sending the image confirmation request to the server, so that the server can respond to the image confirmation request and generate multiple candidate videos based on the target image.

[0176] By displaying the script confirmation prompt information, image confirmation prompt information, video confirmation prompt information, script update prompt information, image update prompt information and video update prompt information on the user interface, the user can update and confirm the target script, target image and scene video, ensure that the target script, target image and scene video meet the user's expectations, ensure the quality of the finally generated target video, and improve the user experience.

[0177] For ease of understanding, Figure 4 A schematic diagram of scene interaction in an actual application of an embodiment of the present application is shown. Figure 4 As shown, the client 401 can establish a connection with the server 402 through a network.

[0178] The user end 401 may generate a video generation request in response to the video generation operation and send it to the server end 402. The server end 402 may generate a target script using the large model 400 in response to the video generation request. The target script may include role description information, scene setting information, and scene description information of at least one scene.

[0179] Specifically, see Figure 5 In the video generation diagram shown:

[0180] Step 501: The server 402 can determine the key information provided by the user in response to the video generation request; generate a first prompt instruction based on the key information and the first generation requirement; input the first prompt instruction into the big model 400 to instruct the big model 400 to generate the script content based on the key information according to the first generation requirement.

[0181] Step 502: A second prompt instruction may be generated based on the script content and the second generation requirement, and the second prompt instruction may be input into the large model 400 to instruct the large model 400 to generate role description information and scene setting information based on the script content according to the second generation requirement. Afterwards, the large model 400 may be used to generate scene description information for at least one scene based on the script content, role description information, and scene setting information; a third prompt instruction may be generated based on the script content, role description information, scene setting information, and the third generation requirement; and the third prompt instruction may be input into the large model 400 to instruct the large model 400 to generate scene description information for at least one scene based on the script content, role description information, and scene setting information according to the third generation requirement.

[0182] Afterwards, a fourth prompt instruction can be generated based on the role description information, scene setting information, scene description information of at least one scene and a fourth generation requirement; the fourth prompt instruction is input into the big model 400 to instruct the big model 400 to verify the scene description information of at least one scene based on the script content, role description information and scene setting information according to the fourth generation requirement, and update the scene description information according to the verification result.

[0183] Among them, the server can send the target script to the user terminal 401, so that the user terminal 401 can display the target script in the user interface. The user terminal 401 can also display the script confirmation prompt information and the script update prompt information for the target script in the user interface. The user terminal 401 can generate a script update request in response to the trigger operation for the script update prompt information, and send the script update request to the server 402. The server 402 can respond to the script update request and re-execute the target script generated using the large model 400.

[0184] The user end 401 can generate a script confirmation request in response to a trigger operation for the script confirmation prompt information, and send the script confirmation request to the server end 402. The server end 402 can respond to the script confirmation request, use the large model 400 based on the target script, and call the image generation tool to generate a target image corresponding to any scene.

[0185] Specifically, see Figure 5 As shown, step 503: the server 402 can generate a fifth prompt instruction for any scene based on the scene description information of at least one scene and the fifth generation requirement; and input the fifth prompt instruction into the large model 400 to instruct the large model 400 to generate multiple image generation instructions based on the scene description information of at least one scene according to the fifth generation requirement.

[0186] Step 504: The server 402 can use the large model 400 to input multiple image generation instructions into the image generation tool for any scene, so as to instruct the image generation tool to generate multiple candidate images corresponding to the scene according to the image generation instructions. For any scene, based on the image generation instructions and the image screening requirements, a sixth prompt instruction is generated; the sixth prompt instruction is input into the large model 400 to instruct the large model 400 to screen the target image from the multiple candidate images of the scene according to the image screening requirements.

[0187] Among them, the server 402 can send the target image corresponding to any scene to the user terminal 401, so that the user terminal 401 can display the target image corresponding to any scene in the user interface. The user terminal 401 can display image confirmation prompt information and image update prompt information for the target image in the user interface. The user terminal generates an image update request in response to the trigger operation for the image update prompt information, and sends the image update request to the server 402. The server 402 can respond to the image update request and return to the step of using the large model 400 based on the target script to call the image generation tool to generate the target image corresponding to any scene and re-execute it.

[0188] The user end 401 can generate an image confirmation request in response to a trigger operation for the image confirmation prompt information, and send the image confirmation request to the server end 402. The server end 402 can respond to the image confirmation request and use the large model 400 to call the video generation tool to generate multiple candidate videos corresponding to the scene based on the target script and the target image.

[0189] Specifically, Figure 5 As shown, step 505: the server 402 can generate a seventh prompt instruction based on the scene description information, role description information, target image and the seventh generation requirement of at least one scene; input the seventh prompt instruction into the large model 400 to instruct the large model 400 to generate multiple video generation instructions based on the scene description information, role description information and target image of at least one scene according to the seventh generation requirement. Generate an eighth prompt instruction based on the video generation instruction, the target image of at least one scene and the eighth generation requirement; input the eighth prompt instruction into the large model 400 to instruct the large model 400 to generate video parameter information based on the video generation instruction and the target image of at least one scene according to the eighth generation requirement.

[0190] Step 506: The server 402 can use the large model 400 to input multiple video generation instructions, video parameter information, and the target image of the scene into the video generation tool for any scene, so as to instruct the video generation tool to generate multiple candidate videos based on the video parameter information and the target image of the scene according to the multiple video generation instructions. Based on the scene description information of at least one scene, the multiple candidate videos, and the video screening requirements, a ninth prompt instruction is generated; the ninth prompt instruction is input into the large model 400 to instruct the large model 400 to screen the scene video from the multiple candidate videos according to the video screening requirements.

[0191] In addition, the server 402 can send the scene video of the scene to the user terminal 401, so that the user terminal 401 can display the scene video in the user interface. The user terminal 401 can also display video confirmation prompt information and video update prompt information for the scene video in the user interface. The user terminal 401 can generate an update request in response to the video update operation triggered by the user for the video update prompt information, and send the video update request to the server. The server can respond to the video update request for the scene video and return to the step of using the large model 400 in combination with the target script to generate multiple candidate videos corresponding to any scene and re-execute it.

[0192] The user end 401 can generate a video confirmation request in response to a trigger operation for the video confirmation prompt information, and send the video confirmation request to the server end 402. The server end 402 can generate a target video based on the scene video of each of the at least one scene in response to the video confirmation request, and send the target video to the user end 401.

[0193] In the embodiment of the present application, the use of a large model to generate a target script can reduce manual creation and improve creation efficiency. After that, the large model is directly used in combination with the target script to generate multiple candidate videos, and scene videos are obtained by screening from the generated multiple candidate videos. The complex process of manual correction and manual construction of roles and scenes can be avoided, shortening the video generation cycle and improving generation efficiency. In the embodiment of the present application, the script generation, video generation, and video screening are automatically realized with the help of a large model, and the automation of video generation is realized, which reduces manual resources, reduces video production costs, and improves video generation efficiency. In addition, in the embodiment of the present application, video screening can be performed with the help of a large model to ensure the quality and effect of the video.

[0194] Figure 6 A schematic diagram of a structure of an embodiment of a video generation device provided in an embodiment of the present application, the method device includes:

[0195] The first generation module 601 is used to generate a target script using a large model in response to a video generation request.

[0196] Among them, the target script may involve at least one scene.

[0197] The second generation module 602 is used to use the large model in combination with the target script to generate multiple candidate videos corresponding to any scene, and determine the scene video matching the target script from the multiple candidate videos.

[0198] The third generating module 603 is used to generate a target video based on the scene video of at least one scene.

[0199] In some embodiments, the device may further include:

[0200] Output module, used to output the target script.

[0201] The first generating module may update the target script in response to a script update request for the target script.

[0202] The second generation module can respond to a script confirmation request for a target script and generate multiple candidate videos corresponding to any scene in combination with the target script.

[0203] The output module can also be used to output scene videos;

[0204] The second generation module can respond to the video update request for the scene video and return to the step of using the large model in combination with the target script to generate multiple candidate videos corresponding to any scene and re-execute it.

[0205] The third generating module generates a target video based on the scene video of at least one scene in response to a video confirmation request for the scene video.

[0206] The first generation module generates a target script using a large model in response to a video generation request, which may include: determining key information provided by a user in response to the video generation request; and generating a target script using the large model based on the key information.

[0207] In some embodiments, the first generation module generates a target script using a big model based on key information, which may include: generating script content using the big model based on the key information; generating character description information and scene setting information based on the script content using the big model; and generating scene description information for at least one scene based on the script content, the character description information and the scene setting information.

[0208] In some embodiments, the device may further include:

[0209] The verification and updating module is used to verify the scene description information of at least one scene based on the script content, character description information and scene setting information using the big model, and update the scene description information according to the verification result.

[0210] In some embodiments, the first generation module generates script content based on key information using a large model, which may include: generating a first prompt instruction based on the key information and a first generation requirement; and inputting the first prompt instruction into the large model to instruct the large model to generate script content based on the key information according to the first generation requirement.

[0211] In some embodiments, the first generation module generates character description information and scene setting information based on the script content using the big model, which may include: generating a second prompt instruction based on the script content and a second generation requirement; inputting the second prompt instruction into the big model to instruct the big model to generate the character description information and scene setting information based on the script content according to the second generation requirement.

[0212] In some embodiments, the first generation module generates scene description information for each of the at least one scene using the big model based on the script content, character description information and scene setting information, which may include: generating a third prompt instruction based on the script content, character description information, scene setting information and a third generation requirement; inputting the third prompt instruction into the big model to instruct the big model to generate scene description information for each of the at least one scene based on the script content, character description information and scene setting information in accordance with the third generation requirement; generating a fourth prompt instruction based on the character description information, scene setting information, scene description information for each of the at least one scene and a fourth generation requirement; inputting the fourth prompt instruction into the big model to instruct the big model to verify the scene description information for each of the at least one scene based on the character description information and scene setting information in accordance with the fourth generation requirement, and update the scene description information according to the verification result.

[0213] In some embodiments, the second generation module uses the big model in combination with the target script to generate multiple candidate videos corresponding to any scene, which may include: for any scene, using the big model to generate a target image corresponding to the scene based on the target script; using the big model to generate multiple candidate videos corresponding to the scene based on the target script and the target image.

[0214] That is, the target image of the scene can be generated first, and then the candidate video can be generated by combining the target image. The candidate video can be generated by combining the target script and the target image, which can further ensure the quality of the video generation.

[0215] Optionally, the second generation module can generate a target image corresponding to any scene by using the large model based on the target script. The large model can be used to generate a target image corresponding to the scene by calling an image generation tool based on the target script.

[0216] The above-mentioned use of the big model based on the target script and the target image to generate multiple candidate videos corresponding to the scene can be: using the big model in combination with the target script and the target image, calling the video generation tool to generate multiple candidate videos corresponding to the scene.

[0217] In some embodiments, the second generation module may specifically target any scene, use the big model based on the target script, call the image generation tool to generate multiple candidate images corresponding to the scene; and use the big model to filter the target image from the multiple candidate images.

[0218] In some embodiments, the image generation tool may be an image generation model, which may be implemented using a large model, etc. The above-mentioned second generation module, for any scene, uses the large model based on the target script, and calls the image generation tool to generate multiple candidate images corresponding to the scene, which may include: for any scene, using the large model based on the target script, generating multiple image generation instructions; calling the image generation tool to generate multiple candidate images based on the multiple image generation instructions.

[0219] Among them, one image generation instruction can be used to instruct the image generation tool to generate a candidate image of any scene, and the large model can call the image generation tool multiple times based on multiple image generation instructions to generate multiple candidate images of any scene.

[0220] The second generation module generates multiple image generation instructions for any scene using the big model based on the target script, which may include: generating a fifth prompt instruction for any scene based on the scene description information of at least one scene and the fifth generation requirement; inputting the fifth prompt instruction into the big model to instruct the big model to generate multiple image generation instructions based on the scene description information of at least one scene in accordance with the fifth generation requirement.

[0221] Optionally, the second generation module can specifically generate a sixth prompt instruction for any scene based on the image generation instruction and the image screening requirement; and input the sixth prompt instruction into the large model to instruct the large model to screen the target image from multiple candidate images of the scene according to the image screening requirement.

[0222] In some embodiments, the output module can also be used to output the target image; the second generation module can respond to an image update request for the target image, return to the step of using the large model based on the target script to call the image generation tool to generate the target image corresponding to the scene and re-execute it; in response to an image confirmation request for the target image, use the large model based on the target script and the target image to call the video generation tool to generate multiple candidate videos corresponding to the scene.

[0223] In some embodiments, the second generation module, for any scene, uses the large model based on the target script to call the image generation tool to generate multiple candidate images corresponding to the scene, which may include: for any scene, using the large model to call the image generation tool based on the character description information to generate a character image; using the large model to call the image generation tool based on the scene setting information to generate a scene image; using the large model to call the image generation tool to combine the character image, scene image and scene description information to generate multiple candidate images corresponding to the scene.

[0224] Optionally, the above-mentioned image generation tool may be an image generation model, and the image generation model may be obtained by training in the following manner: training the image generation model based on the first sample text and the first sample image corresponding to the first sample text.

[0225] In some embodiments, the second generation module uses the big model in combination with the target script to generate multiple candidate videos corresponding to any scene, which may include: for any scene, using the big model based on the target script, calling the image generation tool to generate the target image corresponding to the scene; using the big model based on the target script and the target image, calling the video generation tool to generate multiple candidate videos corresponding to the scene.

[0226] In some embodiments, the second generation module uses the large model based on the target script and the target image, and calls the video generation tool to generate multiple candidate videos corresponding to the scene, which may include: using the large model based on the target script and the target image, generating multiple video generation instructions; calling the video generation tool to generate multiple candidate videos based on the multiple video generation instructions.

[0227] The video generation instruction may be used to instruct a video generation tool to generate a candidate video for any scene, and multiple video generation instructions may instruct the video generation tool multiple times to generate multiple candidate videos for any scene.

[0228] Among them, the second generation module uses the big model to generate multiple video generation instructions based on the target script and the target image, which may include: generating a seventh prompt instruction based on the scene description information, character description information, target image and the seventh generation requirement of at least one scene; inputting the seventh prompt instruction into the big model to instruct the big model to generate multiple video generation instructions based on the scene description information, character description information and target image of at least one scene according to the seventh generation requirement.

[0229] In some embodiments, the second generation module can also be used to generate an eighth prompt instruction based on the video generation instruction, the target image of at least one scene and the eighth generation requirement; and input the eighth prompt instruction into the large model to instruct the large model to generate video parameter information based on the video generation instruction and the target image of at least one scene according to the eighth generation requirement.

[0230] Optionally, for any scene, calling a video generation tool to generate multiple candidate videos based on multiple video generation instructions may include: for any scene, inputting multiple video generation instructions, video parameter information and a target image of the scene into the video generation tool to instruct the video generation tool to generate multiple candidate videos based on the video parameter information and the target image of the scene according to the multiple video generation instructions.

[0231] In some embodiments, the second generation module uses the large model in combination with the target script to determine the scene video that matches the target script from multiple candidate videos, which may include: using the large model in combination with the target script and the scene video corresponding to the historical scene, to determine the scene video that matches the target script from multiple candidate videos.

[0232] Optionally, the above-mentioned video generation tool can be a video generation model, and the video generation model can be trained in the following manner: train the video generation model based on the second sample text and the second sample image corresponding to the second sample text, and the sample video corresponding to the second sample text and the second sample image.

[0233] In some embodiments, the target video may include audio data. The apparatus may also include:

[0234] The audio conversion module is used to determine the character dialogue corresponding to any scene; and convert the character dialogue into audio data.

[0235] The third generating module may generate the target video based on the scene video of at least one scene, and may include: generating the target video based on the scene video of at least one scene and the corresponding audio data.

[0236] In some embodiments, the audio conversion module converts the character dialogue into audio data, which may include: converting the character dialogue into audio data; when the audio duration of the audio data is greater than the video duration of the scene video of the scene, updating the character dialogue according to the video duration using the large model, and re-executing the operation of converting the character dialogue into audio data.

[0237] For example, when the audio duration of the audio data is longer than the video duration of the scene video, the large model can be used to shorten the character dialogue to shorten the audio duration of the character dialogue converted into audio data, so that the audio duration of the audio data matches the duration of the scene video.

[0238] In addition, when the audio duration of the audio data is less than the video duration of the scene video, the large model can be used to expand the character dialogue to increase the audio duration of the character dialogue converted into audio data so that the audio duration of the audio data matches the duration of the scene video.

[0239] Figure 6 The video generation device can execute Figure 1 The implementation principle and technical effect of the video generation method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the video generation device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0240] Figure 7A schematic diagram of a structure of an embodiment of a video generation device provided in an embodiment of the present application, the method device includes:

[0241] The request generation module 701 is used to obtain key information provided by the user and generate a video generation request in response to a video generation operation triggered by the key information.

[0242] The sending module 702 is used to send the video generation request to the server.

[0243] Among them, the video generation request is used to trigger the server to use the big model to generate the target script and combine the target script to generate multiple candidate videos corresponding to any scene, and use the big model combined with the target script to filter and obtain the scene video from the multiple candidate videos; wherein the target script involves at least one scene; the scene videos of each of the at least one scene constitute the target video.

[0244] In some embodiments, the device may further include:

[0245] The acquisition module is used to obtain the video prompt information of the target video from the server and output the video prompt information.

[0246] In some embodiments, the device may further include:

[0247] The display module is used to display the target script and script confirmation prompt information in the user interface; generate a script confirmation request in response to the trigger operation of the script confirmation prompt information, and send the script confirmation request to the server; so that the server can respond to the script confirmation request and generate a target image corresponding to any scene based on the target script using the large model.

[0248] In some embodiments, the display module can be used to display the scene video and video confirmation prompt information corresponding to any scene in the user interface; generate a video confirmation request in response to a trigger operation for the video confirmation prompt information, and send the video confirmation request to the server; so that the server can respond to the video confirmation request and generate a target video based on the scene video of at least one scene.

[0249] In some embodiments, the display module can be used to display script update prompt information for the target script in a user interface; generate a script update request in response to a trigger operation for the script update prompt information, and send the script update request to a server; so that the server can respond to the script update request and re-execute the generation of the target script using the large model.

[0250] In some embodiments, the display module can be used to display video update prompt information for the scene video of any scene in the user interface; generate a video update request in response to a trigger operation for the video update prompt information, and send the video update request to the server; so that the server can respond to the video update request and return to the step of using the large model combined with the target script to generate multiple candidate videos corresponding to any scene and re-execute it.

[0251] In some embodiments, the display module can be used to display image update prompt information for a target image of any scene in a user interface; generate an image update request in response to a trigger operation for the image update prompt information, and send the image update request to a server; so that the server can respond to the image update request and reuse the large model to generate the target image.

[0252] In some embodiments, the display module can be used to display image confirmation prompt information for the target image in the user interface, thereby generating an image confirmation request in response to a trigger operation for the image confirmation prompt information, and sending the image confirmation request to the server, so that the server can respond to the image confirmation request and then generate multiple candidate videos based on the target image.

[0253] Figure 7 The video generation device can execute Figure 3 The implementation principle and technical effect of the video generation method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the video generation device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0254] The present application also provides a computing device, such as Figure 8 As shown, the device may include a storage component 801 and a processing component 802;

[0255] The storage component 801 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component to implement the following Figure 1 The video generation method described in the embodiment shown or Figure 2 The video generation method described in the illustrated embodiment.

[0256] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0257] The display component in the computing device is used to implement the following Figure 3 In the case of the video generation method described in the illustrated embodiment, the computing device may further include a display component to perform corresponding display operations.

[0258] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0259] The processing component 802 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0260] The storage component 801 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0261] The display component may be an electroluminescent (EL) element, a liquid crystal display or a micro display having a similar structure, or a retinal direct display or a similar laser scanning display.

[0262] It should be noted that the above computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. It can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0263] It should be noted that the above computing device is implemented as follows Figure 1 or Figure 2 In the case of the video generation method described in the embodiment shown, it can be a physical device or an elastic computing host provided by a cloud computing platform. It can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. The above computing device is implemented as follows Figure 3 In the case of the video generation method described in the illustrated embodiment, it can be specifically implemented as an electronic device. The electronic device can refer to a device used by a user and having the computing, Internet access, communication and other functions required by the user, such as a mobile phone, tablet computer, personal computer, wearable device, etc.

[0264] The present application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a computer, the above-mentioned Figure 1 or Figure 2 The video generation method described in the embodiment shown or Figure 3 The video generation method described in the illustrated embodiment. The computer readable medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device.

[0265] The present application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium. When the computer program is executed by a computer, the computer program can achieve the above-mentioned Figure 1 or Figure 2 The video generation method described in the embodiment shown or Figure 3 The video generation method described in the illustrated embodiment. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, various functions defined in the system of the present application are executed.

[0266] In the above corresponding embodiments, the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0267] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0268] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0269] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0270] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video generation method, characterized in that: include: In response to the video generation request, determining key information provided by the user; Based on the key information, a target script is generated using a large model; The target script involves at least one scene; Using the large model in combination with the target script, a target image corresponding to any scene is generated; Using the large model in combination with the target script and the target image, a plurality of candidate videos corresponding to the scene are generated, and a scene video matching the target script is determined from the plurality of candidate videos; A target video is generated based on the scene video of each of the at least one scene.

2. The method according to claim 1, characterized in that Based on the key information, generating the target script using the large model includes: Based on the key information, the script content is generated using the big model; Based on the script content, using the large model to generate role description information and scene setting information; Based on the script content, the character description information and the scene setting information, the big model is used to generate scene description information for at least one scene.

3. The method according to claim 2, characterized in that Also includes: The large model is used to verify the scene description information of each of the at least one scene based on the script content, the role description information and the scene setting information, and the scene description information is updated according to the verification result.

4. The method according to claim 2, characterized in that: Based on the key information, the script content is generated by using the big model, including: Based on the key information and the first generation requirement, generate a first prompt instruction; Inputting the first prompt instruction into the big model to instruct the big model to generate script content based on the key information according to the first generation requirement; The method of generating role description information and scene setting information based on the script content using the large model includes: Based on the script content and the second generation requirement, generate a second prompt instruction; Inputting the second prompt instruction into the large model to instruct the large model to generate role description information and scene setting information based on the script content according to the second generation requirement; The step of generating scene description information of at least one scene based on the script content, the role description information and the scene setting information by using the large model comprises: Generate a third prompt instruction based on the script content, the role description information, the scene setting information and the third generation requirement; Inputting the third prompt instruction into the large model to instruct the large model to generate scene description information of at least one scene according to the third generation requirement based on the script content, the role description information and the scene setting information; Generate a fourth prompt instruction based on the role description information, the scene setting information, the scene description information of each of the at least one scene, and a fourth generation requirement; The fourth prompt instruction is input into the big model to instruct the big model to verify the scene description information of at least one scene based on the script content, the character description information and the scene setting information according to the fourth generation requirement, and update the scene description information according to the verification result.

5. The method according to claim 1, characterized in that: The method of using the large model in combination with the target script to generate a target image corresponding to any scene includes: For any scene, the large model is used to call an image generation tool based on the target script to generate multiple candidate images corresponding to the scene, and a target image matching the target script is determined from the multiple candidate images.

6. The method according to claim 5, characterized in that For any scene, using the large model based on the target script, calling an image generation tool to generate a plurality of candidate images corresponding to the scene includes: For any scene, using the large model to generate multiple image generation instructions based on the target script; An image generation tool is called to generate a plurality of candidate images based on the plurality of image generation instructions.

7. The method according to claim 5, characterized in that The target script includes character description information, scene setting information, and scene description information of at least one scene; For any scene, using the large model based on the target script, calling an image generation tool to generate a plurality of candidate images corresponding to the scene includes: For any scene, using the large model to call an image generation tool to generate a character image based on the character description information; Using the large model to call the image generation tool to generate a scene image based on the scene setting information; The image generation tool is called by using the large model to combine the character image, the scene image and the scene description information to generate a plurality of candidate images corresponding to the scene.

8. The method according to claim 1, characterized in that: The generating of a plurality of candidate videos corresponding to the scene by using the large model in combination with the target script and the target image comprises: Generate a plurality of video generation instructions based on the target script and the target image using the large model; A video generation tool is called to generate a plurality of candidate videos based on the plurality of video generation instructions.

9. The method according to claim 1, characterized in that: Determining a scene video matching the target script from the plurality of candidate videos comprises: The large model is used in combination with the target script and the scene video corresponding to the historical scene to determine the scene video matching the target script from the multiple candidate videos.

10. The method according to claim 5, characterized in that The image generation tool is an image generation model, and the image generation model is trained in the following manner: An image generation model is trained based on a first sample text and a first sample image corresponding to the first sample text.

11. The method according to claim 8, characterized in that The video generation tool is a video generation model, and the video generation model is trained in the following manner: A video generation model is trained based on a second sample text and a second sample image corresponding to the second sample text, and a sample video corresponding to the second sample text and the second sample image.

12. The method according to claim 1, characterized in that Also includes: Determine the character dialogue corresponding to any scene; Converting the character dialogue into audio data; The generating a target video based on the scene video of each of the at least one scene comprises: A target video is generated based on the scene video of each of the at least one scene and the corresponding audio data.

13. The method according to claim 12, characterized in that The converting the character dialogue into audio data comprises: Converting the character dialogue into audio data; In the case where the audio duration of the audio data is greater than the video duration of the scene video of the scene, the character dialogue is updated according to the video duration using the large model, and the operation of converting the character dialogue into audio data is re-executed.

14. A video generation method, characterized in that: include: Obtain key information provided by users; In response to a video generation operation triggered by the key information, generating a video generation request; Sending the video generation request to the server; The video generation request is used to trigger the server to generate a target script based on the key information using the big model and generate a target image corresponding to any scene in combination with the target script, and to generate multiple candidate videos corresponding to the scene using the big model in combination with the target script and the target image, and to determine the scene video that matches the target script from the multiple candidate videos, and to generate a target video based on the scene videos of each of at least one scene involved in the target script.

15. The method according to claim 14, characterized in that Also includes: Obtaining video prompt information of the target video from the server; Output the video prompt information.

16. The method according to claim 14, characterized in that Also includes: Displaying the target script and script update prompt information and script confirmation prompt information for the target script in a user interface; generating a script update request in response to a trigger operation for the script update prompt information, and sending the script update request to the server so that the server can update the target script; Generate a script confirmation request in response to a trigger operation for the script confirmation prompt information, and send the script confirmation request to the server, so that the server responds to the script confirmation request and generates a plurality of candidate videos corresponding to any scene in combination with the target script; Displaying a scene video of any scene and video update prompt information and video confirmation prompt information for the scene video in the user interface; generating a video update request in response to a trigger operation for the video update prompt information, and sending the video update request to the server so that the server can regenerate the scene video; A video confirmation request is generated in response to a trigger operation for the video confirmation prompt information, and the video confirmation request is sent to the server, so that the server constructs a target video based on the scene video in response to the video confirmation request.

17. A video generation method, characterized in that: include: In response to a video generation request, generating a target script using the large model; The target script involves at least one scene; Using the large model in combination with the target script, a plurality of candidate videos corresponding to any scene are generated, and a scene video matching the target script is determined from the plurality of candidate videos; A target video is generated based on the scene video of each of the at least one scene.

18. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the video generation method as described in any one of claims 1 to 13 or the video generation method as described in any one of claims 14 to 16 or the video generation method as described in claim 17.

19. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, the video generation method according to any one of claims 1 to 13, the video generation method according to any one of claims 14 to 16, or the video generation method according to claim 17 is implemented.

20. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a computer, implements the video generation method according to any one of claims 1 to 13, the video generation method according to any one of claims 14 to 16, or the video generation method according to claim 17.

Images