Video generation method and electronic equipment

Abstract

The embodiment of the invention discloses a video generation method and electronic equipment. The method comprises the following steps: creating a 3D virtual film field environment based on a 3D development engine; after a video generation demand text is received, a script is generated through mutual cooperation among multiple agents based on an artificial intelligence AI large model, the multiple agents are used for simulating multiple function posts involved in the manual video production process, and the script comprises role brief introductions in the script. A line text and role positions, actions and lens types corresponding to lines; and performing structured processing on the script, and simulating a video shooting process in the 3D virtual film field environment according to the structured script so as to generate a target video. According to the embodiment of the invention, the video generation quality and the character consistency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of video generation, and in particular to a method and electronic device for generating video. Background Art

[0002] Virtual production is a production method that uses virtual environments and digital technology to create film and television content. It can greatly reduce costs and improve efficiency. It has been widely used in the filming of movies and TV series. With the rapid development of artificial intelligence technology in recent years, people have begun to think about replacing some manual operations in the virtual film production process with automated processes in order to further reduce the workload of film production.

[0003] There are two ways to apply artificial intelligence to virtual production in the prior art. One is to directly use a generative model such as "vintage video" to generate videos. For example, if a piece of text is input, the model can generate a video based on the text. However, this large model of Vintage video can usually only generate videos within 10 seconds, and the video quality is low. The generated video may also violate common sense, have poor character consistency (the same character may have inconsistent appearance features in different video frames, etc.), and have no storyline.

[0004] Another way is to generate videos by planning plots, connecting picture generation and video generation tools based on the above-mentioned big model based on Vincent video. For example, first use the big model of "vincent text" to generate a script, then use the big model of "vincent picture" to generate some pictures for different scenes in the script, that is, similar to the director's storyboard, and then give the pictures to the big model of "picture and text to picture" to enrich some details, generate higher quality pictures, and then turn the pictures into animated pictures. Then splice these short-duration videos together to get the final generated video. However, this method only solves the problem of the storyline, and the quality of video generation is still limited by the base model. And because the video content is generated in segments, it is difficult to maintain consistency between the characters in different video segments. Summary of the invention

[0005] The present application provides a method and electronic device for generating videos, which can improve the quality of video generation and the consistency of characters.

[0006] This application provides the following solutions:

[0007] A method for generating a video, comprising:

[0008] Creating a 3D virtual studio environment based on a 3D development engine, wherein the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions, and shot types in the scene;

[0009] After receiving the video generation requirement text, a script is generated through the mutual collaboration between multiple intelligent agents based on an artificial intelligence AI big model, wherein the AI ​​big model is an AI big model of text generation text type, and the multiple intelligent agents are used to simulate multiple functional positions involved in the process of manually producing videos, and the script includes a brief introduction of the characters in the script, the lines text, and the lines annotation text information, and the lines annotation text information includes the character position, action, and shot type corresponding to the lines;

[0010] The script is structured and a video shooting process is simulated in the 3D virtual studio environment according to the structured script to generate a target video.

[0011] The script generation through the mutual collaboration between multiple intelligent agents based on the artificial intelligence AI big model includes:

[0012] By simulating the process of artificial video production, the script generation process is divided into multiple continuous and interdependent stages, and the script is generated through the mutual collaboration between the multiple intelligent agents in each stage.

[0013] The multiple functional positions involved in the artificial video production process simulated by the multiple agents include at least: director, screenwriter and photographer; the multiple continuous and interdependent stages include: planning, script writing, photography;

[0014] The generating of the script through the mutual collaboration among the multiple intelligent agents at each stage includes:

[0015] In the planning stage, the video generation requirement text input by the user is provided to the director agent corresponding to the director position, so that the director agent can plan the roles involved in the script and their introductions, as well as the script outline;

[0016] In the script creation stage, the planning results of the director agent and the information of the positions and action types in the 3D virtual studio environment are provided to the screenwriter agent corresponding to the screenwriter position, so that the screenwriter agent can generate the lines in the script and mark the positions and action types of the characters in the script.

[0017] During the photography stage, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment are provided to the photographer agent corresponding to the photographer position, so that the photographer agent can mark the corresponding lens type for the lines in the script.

[0018] Among them, it also includes:

[0019] In the script creation stage, the script generated by the screenwriter agent is criticized, corrected, and verified in a cycle so as to optimize the generated script;

[0020] Among them, the director agent outputs critical comments on the script;

[0021] The screenwriter agent performs a correction task to correct the script according to the critical comments output by the director agent;

[0022] The director agent performs a verification task to verify the revised script and determine whether further adjustments are needed.

[0023] The director agent outputs critical comments on the script, including:

[0024] After the screenwriter agent initially generates the script, the director agent performs the criticism task to conduct a comprehensive review of the script and provide critical opinions on the coherence of the plot and / or the appropriateness of the character positioning and actions.

[0025] Among them, the multiple functional positions also include: actors;

[0026] The director agent outputs critical comments on the script, including:

[0027] Providing the character profile information generated by the director agent and the script generated by the screenwriter agent to the actor agent, so that the actor agent can provide feedback on the script based on its understanding of the character, so that the script is consistent with the character profile;

[0028] The director agent generates critical comments on the script by summarizing the feedback information generated by the actor agents.

[0029] Among them, there are at least two photographer agents;

[0030] In the photography stage, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment are provided to the at least two photographer agents, and the at least two photographer agents independently select corresponding lens types for the lines in the script;

[0031] The at least two photographer agents process the differences in the lens type selection results by performing a debate task;

[0032] The director agent performs a ruling task to summarize the debate results of the at least two photographers and determine the final lens type selection result.

[0033] Among them, it also includes:

[0034] A large AI model that generates audio from text is used to generate corresponding voices for the lines in the script, and the duration of shots and actions in the generated video is synchronized with the corresponding voice clips to synthesize the voice and video.

[0035] A method for providing video content, comprising:

[0036] Receive video input from the user to generate demand text;

[0037] Submit the video generation requirement text to a server, the server is used to create a 3D virtual studio environment based on a 3D development engine, the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions and lens types in the scene. After receiving the video generation requirement text, generate a script through the mutual collaboration between multiple intelligent agents based on an AI big model, the AI ​​big model is an AI big model of text generation text class, the multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos, the script includes a role profile in the script, a line text, and character positions, actions and lens type annotation text information corresponding to the lines; structure the script, and simulate the video shooting process in the 3D virtual studio environment according to the structured script to generate a target video;

[0038] Receive the video generation results from the server and provide them to the user.

[0039] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0040] An electronic device, comprising:

[0041] one or more processors; and

[0042] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described above.

[0043] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions implement the steps of any of the aforementioned methods when executed by a processor in an electronic device.

[0044] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0045] Through the embodiment of the present application, a video generation service can be provided, through which a 3D virtual studio environment can be created based on a 3D (3 Dimensional, three-dimensional) development engine, which may include at least one virtual space scene, at least one virtual character image, and a variety of positions, actions and lens types in the scene. After receiving the video generation requirement text input by the user, a script can be generated by mutual collaboration between multiple intelligent agents based on an AI (Artificial Intelligence, artificial intelligence) large model, wherein the specific AI large model is an AI large model of text generation text class, and the multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos, and the script includes a brief introduction to the characters in the script, lines and text information of the lines, and the text information of the lines annotation includes the character positions, actions and lens types corresponding to the lines. Afterwards, the script can be structured, and the video shooting process can be simulated in the 3D virtual studio environment according to the structured script to generate the target video. In this way, we do not directly use the "vincent video" type of large model for video generation, but use the "vincent text" type of AI large model to complete the script line generation, as well as the annotation of the character position, action, lens type and other information about the specific lines. Finally, the video is generated in the 3D development engine according to the annotation information in the structured script. This makes the video quality depend on the performance of the 3D development engine, so it is easier to ensure the quality of the video. In addition, the video generation process will not be limited by the length of time, and does not involve the segmented generation and splicing of multiple videos. Therefore, the consistency of the characters in the video is guaranteed.

[0046] In the specific process of script generation, collaborative strategies between multiple intelligent agents are also provided, which can include criticism-correction-verification and debate-judgment strategies to achieve script optimization.

[0047] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0050] Figure 2is a flow chart of the server method provided in an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of a 3D virtual studio environment provided in an embodiment of the present application;

[0052] Figure 4 This is a schematic diagram of script generation provided by an embodiment of the present application;

[0053] Figure 5 is a flowchart of a client method provided in an embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0056] In order to facilitate the understanding of the solution provided by the embodiment of the present application, the concept of the agent is briefly introduced below. Agent is a concept that appeared a long time ago, generally referring to an agent that can perceive the environment and take actions to achieve specific goals. In other words, any existence that can interact with the real world and have an impact can be called an agent. Some robots that can plan their own driving paths can be called an agent, and some software that can automatically operate the mouse on a web page to complete shopping operations on behalf of humans can also be called an agent, and so on. However, before the emergence of AI big models, the intelligence of agents was usually achieved through reinforcement learning of knowledge in a specific field. This method makes the agent only play a role in a specific field and cannot achieve cross-domain. With the emergence of AI big models, the excellent emergence ability it has demonstrated has led to the birth of "language agents". The significant breakthrough of this kind of agent based on AI big models is that they can use language as the core medium for thinking and communication. This feature is close to the unique ability range of humans, enabling language agents to achieve cross-domain capabilities.

[0057] In the context of this language agent, virtual production has a new way of implementation. However, virtual production is a very complex task after all. If a single agent is used to implement it, it may still only be able to generate short videos, or it may be difficult to achieve good performance in terms of video quality, character consistency, etc.

[0058] Based on the above situation, in the embodiment of the present application, a virtual production framework based on multi-agent cooperation is constructed. In the specific implementation, a 3D virtual studio environment can be created using a 3D development engine (for example, some 3D game engines, etc.), for example, multiple spatial scenes can be constructed, and multiple positions (where the characters can appear), multiple action types, multiple lens types, and so on can be defined. After the user inputs the specific video generation requirements (which can be expressed through text), multiple agents based on the AI ​​big model can play the functional positions such as director, screenwriter, actor, photographer, etc., and the script can be generated through the mutual cooperation between multiple agents based on the AI ​​big model. The specific generated script can include the required role profile, and specific lines can also be generated, and the corresponding role position, action type, lens type, etc. can be determined for each line. Then this information can be structured and configured in the 3D virtual studio environment, and the video shooting process can be simulated through the functions provided in the 3D development engine, and the target video can be generated. Among them, in a preferred manner, the script generation process can also be divided into multiple continuous and interdependent stages (including planning, script writing, photography, etc.) by simulating the efficient human on-set workflow, and then the script creation process can be completed through the mutual collaboration of multiple intelligent agents at each stage.

[0059] This implementation scheme can be applied in a variety of scenarios, especially for some IP (Intellectual Property) (which can be a general term for all famous cultural and creative works (literature, film, television, animation, games, etc.)) creation, etc. For example, the animation of a certain IP has ended, but players or viewers may still want to continue watching some derivative videos. At this time, a 3D virtual studio can be built, and the key scenes and characters contained in the animation can be arranged in the game engine. Players or viewers can enter demand texts such as "I want to watch a certain plot", and then the specific plot can be performed in the 3D virtual studio through the solution provided in the embodiment of this application, and so on.

[0060] From the perspective of system architecture, see Figure 1, the embodiment of the present application can provide a video generation service, which can provide a 3D virtual studio environment construction service and can also provide multiple intelligent agents. The users of the above-mentioned video generation service can be consumer users, or can also be practitioners in some professional video production industries, etc. In order to facilitate interaction with users, relevant service pages can be provided to users, or mobile terminal Apps (applications), or mini-programs, light applications, etc. can also be provided. In short, a specific page can be used to provide users with an entrance for inputting specific video generation requirements. If a variety of different 3D virtual studio environments are pre-built, relevant lists can be displayed on the page for users to choose. Users can choose the 3D virtual studio environment of interest and enter their own demand text, including what kind of plot they want to see. Correspondingly, after receiving the video generation demand text entered by the user, the video generation service can coordinate the collaboration between various intelligent agents (including directors, screenwriters, actors, photographers, etc.) to complete specific script generation and annotation tasks, and simulate a specific video shooting process in a 3D virtual studio environment, and finally return the generated target video to the user.

[0061] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0062] Embodiment 1

[0063] First, the first embodiment of the present application provides a method for generating a video from the perspective of the server of the above-mentioned video generation service, see Figure 2 , the method may specifically include:

[0064] S201: Creating a 3D virtual studio environment based on a 3D development engine, wherein the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and a variety of positions, actions, and shot types in the scene.

[0065] In an embodiment of the present application, a 3D virtual studio environment can first be built based on a 3D development engine (for example, a game engine such as Unity). Among them, the 3D virtual studio environment can be said to be a collection of one or more virtual space scenes. For example, a 3D virtual studio environment may contain 15 scenes, which cover various life scenes in daily life, such as living rooms, kitchens, offices, roadsides, and so on. Of course, other types of 3D virtual studio environments can also be built, which may include more technological virtual space scenes, and so on.

[0066] In addition to building a 3D virtual studio environment, you can also define a variety of actor positions, action types, shot types, etc. in the environment. Among them, the actor positions can be divided into standing points and sitting positions, etc. For example, in the 3D virtual studio environment of the aforementioned life scene, it can include 32 standing points and 33 sitting positions, etc. Among them, various actor positions can be numbered and described in text form, specifically describing the relative position relationship with an object and / or other positions in the scene, for example, position A: located next to the sofa, between position B and position C, and so on.

[0067] Action types refer to the different actions that actors can perform, which can be defined separately as needed. For example, in the above example, each actor can perform 21 different actions, including basic movement actions such as sitting, walking, etc., and more expressive actions such as jumping, shaking his head in anger, etc.

[0068] Regarding the lens type, that is, the specific lens type that can be used when taking pictures. For example, in the scene of the above example, 9 lens types can be defined, including 3 static lenses shot from different distances (such as close-up, medium shot and panoramic shot) and 6 dynamic lenses that track or surround characters (such as lenses that follow the movement of a certain character, panning lenses, zoom lenses, arc lenses, etc.).

[0069] In addition, at least one virtual character image (which can be a person or an animal, etc.) can be configured in the 3D virtual studio environment. For example, 5 male characters and 5 female characters can be virtualized in advance. When the video is generated later, you can choose from these character images and perform specific actions. The specific character image can be set arbitrarily, or, if it corresponds to a certain animation, the specific character image can also correspond to the character prototype in the animation.

[0070] After defining the above-mentioned multiple character images, positions, action types, shot types, etc., you can configure multiple actor positions, shots, etc. for each specific scene. Figure 3 A vertical view of a living room scene in a 3D virtual studio environment is shown, in which three stations A, B, and C are configured, as well as multiple lens types, etc.

[0071] S202: After receiving the video generation requirement text, a script is generated through the mutual collaboration between multiple intelligent agents based on an artificial intelligence AI big model, wherein the AI ​​big model is an AI big model of text generation text type, and the multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos, and the script includes a brief introduction to the characters in the script, the dialogue text, and the dialogue annotation text information, and the dialogue annotation text information includes the character position, action, and shot type corresponding to the dialogue.

[0072] In addition to creating a 3D virtual studio environment, multiple agents can be created in advance to simulate multiple functional positions involved in the process of artificial video production. For example, at least agents corresponding to the three functional positions of director, screenwriter, and photographer can be included. In the preferred implementation, agents corresponding to actors can also be included, and so on. Among them, different agents can use the same base model, for example, they can all be the same AI large language model, and so on. The so-called large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. That is to say, in the embodiment of the present application, instead of directly using a large model of the "Vincentive Video" class, a large model of the "Vincentive Text (Text Generated Text)" class is used to generate the script, and the actor's position, action, lens type, etc. are marked for each line in the generated script (even in the photography stage, the photographer agent can mark the lens type for the specific lines without actually performing the process of generating the video). Subsequently, this information can be expressed in a structured manner so that it can correspond to the functions in the 3D development engine. In this way, by calling the functions in the specific 3D development engine, the information configured in the specific script can be applied to the specific 3D virtual studio environment and the corresponding target video can be generated.

[0073] Among them, the agents corresponding to different functional positions can be used to realize the functions of the corresponding positions. For example, the director agent is responsible for initiating and supervising the entire video production project. The basic functions of this position include setting role profiles, planning video outlines, and optionally providing feedback on the script, discussing with other staff members, and making final decisions when conflicts arise.

[0074] The screenwriter agent is responsible for working closely under the guidance of the director. Specifically, the screenwriter can not only be responsible for writing dialogues, but also specifying the character positions and action types for each line. In an optimal way, the script can also be continuously updated based on the director's criticism to ensure that it is coherent, attractive, and well-structured.

[0075] The Actor Agent is optional, i.e., it may not exist, but if it does, it may be responsible for fine-tuning the lines based on its character profile, ensuring that the dialogue is consistent with the character, and informing the director of any necessary changes.

[0076] The cameraman agent's job is to select camera settings for each line based on the lens usage guidelines, and preferably compare and discuss these camera settings with other cameraman agents on set to ensure the appropriateness of the lens settings.

[0077] It should be noted that, as mentioned above, since the multiple agents in the embodiments of the present application are used to execute the processing of "vintage text", the agents corresponding to different functional positions can use the same base model, that is, use the same large language model of the Vintage text category, but under different functional requirements, different prompt information can be input to different agents, so that each agent can complete its own different tasks.

[0078] Specifically, based on the generation of multiple intelligent agents corresponding to functional positions, in a preferred implementation, the manual video production process can also be simulated to divide the script generation process into multiple continuous and interdependent stages. For example, in the embodiment of the present application, it can be divided into three stages: planning, script creation, and photography, and specific collaboration strategies are applied in these stages to achieve mutual collaboration between multiple different intelligent agents and jointly complete the creation of the script.

[0079] Among them, in the planning stage, the video generation requirement text input by the user can be provided to the director agent corresponding to the director position, and the director agent plans the characters involved in the script and their introductions, as well as the script outline. Among them, since the video generation requirement text input by the user usually describes the required plot, etc., it can be regarded as a short story idea. Therefore, the director agent can start from this short story idea and generate various character introductions related to the story. Specifically, the character introduction can include the key attributes of the character, such as gender, occupation, personality and other characteristics. Based on these character introductions and multiple predefined scenes in the pre-created 3D virtual studio environment, the director agent can expand the preliminary story idea into a detailed scene outline, specifying the location, plot and characters of each clip, etc.

[0080] In the script creation stage, the planning results of the director agent and the information such as the position and action type in the 3D virtual studio environment (all of the above information can exist in the form of text) can be provided to the screenwriter agent corresponding to the screenwriter position, so that the screenwriter agent can generate the lines in the script and specify the position and action type of the character for the lines in the script. Among them, since the script generated by the screenwriter agent for the first time may have some defects, therefore, in a preferred manner, the embodiment of the present application can also use a collaboration between multiple agents to complete the script creation.

[0081] In order to achieve collaboration between different agents, the embodiment of the present application also provides two collaboration strategies, namely, Critique-Correct-Verify and Debate-Judge. Among them, in the Critique-Correct-Verify strategy, the action agent P generates a response R based on the given context C and instruction I. Then, the criticism agent Q reviews the response R and writes a criticism F, pointing out potential points for improvement. Then, the action agent P integrates these criticisms and corrects the response. Finally, the criticism agent Q evaluates the updated response R to determine whether the criticisms have been adequately addressed or whether further iterations are needed.

[0082] Debate-referee collaboration means that in each iteration, two or more peer agents M and N independently generate their own responses and criticize each other's work. Based on the criticism received, each agent may modify its response or keep the original response. After several rounds of debate, the adjudicating agent J summarizes the discussion and makes the final decision. It should be noted here that in the embodiment of the present application, although different agents may use the same base model, due to the randomness of the AI ​​large model when generating content, that is, even the same model under the same input conditions, the generated content may be different. Therefore, the generation results of two or more peer agents are not exactly the same. By comparing the differences between the two and debating, it is conducive to outputting better results.

[0083] Specifically, during the script creation stage, the above-mentioned criticism-correction-verification strategy can be adopted. Specifically, the screenwriter agent can first write a preliminary script, including character dialogues, positions and action types, etc. Afterwards, the script generated by the screenwriter agent can be criticized, corrected, and verified in a cycle to optimize the generated script. Among them, the director agent can generate critical opinions on the script, and the screenwriter agent can perform a correction task to correct the script according to the critical opinions output by the director agent. Then, the director agent can also perform a verification task to verify the corrected script and determine whether further adjustments are needed.

[0084] There are multiple ways for the director agent to generate critical comments on the script. For example, in one way, after the screenwriter agent initially generates the script, the director agent performs the critical task to conduct a comprehensive review of the script and provide critical comments on the coherence of the plot and / or the appropriateness of the character's position and action. For example, suppose the initially generated script includes a line like "(standing and thinking) Wait, what?", that is, the screenwriter agent marks the emotion as "thinking", but the director agent finds through a comprehensive review that this line should reflect the emotion of confusion or surprise, and then the screenwriter agent makes a correction to the line as "(standing in surprise) Wait, what?".

[0085] Alternatively, in another way, if there is an actor agent, the role profile information generated by the director agent and the script generated by the screenwriter agent can also be provided to the actor agent, so that the actor agent can provide feedback on the script based on its understanding of the role, so that the script is consistent with the role profile. Since there may be multiple roles, there can be multiple actor agents corresponding to each role, and each actor agent can provide feedback on the script from their own perspective. Afterwards, the director agent can generate critical comments on the script by summarizing the feedback information generated by the actor agents.

[0086] Of course, in the specific implementation, the above two methods can also be combined. That is, the director agent and the screenwriter agent can first conduct one or more criticism-correction-verification cycles, and then the actor agent can provide feedback based on their understanding of the role. The director agent screens and summarizes these feedbacks, and then cooperates with the screenwriter agent to optimize the script through the same criticism-correction-verification cycle.

[0087] During the photography phase, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment can be provided to the photographer agent corresponding to the photographer position, so that the photographer agent can mark the corresponding lens type for the lines in the script. In a preferred manner, specific lens usage guide and other knowledge information can also be provided to the photographer agent, so that the photographer agent can complete the task of marking the lens type for the lines under the guidance of such more professional knowledge.

[0088] As mentioned earlier, during the photography stage, the text is not actually converted into a video, but the shot type is labeled for the lines. This annotation information still exists in the form of text. Therefore, for the photography agent, the AI ​​big model it uses can still be the big language model of the literary genre.

[0089] Among them, similar to the situation that the screenwriter agent may have some defects when generating the script for the first time, the photographer agent may also have the situation that the selected lens type is not reasonable when marking the lens type information. Therefore, in a preferred manner, a collaborative strategy between multiple agents can also be used to optimize the lens type selection result. Specifically, in this photography stage, a debate-referee collaborative strategy can be adopted, and the photographer agent can be at least two. The lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment can be provided to the at least two photographer agents, and the at least two photographer agents independently select the corresponding lens type for the lines in the script. Then, the at least two photographer agents can handle the differences in the lens type selection results by performing debate tasks. For example, for a certain line, the lens type selected by the first photographer agent is a medium shot, and the lens type selected by the second photographer agent is a zoom lens. The two can debate and each explain the reasons for selecting the corresponding lens. Finally, the director agent can also perform the adjudication task to summarize the debate results of the at least two photographers and determine the final lens type selection result.

[0090] S203: Structuring the script and simulating a video shooting process in the 3D virtual studio environment according to the structured script to generate a target video.

[0091] After completing the character settings and generating the script, the specific lines, as well as the corresponding character positions, actions, shot types, and other information have been determined. Figure 4 As shown in the figure, it shows the various functional position agents and the corresponding processing results at each stage in a specific example. It can also be seen intuitively from the figure that these lines and annotation information are described by text ( Figure 4The screen corresponding to each line or dialogue is only used to illustrate the corresponding screen after the final video is generated, and the generation of these screens is not included in the process of script generation by the intelligent agent). Therefore, this script text can also be structured so that the positions, actions, shots, etc. in the structured script information can correspond to the functions in the 3D development engine respectively, so that the video shooting process can be simulated in the 3D virtual studio environment by calling the functions in the 3D development engine, and the target video can be generated, wherein the generated target video can be a 2D video. In other words, the video in the embodiment of the present application is not directly generated by the AI ​​big model. The AI ​​big model is only responsible for the generation of script lines and the text annotation of information such as the position, action, and shot type of the role of the specific lines. Finally, the 3D development engine generates the 2D video according to the annotation information in the structured script. The video quality depends on the performance of the 3D development engine. Therefore, it is easier to ensure the quality of the video. In addition, the video generation process will not be limited by the length of time. Therefore, it does not involve the segmented generation and splicing of multiple videos, so that the consistency of the characters in the video is guaranteed.

[0092] It should be noted that in actual applications, since the script involves lines, the voice generation part may also be involved. In specific implementation, in order to create more natural and expressive audio, a large AI model of the "text-generated audio" class can be used to generate the voice of each line. The duration of each shot and action in the video can be synchronized with the corresponding audio clip, that is, the duration of each line in the video can be determined by the length of the corresponding voice clip. Finally, the generated video is synthesized with the audio part to generate the final video content and return it to the user.

[0093] It can be seen that the embodiment of the present application systematically studies automated virtual production from multiple aspects such as basic scene construction, full-process method proposal and evaluation method, providing a solid foundation for the intelligent and efficient transformation of the future film production field. From a technical perspective, the embodiment of the present application shoots videos based on the virtual studio of the game physics engine, which effectively solves the problems of missing plots, low video quality, violation of common sense, and inconsistency of characters in the current Vincent video field. The videos generated under the multi-agent framework surpass the previous single-agent baseline in many aspects such as script fluency, character movements, lines and lens settings, reflecting the effectiveness of the framework. In addition, by manually comparing the scripts and lens settings before and after the cooperation of the agents, it was found that the results after the cooperation were more popular, reflecting the important role of multi-agent cooperation in polishing the script and improving decision-making. In the virtual production scenario, the cooperation of multiple agents based on the Vincent AI large model can surpass the performance of a single model with stronger capabilities, reflecting that the weaker model can reach or even surpass the stronger model by reasonably arranging the agent workflow.

[0094] In summary, through the embodiments of the present application, a video generation service can be provided, through which a 3D virtual studio environment can be created based on a 3D development engine, which may include at least one virtual space scene, at least one virtual character image, and a variety of positions, actions and lens types in the scene. After receiving the video generation requirement text input by the user, a script can be generated through the mutual collaboration between multiple intelligent agents based on an AI (artificial intelligence) big model, wherein the specific AI big model is an AI big model of text generation text class, and the multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos, and the script includes the role profile, lines and line annotation text information in the script, and the line annotation text information includes the role position, action and lens type corresponding to the lines. Afterwards, the script can be structured and the video shooting process can be simulated in the 3D virtual studio environment according to the structured script to generate the target video. In this way, we do not directly use the "vincent video" type of large model for video generation, but use the "vincent text" type of AI large model to complete the script line generation, as well as the annotation of the character position, action, lens type and other information about the specific lines. Finally, the video is generated in the 3D development engine according to the annotation information in the structured script. This makes the video quality depend on the performance of the 3D development engine, so it is easier to ensure the quality of the video. In addition, the video generation process will not be limited by the length of time, and does not involve the segmented generation and splicing of multiple videos. Therefore, the consistency of the characters in the video is guaranteed.

[0095] In the specific process of script generation, collaborative strategies between multiple intelligent agents are also provided, which can include Critique-Correct-Verify and Debate-Judge strategies to achieve script optimization.

[0096] Embodiment 2

[0097] The second embodiment corresponds to the first embodiment and provides a method for providing video content from the perspective of the client. Figure 5 , the method may include:

[0098] S501: receiving a video generation requirement text input by a user;

[0099] S502: Submitting the video generation requirement text to a server, the server is used to create a 3D virtual studio environment based on a 3D development engine, the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions and lens types in the scene. After receiving the video generation requirement text, a script is generated through the mutual collaboration between multiple intelligent agents based on an AI big model, the AI ​​big model is an AI big model of text generation text class, the multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos, the script includes a role profile in the script, a line text, and character positions, actions and lens type annotation text information corresponding to the lines; the script is structured, and a video shooting process is simulated in the 3D virtual studio environment according to the structured script to generate a target video;

[0100] S502: Receive the video generation result from the server and provide it to the user.

[0101] For the parts not described in detail in the second embodiment, please refer to the description of the first embodiment and other parts of this specification, which will not be repeated here.

[0102] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0103] Corresponding to the first embodiment, the embodiment of the present application further provides a device for generating a video, which may include:

[0104] A 3D virtual studio environment creation unit, used to create a 3D virtual studio environment based on a 3D development engine, wherein the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions, and shot types in the scene;

[0105] A script generation unit is used to generate a script through the mutual collaboration of multiple intelligent agents based on an artificial intelligence AI model after receiving a video generation requirement text, wherein the AI ​​model is an AI model of text generation text type, and the multiple intelligent agents are used to simulate multiple functional positions involved in the process of manually producing videos, and the script includes a brief introduction of the characters in the script, a line text, and text information of the line annotations, and the text information of the line annotations includes the character position, action, and shot type corresponding to the line;

[0106] The video generation unit is used to generate a target video by structuring the script and simulating a video shooting process in the 3D virtual studio environment according to the structured script.

[0107] In specific implementation, the script generation unit can be used to:

[0108] By simulating the process of artificial video production, the script generation process is divided into multiple continuous and interdependent stages, and the script is generated through the mutual collaboration between the multiple intelligent agents in each stage.

[0109] The multiple functional positions involved in the artificial video production process simulated by the multiple agents include at least: director, screenwriter and photographer; the multiple continuous and interdependent stages include: planning, script writing, photography;

[0110] At this point, the script generation unit can be specifically used for:

[0111] In the planning stage, the video generation requirement text input by the user is provided to the director agent corresponding to the director position, so that the director agent can plan the roles involved in the script and their introductions, as well as the script outline;

[0112] In the script creation stage, the planning results of the director agent and the information of the positions and action types in the 3D virtual studio environment are provided to the screenwriter agent corresponding to the screenwriter position, so that the screenwriter agent can generate the lines in the script and mark the positions and action types of the characters in the script.

[0113] During the photography stage, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment are provided to the photographer agent corresponding to the photographer position, so that the photographer agent can mark the corresponding lens type for the lines in the script.

[0114] In addition, in order to further improve the quality of the script, the device may also include:

[0115] An optimization processing unit is used to optimize the generated script by criticizing, correcting and verifying the script generated by the screenwriter intelligent agent during the script creation stage;

[0116] Among them, the director agent outputs critical comments on the script;

[0117] The screenwriter agent performs a correction task to correct the script according to the critical comments output by the director agent;

[0118] The director agent performs a verification task to verify the revised script and determine whether further adjustments are needed.

[0119] Specifically, after the screenwriter agent initially generates the script, the director agent may perform the criticism task to conduct a comprehensive review of the script and provide critical comments on the coherence of the plot and / or the appropriateness of the character positioning and actions.

[0120] Alternatively, the multiple functional positions also include: actors; in this case, the role profile information generated by the director intelligent agent and the script generated by the screenwriter intelligent agent can be provided to the actor intelligent agent, so that the actor intelligent agent can provide feedback on the script based on its understanding of the role to make the script consistent with the role profile; then, the director intelligent agent generates critical comments on the script by summarizing the feedback information generated by the actor intelligent agent.

[0121] In addition, there may be at least two photographer agents; in this case, during the photography stage, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment may be provided to the at least two photographer agents, and the at least two photographer agents independently select corresponding lens types for the lines in the script; then, the at least two photographer agents process the differences in the lens type selection results by performing a debate task; the director agent performs a ruling task to summarize the debate results of the at least two photographers and determine the final lens type selection result.

[0122] Furthermore, the device may further include:

[0123] The speech generation unit is used to generate the corresponding speech for the lines in the script by using the AI ​​big model of text-to-audio generation, and synchronize the duration of shots and actions in the generated video with the corresponding speech clips so as to synthesize the speech and video.

[0124] Corresponding to the second embodiment, the embodiment of the present application further provides a device for providing video content, which may include:

[0125] A demand text receiving unit, used for receiving a video input by a user to generate a demand text;

[0126] A submission unit is used to submit the video generation requirement text to a server, and the server is used to create a 3D virtual studio environment based on a 3D development engine, and the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions and lens types in the scene. After receiving the video generation requirement text, a script is generated through the mutual cooperation between multiple intelligent agents based on an AI big model, and the AI ​​big model is an AI big model of text generation text class. The multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos. The script includes a role profile in the script, a line text, and text information annotated with the role position, action and lens type corresponding to the line; the script is structured, and a video shooting process is simulated in the 3D virtual studio environment according to the structured script to generate a target video;

[0127] The video return unit is used to receive the video generation result from the server and provide it to the user.

[0128] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0129] And an electronic device, comprising:

[0130] one or more processors; and

[0131] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0132] A computer program product includes a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the steps of the method described in the aforementioned method embodiment.

[0133] in, Figure 6 The architecture of the electronic device is shown as an example, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620 may be communicatively connected via a communication bus 630.

[0134] Among them, the processor 610 can be implemented by a general-purpose CPU (Central Processing Unit, processor), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0135] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 600. In addition, a web browser 623, a data storage management system 624, and a video generation processing system 625, etc. can also be stored. The above-mentioned video generation processing system 625 can be an application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0136] The input / output interface 613 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0137] The network interface 614 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0138] The bus 630 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620).

[0139] It should be noted that, although the above device only shows a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, a bus 630, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0140] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0142] The above is a detailed introduction to the method and electronic device for generating videos provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for generating a video, characterized in that: include: Creating a 3D virtual studio environment based on a 3D development engine, wherein the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions, and shot types in the scene; After receiving the video generation requirement text, a script is generated through the mutual collaboration between multiple intelligent agents based on an artificial intelligence AI big model, wherein the AI ​​big model is an AI big model of text generation text type, and the multiple intelligent agents are used to simulate multiple functional positions involved in the process of manually producing videos, and the script includes a brief introduction of the characters in the script, the lines text, and the lines annotation text information, and the lines annotation text information includes the character position, action, and shot type corresponding to the lines; The script is structured and a video shooting process is simulated in the 3D virtual studio environment according to the structured script to generate a target video.

2. The method according to claim 1, characterized in that The script is generated through the mutual collaboration between multiple intelligent agents based on the artificial intelligence AI big model, including: By simulating the process of artificial video production, the script generation process is divided into multiple continuous and interdependent stages, and the script is generated through the mutual collaboration between the multiple intelligent agents in each stage.

3. The method according to claim 2, characterized in that The multiple functional positions involved in the artificial video production process simulated by the multiple agents include at least: director, screenwriter and photographer; the multiple continuous and interdependent stages include: planning, script writing, and photography; The generating of the script through the mutual collaboration among the multiple intelligent agents at each stage includes: In the planning stage, the video generation requirement text input by the user is provided to the director agent corresponding to the director position, so that the director agent can plan the roles involved in the script and their introductions, as well as the script outline; In the script creation stage, the planning results of the director agent and the information of the positions and action types in the 3D virtual studio environment are provided to the screenwriter agent corresponding to the screenwriter position, so that the screenwriter agent can generate the lines in the script and mark the positions and action types of the characters in the script. During the photography stage, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment are provided to the photographer agent corresponding to the photographer position, so that the photographer agent can mark the corresponding lens type for the lines in the script.

4. The method according to claim 3, characterized in that Also includes: In the script creation stage, the script generated by the screenwriter agent is criticized, corrected, and verified in a cycle so as to optimize the generated script; Among them, the director agent outputs critical comments on the script; The screenwriter agent performs a correction task to correct the script according to the critical comments output by the director agent; The director agent performs a verification task to verify the revised script and determine whether further adjustments are needed.

5. The method according to claim 4, characterized in that The director agent outputs critical comments on the script, including: After the screenwriter agent initially generates the script, the director agent performs the criticism task to conduct a comprehensive review of the script and provide critical opinions on the coherence of the plot and / or the appropriateness of the character positioning and actions.

6. The method according to claim 4, characterized in that The multiple functional positions also include: actors; The director agent outputs critical comments on the script, including: Providing the character profile information generated by the director agent and the script generated by the screenwriter agent to the actor agent, so that the actor agent can provide feedback on the script based on its understanding of the character, so that the script is consistent with the character profile; The director agent generates critical comments on the script by summarizing the feedback information generated by the actor agents.

7. The method according to claim 3, characterized in that There are at least two photographer agents; In the photography stage, the lines in the script and the corresponding character positions, action type information, and lens type information in the 3D virtual studio environment are provided to the at least two photographer agents, and the at least two photographer agents independently select corresponding lens types for the lines in the script; The at least two photographer agents process the differences in the lens type selection results by performing a debate task; The director agent performs a ruling task to summarize the debate results of the at least two photographers and determine the final lens type selection result.

8. The method according to any one of claims 1 to 7, characterized in that: Also includes: A large AI model that generates audio from text is used to generate corresponding voices for the lines in the script, and the duration of shots and actions in the generated video is synchronized with the corresponding voice clips to synthesize the voice and video.

9. A method for providing video content, characterized in that: include: Receive video input from the user to generate demand text; Submit the video generation requirement text to a server, the server is used to create a 3D virtual studio environment based on a 3D development engine, the 3D virtual studio environment includes at least one virtual space scene, at least one virtual character image, and multiple positions, actions and lens types in the scene. After receiving the video generation requirement text, generate a script through the mutual collaboration between multiple intelligent agents based on an AI big model, the AI ​​big model is an AI big model of text generation text class, the multiple intelligent agents are used to simulate multiple functional positions involved in the process of artificially producing videos, the script includes a role profile in the script, a line text, and character positions, actions and lens type annotation text information corresponding to the lines; structure the script, and simulate the video shooting process in the 3D virtual studio environment according to the structured script to generate a target video; Receive the video generation results from the server and provide them to the user.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

11. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 9.

12. A computer program product comprising a computer program / computer executable instructions, characterized in that: When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 9 are implemented.

Images