Intelligent picture or video generation method and system for large model interaction

Abstract

Provided in the present invention are an intelligent picture or video generation method and system for large model interaction. The method comprises a picture or video generation process: S11, receiving an original prompt request from a user, extracting description information from the original prompt request, and filling same into a storyboard template inside a system; alternatively, directly importing the original prompt request and a storyboard template outside the system, and mapping the storyboard template outside the system into the storyboard template inside the system; S12, serializing and textualizing the content in the storyboard template, so as to obtain at least one prompt request segment, wherein each prompt request segment corresponds to one picture or one storyboard video; and S13, submitting to a large model the prompt request segments one by one or in batches, and the large model generating picture or video content and returning same. In the present invention, a systematic storyboard template is constructed and provided for a user, thereby realizing systematic and structured editing and management; moreover, batch generation can be supported, and interaction modes are rich and diverse, thereby improving the interaction experience of the user.

Description

A method and system for generating intelligent pictures or videos for large-scale model interaction Technical Field

[0001] The present invention relates to the field of large model interaction, and in particular to a method and system for generating intelligent pictures or videos of large model interaction. Background Art

[0002] Currently, large multimodal AI models are developing rapidly. For example, models such as stableVideo and openAI sora have realized the function of generating text into images or videos.

[0003] The existing AI multimodal large model's text-to-image or video generation process is as follows: the user makes a prompt request, provides a verbal description of the image or screen to the large model, and the large model returns the generated image or video.

[0004] Since the prompt request function is mainly based on text / string templates, it contains the picture theme, main body description, picture background and environment description, style definition and model parameters of the picture or video.

[0005] As shown in FIG1 , when a large model implements the function of generating text into an image based on a prompt request, the user provides the prompt request as shown on the left side of FIG1 , and the large model generates the image as shown on the right side of FIG1 .

[0006] As shown in Figure 2, a large model implements the function of generating videos from text based on prompt requests. When the user provides a prompt request, the large model generates a video as shown in Figure 3.

[0007] However, current AI multimodal large models have the following shortcomings when implementing the generation function of text to images or videos:

[0008] 1. The interaction method for the generated page is simple and only suitable for single, sequential input of text, images, and videos. The various dimensions of the image are contained in the prompt. The newly released stableVideo places the camera status configuration on the secondary draft generation page for selection, resulting in a lack of systematic management of prompt requests for each image.

[0009] 2. When the prompt request involves a long text description, the user may lack description dimensions, affecting the quality of the large model's generation of images or videos;

[0010] 3. When a large model returns generated content, the playback interaction mode is limited. Usually, the browser only provides pictures and videos, and only supports playback interaction modes such as play, zoom, double-speed playback, and jump playback.

[0011] 4. It does not support batch generation and is inefficient. Technical issues

[0012] The technical problem to be solved by the present invention is to provide a method and system for generating intelligent pictures or videos of large-scale model interactions, providing users with systematic storyboard templates, thereby realizing systematic and structured editing and management, and supporting batch generation, with rich and diverse interaction methods, thereby enhancing the user interaction experience. Technical Solutions

[0013] In a first aspect, the present invention provides a method for generating intelligent images or videos for large-scale model interaction, including a process for generating images or videos, wherein the process for generating images or videos includes the following steps:

[0014] S11, receiving the user's original prompt request, extracting the description information in the original prompt request and filling it into the storyboard template inside the system; or

[0015] Directly import the original prompt request and the storyboard template outside the system, and map the storyboard template outside the system to the storyboard template inside the system;

[0016] S12. Serialize and textualize the content in the storyboard template to obtain at least one prompt request segment, where each prompt request segment corresponds to an image or storyboard video;

[0017] S13. Submit the prompt request in segments, one by one or in batches, to the big model, which generates the image or video content and returns it.

[0018] In a second aspect, the present invention provides a system for generating intelligent images or videos of large-scale model interactions, including a template management module, wherein the template management module is configured to perform the following steps:

[0019] S11, receiving the user's original prompt request, extracting the description information in the original prompt request and filling it into the storyboard template inside the system; or

[0020] Directly import the original prompt request and the storyboard template outside the system, and map the storyboard template outside the system to the storyboard template inside the system;

[0021] S12. Serialize and textualize the content in the storyboard template to obtain at least one prompt request segment, where each prompt request segment corresponds to an image or storyboard video;

[0022] S13. Submit the prompt request in segments, one by one or in batches, to the big model, which generates the image or video content. Beneficial effects

[0023] The one or more technical solutions provided by the present invention have at least the following technical effects or advantages: a systematic storyboard template is constructed for the user, which can automatically extract the entity configuration of the prompt request, import and edit the configuration to generate the storyboard template, or directly import the storyboard template outside the system, thereby realizing the systematic and structured editing and management of the prompt request, and improving the interactive experience of the generated page; at the same time, it can support the single generation or batch generation of prompt request segments, greatly improving the generation efficiency of pictures and videos; based on the dynamic information editing function of the structured storyboard template, the user can supplement the description dimension at any time to ensure the generation quality of the picture or video by the large model; and during the playback of the generated result, the corresponding parameters on the storyboard template can be modified or added in real time according to the user's playback interaction input, and submitted to the large model, realizing dynamic playback interaction, and greatly improving the playback interaction experience.

[0024] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] FIG1 is a schematic diagram of an interactive page for realizing the function of generating text into images according to a prompt request in an existing large model.

[0027] FIG2 is a schematic diagram of an interactive page for realizing the function of generating text to video according to a prompt request in an existing large model.

[0028] FIG3 is a diagram illustrating one frame of a video generated by an existing large model based on a prompt request.

[0029] FIG4 is a flow chart of the method in the first embodiment of the present invention.

[0030] FIG5 is a schematic diagram of the structure of a storyboard template within the system according to an embodiment of the present invention.

[0031] FIG6 is a schematic structural diagram of a storyboard template outside the system according to an embodiment of the present invention.

[0032] FIG7 is a flow chart of the method in the second embodiment of the present invention.

[0033] FIG8 is a system block diagram of the third embodiment of the present invention.

[0034] FIG9 is a flowchart of a dynamic interaction playback process according to an embodiment of the present invention. Modes for Carrying Out the Invention

[0035] The embodiments of the present application provide a method and system for generating intelligent pictures or videos of large-scale model interactions, thereby providing users with systematic storyboard templates, thereby realizing systematic and structured editing and management. At the same time, it can support batch generation, and has rich and diverse interaction methods, thereby enhancing the user interaction experience.

[0036] The technical solution in the embodiments of the present application has the following overall idea: a systematic storyboard template is constructed for the user, the entity configuration of the prompt request is automatically extracted, the configuration is imported and edited to generate the storyboard template, or a storyboard template outside the system is directly imported, thereby realizing the systematic and structured editing and management of the prompt request and improving the interactive experience of the generated page; at the same time, it can support the single generation or batch generation of prompt request segments, greatly improving the generation efficiency of pictures and videos; based on the dynamic information editing function of the structured storyboard template, the user can supplement the description dimension at any time to ensure the generation quality of the picture or video by the large model; and during the playback of the generated results, the corresponding parameters on the storyboard template can be modified or added in real time according to the user's playback interaction input, and submitted to the large model to realize dynamic playback interaction, which greatly improves the playback interaction experience. Example 1

[0037] As shown in FIG4 , this embodiment provides a method for generating a large-scale model interactive intelligent picture or video, including a picture or video generation process, wherein the picture or video generation process includes the following steps:

[0038] S11, receiving the user's original prompt request, extracting the description information in the original prompt request and filling it into the storyboard template inside the system; or

[0039] Directly import the original prompt request and the storyboard template outside the system, and map the storyboard template outside the system to the storyboard template inside the system; the storyboard template outside the system can be a text file, a table file, or a json or yaml markdown configuration file.

[0040] As shown in FIG5 and FIG6, the storyboard template is displayed in the generated interactive page in the form of a table and is in a secondary editable state;

[0041] The content of the storyboard template includes scene number, shot number, picture reference, object, scene size, camera number, shooting angle, camera movement method, equipment, shot length, sound, picture description, editing and transition method, lighting, generation number, and shooting time; thus, it covers the tools for creative planning, picture design, shooting production, and post-editing for understanding and communicating the pictures, and can more accurately convey the needs of visual planning, production or directing, and speed up communication and production efficiency.

[0042] S12. Serialize and textualize the content in the storyboard template to obtain at least one prompt request segment, where each prompt request segment corresponds to an image or storyboard video;

[0043] The prompt request segment may be filled in the storyboard template accordingly.

[0044] S13. Submit the prompt request segments one by one or in batches to the big model, that is, set a trigger button for single execution for each prompt request segment and set a trigger button for batch execution for the entire storyboard template. After the trigger is executed, the big model can generate the picture or video content and return it.

[0045] As shown in FIG4 , conventional video playback includes normal playback, zoom playback, double-speed playback, jump playback, and other conventional playback modes. The generation method of this embodiment may also include a dynamic interactive playback process:

[0046] S21. When generating a video result to play, receiving the user's playback interaction input in real time;

[0047] S22. Based on the preset playback interaction rules or playback interaction use cases, according to the user's playback interaction input and based on the timestamp, searching for the corresponding video reference frame and description text, modifying or adding the corresponding parameters on the storyboard template in real time, and submitting the modified parameters to the master model;

[0048] S23. The large model regenerates the image or video content in real time and then returns.

[0049] As shown in Figure 9, the dynamic interactive playback process allows users to interact with the sequence frames in the video. Through playback interaction input, based on the timestamp, index corresponding video reference frame and description text, the corresponding parameters on the storyboard template are modified or added in real time to achieve screen content interaction, such as video screen shot action simulated based on mouse interaction, or screen special effects corresponding to keyboard shortcuts, etc., thereby expanding the video playback interaction from static sequence frame playback interaction to immersive interaction within the generated content.

[0050] The user's playback interaction input includes but is not limited to keyboard input, mouse input, audio input, motion input, video input, etc.; the playback interaction rules or the playback interaction use cases are provided by the system or defined by the user.

[0051] The playback interaction rules take mouse operation as an example:

[0052] When the video is playing, left-click and drag left / right to change the camera movement parameters to push left / right;

[0053] When the video is playing, right-click and drag left / right to change the camera parameters to pan left / right; right-click and drag forward / backward to pan up / down;

[0054] When the video is playing, rolling the wheel will zoom in; clicking the wheel will move forward / backward.

[0055] The interactive playback may be, for example, rain or snow on the screen. Example 2

[0056] As shown in Figure 7, this second embodiment provides a method for generating intelligent images or videos interactive with a large model. This method, based on the first embodiment, also provides the function of generating suggestion prompts. Specifically, in step S12, after the content of the storyboard template is serialized and textualized to obtain at least one prompt request segment, each prompt request segment can also generate a suggestion prompt based on user triggering, and the suggestion prompt is submitted to the large model in place of the prompt request segment. This method can compensate for the shortcomings of the user's prompt request segment and further improve the quality of the large model's image or video generation.

[0057] The user's trigger may be set at the position of each prompt request segment corresponding to the storyboard template in the generated interactive page.

[0058] This second embodiment also provides the option of submitting prompt request segments to the big model one by one or in batches, that is, setting a trigger button for single execution for each prompt request segment and a trigger button for batch execution for the entire storyboard template. After the user triggers the execution, the big model can generate the image or video content and then return it. Example 3

[0059] As shown in FIG8 , in this embodiment, a system for generating intelligent pictures or videos of large-scale model interactions is provided, which is a device corresponding to the method in the first embodiment, including a template management module and a dynamic interaction playback module.

[0060] As shown in FIG4 , the template management module is used to perform the following steps:

[0061] S11, receiving the user's original prompt request, extracting the description information in the original prompt request and filling it into the storyboard template inside the system; or

[0062] Directly import the original prompt request and the storyboard template outside the system, and map the storyboard template outside the system to the storyboard template inside the system; the storyboard template outside the system can be a text file, a table file, or a json or yaml markdown configuration file.

[0063] S12. Serialize and textualize the content in the storyboard template to obtain at least one prompt request segment, where each prompt request segment corresponds to an image or storyboard video;

[0064] S13. Submit the prompt request in segments, one by one or in batches, to the big model, which generates the image or video content.

[0065] This second embodiment also provides the option of submitting prompt request segments to the big model one by one or in batches, that is, setting a trigger button for single execution for each prompt request segment and a trigger button for batch execution for the entire storyboard template. After the user triggers the execution, the big model can generate the image or video content and then return it.

[0066] As shown in FIG5 and FIG6, the storyboard template is displayed in the generated interactive page in the form of a table and is in a secondary editable state;

[0067] The content of the storyboard template includes scene number, shot number, picture reference, object, scene type, camera number, shooting angle, camera movement method, equipment, shot length, sound, picture description, editing and transition method, lighting, generation number, and shooting time;

[0068] The template management module is also used for:

[0069] Fill the prompt request segments into the storyboard template accordingly;

[0070] According to the user's trigger, the corresponding prompt request segment is used to generate a suggestion prompt to replace the prompt request segment and submit it to the big model.

[0071] As shown in FIG4 , the dynamic interaction module is used to perform the following process:

[0072] S21. When generating a video result to play, receiving the user's playback interaction input in real time;

[0073] S22. Based on the preset playback interaction rules or playback interaction use cases, according to the user's playback interaction input and based on the timestamp, searching for the corresponding video reference frame and description text, modifying or adding the corresponding parameters on the storyboard template in real time, and submitting the modified parameters to the master model;

[0074] S23. The large model regenerates the image or video content in real time and then returns.

[0075] The playback interaction rules or the playback interaction use cases are provided by the system or defined by the user.

[0076] As shown in Figure 9, the dynamic interactive playback process allows users to interact with the sequence frames in the video. Through playback interaction input, based on the timestamp, index corresponding video reference frame and description text, the corresponding parameters on the storyboard template are modified or added in real time to achieve screen content interaction, such as video screen shot action simulated based on mouse interaction, or screen special effects corresponding to keyboard shortcuts, etc., thereby expanding the video playback interaction from static sequence frame playback interaction to immersive interaction within the generated content.

[0077] The user's playback interaction input includes but is not limited to keyboard input, mouse input, audio input, motion input, video input, etc.; the playback interaction rules or the playback interaction use cases are provided by the system or defined by the user.

[0078] The playback interaction rules take mouse operation as an example:

[0079] When the video is playing, left-click and drag left / right to change the camera movement parameters to push left / right;

[0080] When the video is playing, right-click and drag left / right to change the camera parameters to pan left / right; right-click and drag forward / backward to pan up / down;

[0081] When the video is playing, rolling the wheel will zoom in; clicking the wheel will move forward / backward.

[0082] The interactive playback may be, for example, rain or snow on the screen.

[0083] In summary, the technical solution provided in the embodiments of the present application has at least the following technical effects or advantages: building a systematic storyboard template for the user, automatically extracting the entity configuration of the prompt request, importing and editing the configuration to generate the storyboard template, or directly importing the storyboard template outside the system, thereby realizing the systematic and structured editing and management of the prompt request, and improving the interactive experience of the generated page; at the same time, it can support the single generation or batch generation of prompt request segments, greatly improving the generation efficiency of pictures and videos; based on the dynamic information editing function of the structured storyboard template, the user can supplement the description dimension at any time to ensure the generation quality of the picture or video by the large model; and during the playback process of the generated result, the corresponding parameters on the storyboard template can be modified or added in real time according to the user's playback interaction input, and submitted to the large model to realize dynamic playback interaction, which greatly improves the playback interaction experience.

[0084] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for generating intelligent pictures or videos of large-scale model interactions, characterized by: The process of generating a picture or video includes the following steps: S11, receiving the user's original prompt request, extracting the description information in the original prompt request and filling it into the storyboard template within the system; or Directly import the original prompt request and the storyboard template outside the system, and map the storyboard template outside the system to the storyboard template inside the system; S12. Serialize and textualize the content in the storyboard template to obtain at least one prompt request segment, where each prompt request segment corresponds to an image or storyboard video; S13. Submit the prompt request in segments, one by one or in batches, to the big model, which generates the image or video content and returns it.

2. The method for generating a large-scale interactive intelligent picture or video according to claim 1, characterized in that: The storyboard template is displayed in the generated interactive page in the form of a table and is in a secondary editable state; The content of the storyboard template includes scene number, shot number, picture reference, object, scene type, camera number, shooting angle, camera movement method, equipment, shot length, sound, picture description, editing and transition method, lighting, generation number, and shooting time; The storyboard template outside the system is a text file, a table file, or a markdown configuration file in json or yaml.

3. The method for generating intelligent pictures or videos of large-scale model interaction according to claim 1, characterized in that: The prompt request segment is filled in the storyboard template accordingly. Each prompt request segment can also generate a suggestion prompt based on the user's trigger. The suggestion prompt replaces the prompt request segment and is submitted to the big model.

4. The method for generating a large-scale interactive intelligent picture or video according to claim 1, characterized in that: It also includes the playback of dynamic interactive processes: S21. When generating a video result to play, receiving the user's playback interaction input in real time; S22. Based on the preset playback interaction rules or playback interaction use cases, according to the user's playback interaction input and based on the timestamp, searching for the corresponding video reference frame and description text, modifying or adding the corresponding parameters on the storyboard template in real time, and submitting the modified parameters to the master model; S23. The large model regenerates the image or video content in real time and then returns.

5. The method for generating intelligent pictures or videos of large-scale model interaction according to claim 4, characterized in that: The playback interaction rules or the playback interaction use cases are provided by the system or defined by the user.

6. A system for generating intelligent pictures or videos of large-scale model interactions, characterized by: The template management module is included, and the template management module is used to perform the following steps: S11, receiving the user's original prompt request, extracting the description information in the original prompt request and filling it into the storyboard template within the system; or Directly import the original prompt request and the storyboard template outside the system, and map the storyboard template outside the system to the storyboard template inside the system; S12. Serialize and textualize the content in the storyboard template to obtain at least one prompt request segment, where each prompt request segment corresponds to an image or storyboard video; S13. Submit the prompt request in segments, one by one or in batches, to the big model, which generates the image or video content.

7. The system for generating intelligent pictures or videos of large-scale model interactions according to claim 6, characterized in that: The storyboard template is displayed in the generated interactive page in the form of a table and is in a secondary editable state; The content of the storyboard template includes scene number, shot number, picture reference, object, scene type, camera number, shooting angle, camera movement method, equipment, shot length, sound, picture description, editing and transition method, lighting, generation number, and shooting time; The storyboard template outside the system is a text file, a table file, or a markdown configuration file in json or yaml.

8. The system for generating intelligent pictures or videos of large-scale model interactions according to claim 6, characterized in that: The template management module is also used for: Fill the prompt request segments into the storyboard template accordingly; According to the user's trigger, the corresponding prompt request segment is used to generate a suggestion prompt to replace the prompt request segment and submit it to the big model.

9. The system for generating intelligent pictures or videos of large-scale model interactions according to claim 6, characterized in that: The module also includes a dynamic interaction module for playing the video. The dynamic interaction module is used to perform the following process: S21. When generating a video result to play, receiving the user's playback interaction input in real time; S22. Based on the preset playback interaction rules or playback interaction use cases, according to the user's playback interaction input and based on the timestamp, searching for the corresponding video reference frame and description text, modifying or adding the corresponding parameters on the storyboard template in real time, and submitting the modified parameters to the master model; S23. The large model regenerates the image or video content in real time and then returns.

10. The system for generating intelligent pictures or videos of large-scale model interactions according to claim 9, characterized in that: The playback interaction rules or the playback interaction use cases are provided by the system or defined by the user.

Images