Intelligent split-screen creation method, electronic device, storage medium and program product

Abstract

The present disclosure relates to an intelligent split-screen creation method, an electronic device, a storage medium and a program product. The method comprises: in response to a user importing at least one split-screen sketch in a split-screen creation interface, displaying the imported at least one split-screen sketch in a split-screen canvas of the split-screen creation interface; in response to the user selecting one or a plurality of split-screen sketches from the displayed split-screen sketches, displaying a split-screen generation component corresponding to the split-screen sketch; in response to the user completing a corresponding setting operation through the split-screen generation component, generating a split-screen script corresponding to each selected split-screen sketch according to a user-set generation mode and corresponding generation parameters by using artificial intelligence technology, and displaying the split-screen script in the split-screen canvas, wherein the split-screen script is a split-screen image or a split-screen video. Thus, the split-screen creation efficiency can be effectively improved, the personalized creation needs of the user can be met, the split-screen production threshold and cost can be greatly reduced, and the convenience and use experience are better.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to an intelligent storyboard creation method, electronic device, storage medium, and program product. Background Technology

[0002] As the film, advertising, and animation industries continue to demand higher quality and efficiency in video production, the technical requirements for storyboard adjustments and video creation are gradually increasing. Storyboards are a fundamental element in film or animation works, determining the narrative style, visual expressiveness, and the audience's visual experience.

[0003] Traditional storyboard production largely relies on manual operation. Production staff manually draw compositions, characters, and scenes based on the script, director's intentions, and visual requirements. Some storyboards even require multiple drawings to express the story. This often requires a long time and high labor costs, resulting in a huge workload and low efficiency in storyboard creation. It is also difficult to meet rapidly changing creative needs. Furthermore, existing storyboard production uses professional and complex editing or animation software, which requires a certain learning curve and is costly for ordinary users to learn and use. Summary of the Invention

[0004] In view of this, this disclosure proposes an intelligent storyboard creation method, electronic device, storage medium and program product, which can effectively improve the efficiency of storyboard creation, meet the personalized creation needs of users, and greatly reduce the threshold and cost of storyboard production, with better convenience and user experience.

[0005] According to one aspect of this disclosure, an intelligent storyboard creation method is provided, comprising: responding to a user importing at least one storyboard sketch into a storyboard creation interface, displaying the imported at least one storyboard sketch in a storyboard canvas of the storyboard creation interface; responding to a user selecting one or multiple storyboard sketches from the displayed storyboard sketches, displaying a storyboard generation component corresponding to the storyboard sketch, the storyboard generation component being used to set a generation mode and corresponding generation parameters for the storyboard sketch, the generation mode being used to determine the type of storyboard draft generated based on the storyboard sketch, and the generation parameters being used to determine the effect of the storyboard draft generated based on the storyboard sketch; responding to a user completing the corresponding setting operation through the storyboard generation component, using artificial intelligence technology according to the generation mode and corresponding generation parameters set by the user, generating a storyboard draft corresponding to each selected storyboard sketch and displaying it in the storyboard canvas, wherein the storyboard draft corresponding to any selected storyboard sketch is displayed on one side of the selected storyboard sketch, and the storyboard draft is a storyboard image or a storyboard video.

[0006] According to another aspect of this disclosure, an intelligent storyboard creation device is provided, comprising: an import module, configured to respond to a user importing at least one storyboard sketch into a storyboard creation interface, and displaying the imported at least one storyboard sketch in a storyboard canvas of the storyboard creation interface; a component module, configured to respond to a user selecting one or multiple storyboard sketches from the displayed storyboard sketches, and displaying a storyboard generation component corresponding to the storyboard sketch, the storyboard generation component being configured to set a generation mode and corresponding generation parameters for the storyboard sketch, the generation mode being configured to determine the type of storyboard draft generated based on the storyboard sketch, and the generation parameters being configured to determine the effect of the storyboard draft generated based on the storyboard sketch; and a generation module, configured to respond to a user completing a corresponding setting operation through the storyboard generation component, and using artificial intelligence technology to generate a storyboard draft corresponding to each selected storyboard sketch according to the user-set generation mode and corresponding generation parameters, and displaying it in the storyboard canvas, wherein the storyboard draft corresponding to any selected storyboard sketch is displayed on one side of the selected storyboard sketch, and the storyboard draft is a storyboard image or a storyboard video.

[0007] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0008] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0010] According to various aspects of this disclosure, by displaying user-imported storyboard sketches in the storyboard canvas and in the storyboard sketch... Figure 1 The generated storyboard is displayed sideways, forming an AI-generated creative canvas based on a storyboard table. This aligns better with creators' usage habits and improves the user experience. By displaying the storyboard generation component, users can set the generation mode and parameters according to their actual needs. Artificial intelligence technology can then automatically generate storyboards that meet the creator's requirements based on the user-defined generation mode and parameters. Storyboards can be static storyboard images or dynamic storyboard videos, effectively improving storyboard creation efficiency while meeting users' personalized creative needs. Furthermore, storyboard images or videos can be created within a single interface without switching between different functional software, enhancing user experience, convenience, and creative efficiency.

[0011] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0013] Figure 1 A flowchart illustrating an intelligent storyboard creation method according to an embodiment of the present disclosure is shown.

[0014] Figure 2 and Figure 3 Schematic diagrams of a storyboard creation interface according to an embodiment of the present disclosure are shown respectively.

[0015] Figure 4 , Figure 5 and Figure 6 The diagrams show the layout of a storyboard creation interface according to an embodiment of the present disclosure.

[0016] Figure 7 and Figure 8 The diagrams show schematic representations of a storyboard creation interface for batch selection of multiple storyboard sketches or multiple storyboard images according to an embodiment of the present disclosure.

[0017] Figure 9 This diagram illustrates a storyboard creation interface for batch selecting multiple storyboard videos according to an embodiment of the present disclosure.

[0018] Figure 10 This diagram illustrates a storyboard generation component displayed when a storyboard sketch or a storyboard is selected, according to an embodiment of the present disclosure.

[0019] Figure 11 A schematic diagram of the relevant components of a partial redraw function according to an embodiment of the present disclosure is shown.

[0020] Figure 12 and Figure 13 A schematic diagram of the components related to an image expansion function according to an embodiment of the present disclosure is shown.

[0021] Figure 14 This diagram illustrates a storyboard generation component for selecting a storyboard video according to an embodiment of the present disclosure.

[0022] Figure 15 , Figure 16 , Figure 17 , Figure 18 and Figure 19 A schematic diagram of related components for a lip-syncing function according to an embodiment of the present disclosure is shown.

[0023] Figure 20 and Figure 21 A schematic diagram of a storyboard creation interface according to an embodiment of the present disclosure is shown.

[0024] Figure 22 A schematic diagram showing the generation of grid states in some storyboard canvases according to an embodiment of the present disclosure is provided.

[0025] Figure 23 The diagram illustrates some of the storyboards labeled according to an embodiment of the present disclosure.

[0026] Figure 24 A schematic diagram of a storyboard creation interface in a static storyboard view according to an embodiment of the present disclosure is shown.

[0027] Figure 25 A schematic diagram illustrating related operations under a static storyboard view according to an embodiment of the present disclosure is shown.

[0028] Figure 26 This diagram illustrates a switching process between a static storyboard view and a creation view according to an embodiment of the present disclosure.

[0029] Figure 27 The diagram illustrates some of the labeling of storyboard videos according to an embodiment of the present disclosure.

[0030] Figure 28 A schematic diagram of a storyboard creation interface under a dynamic storyboard view according to an embodiment of the present disclosure is shown.

[0031] Figure 29 This diagram illustrates a switching process between a dynamic storyboard view and a creation view according to an embodiment of the present disclosure.

[0032] Figure 30 , Figure 31 and Figure 32 Schematic diagrams of a character library management component, a scene library management component, and a style library management component according to an embodiment of the present disclosure are shown respectively.

[0033] Figure 33 A schematic diagram of a clipper interface according to an embodiment of the present disclosure is shown.

[0034] Figure 34 A schematic diagram of the framework structure of an intelligent canvas system according to an embodiment of the present disclosure is shown.

[0035] Figure 35 A schematic diagram of a storyboard generation component in follow mode according to an embodiment of the present disclosure is shown.

[0036] Figure 36A schematic diagram of a storyboard generation component in follow mode according to an embodiment of the present disclosure is shown.

[0037] Figure 37 , Figure 38 and Figure 39 Schematic diagrams of a storyboard generation component in global mode according to an embodiment of the present disclosure are shown respectively.

[0038] Figure 40 and Figure 41 A schematic diagram of setting controls according to an embodiment of the present disclosure is shown.

[0039] Figure 42 A schematic diagram of a weight adjustment control according to an embodiment of the present disclosure is shown.

[0040] Figure 43 and Figure 44 A schematic diagram of a character diagram setting control according to an embodiment of the present disclosure is shown.

[0041] Figure 45 and Figure 46 A schematic diagram is shown regarding an image cropping function according to an embodiment of the present disclosure.

[0042] Figure 47 and Figure 48 A schematic diagram of a storyboard generation component in a graphic video mode according to an embodiment of the present disclosure is shown.

[0043] Figure 49 A schematic diagram of a video magnification function according to an embodiment of the present disclosure is shown.

[0044] Figure 50 A schematic diagram of a storyboard generation component in global mode according to an embodiment of the present disclosure is shown.

[0045] Figure 51 This diagram illustrates a node identifier in a prompt word editing area according to an embodiment of the present disclosure.

[0046] Figure 52 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0047] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0048] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0049] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0050] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0051] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0052] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0053] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0054] As mentioned above, traditional storyboard production involves creators manually drawing storyboards using specialized and complex editing or animation software. This process is labor-intensive, inefficient, and requires a certain learning curve, making it costly for ordinary users to learn and use. Furthermore, when generating dynamic storyboards (such as storyboard videos), creators need to frequently switch between different tools such as editing software, drawing software, and audio / video generation tools, further reducing the efficiency of dynamic storyboard creation.

[0055] Therefore, this disclosure proposes an intelligent storyboard creation method by considering the usage habits of storyboard creators. It deeply integrates traditional storyboards with AI generation capabilities, creating a unified storyboard canvas that is "manageable, generative, and previewable," achieving a closed loop from idea to visual result. Furthermore, this disclosure transforms the concept of a "library" into the core engine of the creative process. Users generate storyboards by calling library assets, ensuring a high degree of consistency in visual style and character design across all storyboards within a project. This is key to addressing the issue of high randomness in AI-generated storyboards, making them unsuitable for long narratives, while also helping storyboard creators accumulate and enhance their creative assets. For the professional market (such as film and animation production), it can serve as a tool to improve storyboard creation efficiency; for the mass market, it can serve as a tool for realizing creative ideas, lowering the barrier to entry for dynamic storyboard creation. This approach liberates the creation of dynamic storyboards from complex professional editing or animation software, allowing both professional producers and ordinary users to easily and quickly generate and adjust dynamic storyboards, thus reducing the production cost and barrier to entry. Furthermore, users no longer need to switch between different software tools; they can quickly generate personalized storyboard drafts by switching between different modes using a unified storyboard generation component. The canvas-style display of storyboard drafts allows for a global preview of all drafts, while also supporting multiple view switching options for easy viewing of the final draft. This improves efficiency for storyboard creators and lowers the creative threshold.

[0056] The intelligent storyboard creation method of this disclosure can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this disclosure can refer to devices with wireless and / or wired connection functions. Wireless connection means that they can connect to other devices via Wi-Fi, Bluetooth, or other wireless methods. The terminal devices involved in this disclosure can also communicate with other devices via wired connection. The terminal devices involved in this disclosure can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers or styluses. Non-touchscreen devices can connect to input devices such as mice, keyboards, or touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For example, the terminal devices in this application can include, but are not limited to, mobile terminals, tablet computers, laptops, and desktop computers.

[0057] Figure 1 A flowchart illustrating an intelligent storyboard creation method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes:

[0058] Step S11: In response to the user importing at least one storyboard sketch into the storyboard creation interface, display the imported at least one storyboard sketch in the storyboard canvas of the storyboard creation interface.

[0059] Step S12: In response to the user selecting one or multiple storyboard sketches from the displayed storyboard sketches, the storyboard generation component corresponding to the storyboard sketch is displayed. The storyboard generation component is used to set the generation mode and corresponding generation parameters for the storyboard sketch. The generation mode is used to determine the type of storyboard generated based on the storyboard sketch, and the generation parameters are used to determine the effect of the storyboard generated based on the storyboard sketch.

[0060] Step S13: In response to the user completing the corresponding setting operation through the storyboard generation component, artificial intelligence technology is used to generate the storyboard draft corresponding to each selected storyboard sketch according to the user-set generation mode and corresponding generation parameters, and display it in the storyboard canvas. The storyboard draft corresponding to any selected storyboard sketch is displayed on one side of the selected storyboard sketch. The storyboard draft is either a storyboard image or a storyboard video.

[0061] According to the method of this disclosure, by displaying a user-imported storyboard sketch in the storyboard canvas and in the storyboard sketch... Figure 1 The generated storyboard is displayed sideways, forming an AI-generated creative canvas based on a storyboard table. This aligns better with creators' usage habits and improves the user experience. By displaying the storyboard generation component, users can set the generation mode and parameters according to their actual needs. Artificial intelligence technology can then automatically generate storyboards that meet the creator's requirements based on the user-defined generation mode and parameters. Storyboards can be static storyboard images or dynamic storyboard videos, effectively improving storyboard creation efficiency while meeting users' personalized creative needs. Furthermore, storyboard images or videos can be created within a single interface without switching between different functional software, enhancing user experience, convenience, and creative efficiency.

[0062] In some embodiments, a storyboard sketch can be a simplified depiction of the storyboard content (such as a simplified character design, a simplified scene, a simplified composition, etc.). The storyboard sketch can be a user-drawn sketch or any reference sketch. This reference sketch may have the same overall composition as the user's desired composition, but may differ from the user's desired character design, style, or scene, etc. This disclosure does not limit the specific content of the storyboard sketch. A storyboard diagram or storyboard video can be understood as an image or video that details the storyboard content described in the storyboard sketch. Compared to a storyboard sketch, a storyboard draft is more detailed and meets the user's needs regarding character design, style, and scene, etc.

[0063] Considering that the storyboard generated based on the storyboard sketch may still not meet the user's needs, or the user wants to first generate a static storyboard that meets the requirements based on the storyboard sketch, and then generate a dynamic storyboard video based on the static storyboard, therefore, in some embodiments, the method may further include:

[0064] Step S14: In response to the user selecting one or more storyboards from the displayed storyboards, a storyboard generation component corresponding to the storyboard is displayed. This component generates a new storyboard based on the selected storyboard. This method is equivalent to further modifying and adjusting the existing storyboards to generate a new one. This allows users to efficiently create storyboards within a unified interface using the storyboard generation component, eliminating the need to switch between different software programs and improving user experience, convenience, and creative efficiency.

[0065] In practical applications, users can first generate storyboard images based on storyboard sketches, which is equivalent to generating higher-quality storyboard images by optimizing the details of the storyboard sketches; they can also generate storyboard videos based on the storyboard images, or users can directly generate storyboard videos based on storyboard sketches, which is equivalent to generating dynamic storyboards based on static storyboards, which can better showcase the storyboard content; and users can also generate new storyboard images based on storyboard images, which is equivalent to further optimizing the storyboard images to generate higher-quality storyboard images, and this disclosure does not limit the scope of the embodiments.

[0066] For example, when entering the storyboard creation interface for the first time and without importing any storyboard sketches, the following can be displayed: Figure 2 A schematic diagram of the storyboard creation interface is shown, such as... Figure 2 As shown, users can import storyboard sketches by clicking the "Upload Image" control, or they can also click... Figure 2 The storyboard sketch is imported using the transfer control indicated by the "+" sign. This embodiment of the disclosure does not limit the import process of the storyboard sketch; existing file import methods can be referenced, and this embodiment of the disclosure does not impose any restrictions on this. After the user completes the import of the storyboard sketch, exemplarily, the following can be displayed: Figure 3 The storyboard creation interface shown is as follows: Figure 3 As shown, the storyboard canvas can include two areas: "Storyboard" and "Generation Area". Multiple storyboard sketches imported by the user can be displayed vertically in the "Storyboard" area on the left, and the subsequently generated storyboards can be displayed horizontally in the "Generation Area" on the right. This display is more in line with the traditional storyboard usage habits of creators and can improve the user experience.

[0067] Each storyboard sketch can correspond to an image row and a video row. It defaults to displaying a single row; when different types of storyboards are generated, another row is displayed. By separating images and videos into different rows within the same storyboard sketch, the types are distinguished, making it easier for users to find what they need. For example, as shown... Figure 4 As shown, when a storyboard is generated based solely on a storyboard sketch, the storyboard can be displayed horizontally in a single row on the corresponding storyboard sketch. Figure 1 Side view (e.g., the image row corresponding to storyboard 2); when the storyboard video is generated solely based on the storyboard sketch, the storyboard video can be displayed horizontally in a single row on the corresponding storyboard sketch. Figure 1 Side (e.g., the video row corresponding to storyboard 3); after generating storyboard video and storyboard diagram based on storyboard sketch, the storyboard video and storyboard diagram can be displayed in two rows respectively (e.g., the image row and video row corresponding to storyboard 1).

[0068] In some embodiments, the number of rows and columns of storyboard sketches and storyboard drafts displayed in the storyboard canvas can be adapted to an initial value based on the screen size. Then, the default number of columns and rows to be displayed is determined based on the initial value. The smaller the screen size, the smaller the initial value. For example, the default total number of columns in the canvas = initial value + 5 columns, and the number of rows = initial value + 5 rows. If, after a user operation (e.g., generating storyboard video or storyboard diagrams), rows or columns need to be added, empty rows and columns can be filled according to this rule. If the number of rows / columns generated after the user operation is less than or equal to the initial value, the default total number of rows / columns displayed on the canvas = initial value + 5 rows / 5 columns. If the number of rows / columns generated after the user operation is greater than the initial value, the default total number of rows / columns displayed on the canvas = generated number + 5 rows / 5 columns. This disclosure does not impose any limitations on this aspect.

[0069] In some embodiments, the order of storyboard drafts in a row corresponding to the same storyboard sketch can be adjusted by dragging, and the storyboard draft corresponding to any storyboard sketch can be copied to the storyboard draft corresponding to other storyboard sketches or copied as a storyboard sketch by dragging; for example, such as Figure 5 As shown, the shooting order of two storyboard sketches SC01-A and SC0A-B corresponding to the same storyboard sketch can be adjusted by dragging the sketches. For example, SC0A-B can be dragged between SC01-A. Different types of storyboard sketches can be set to not be mixed, that is, storyboard sketches and storyboard videos cannot be arranged in the same row. When dragging across storyboard sketches, the storyboard sketch being dragged can be copied and moved to the position indicated by the pointer. For example, storyboard sketch SC01-A can be dragged and moved to the position indicated by the movement pointer after storyboard sketch SC02-A (at this time, the movement pointer is in a column of the generation area, indicating which two storyboard sketches are inserted between). Alternatively, storyboard sketch SC01-A can be dragged and moved after storyboard sketch SC02 as a new storyboard sketch (at this time, the movement pointer is in the storyboard table to indicate which two storyboard sketches are inserted between). Figure 6 As shown, when a user wants to drag and copy the storyboard corresponding to a certain storyboard sketch to the storyboard corresponding to other storyboard sketches, but the other storyboard sketch is a video line (that is, only the storyboard video), an image line can be automatically added. For example, when the user drags the storyboard SC01-A to a position where the movement pointer is after the storyboard video SC02-A, an image line can be automatically added to display the copied storyboard SC01-A as the storyboard for storyboard sketch SC02.

[0070] In some embodiments, to facilitate users' viewing of the content displayed in the storyboard canvas, the storyboard canvas in the storyboard creation interface also supports moving, zooming in, and zooming out operations; the storyboard creation interface also displays the current zoom level of the storyboard image; wherein, the storyboard creation interface may adopt the underlying infinite canvas capability to support the movement and scaling of the storyboard canvas on this underlying layer, but the storyboard grids (i.e., the grids displaying storyboard sketches) and generated grids (i.e., the grids displaying storyboard drafts) displayed in the storyboard canvas cannot be moved to areas other than the storyboard canvas, that is, the storyboard sketches and storyboard drafts cannot be moved to areas other than the storyboard canvas. It should be understood that those skilled in the art can use known methods of movement and zooming operations. For example, the screen can be zoomed by moving two fingers on the touchpad and spreading them apart, or by using the Ctrl / Ctrl key on the keyboard and the mouse wheel, with the mouse as the center. The screen can be moved in any direction by moving two fingers on the touchpad, or by using the Shift key on the keyboard and the mouse wheel, or by using only the mouse wheel, etc. The embodiments disclosed herein do not limit this.

[0071] In practical applications, when a user wants to select a single storyboard sketch or storyboard piece, they can do so by clicking on the sketch or piece; when a user wants to select multiple storyboard sketches or pieces in batches, they can do so by dragging the mouse across a blank area of ​​the canvas, or by using the shift key on the keyboard and the left mouse button to select them consecutively. This disclosure does not limit the scope of the embodiments.

[0072] In some embodiments, the storyboard generation component includes a follow mode and a global mode; the generation modes include an image-to-image mode and an image-to-video mode. The image-to-image mode is used to generate images based on images, and the image-to-video mode is used to generate videos based on images. When multiple storyboard sketches or multiple storyboard images are selected in batches, the storyboard generation component is in global mode. The storyboard generation component in global mode includes settings controls for the generation mode and corresponding generation parameters, and is displayed in a designated location on the storyboard creation interface, for example... Figure 7 Provide a storyboard generation component that operates in global mode when multiple storyboard sketches or multiple storyboard images are selected in batches. Figure 7 Displayed in raw image mode. Figure 8 This shows the style of the storyboard generation component in global mode in image-generated video mode. It should be noted that... Figure 7 and Figure 8 The storyboard generation components shown are just examples. In reality, only one storyboard generation component will be displayed when multiple storyboards or multiple storyboard sketches are selected in batches.

[0073] In global mode, the storyboard generation component is displayed by default at the bottom center of the storyboard generation interface. This component allows users to set the desired generation mode (as shown in the image, "Batch Generation" represents the image-to-image mode, and "Video Generation" represents the image-to-video mode) and corresponding generation parameters (such as the type of image model, the number of images generated, aspect ratio, resolution, and reference images, character images, scene images, style images, etc.). After setting the image mode and corresponding generation parameters, users can click the "Generate Now" button to trigger the generation of a new storyboard for each selected storyboard sketch, or a new storyboard for each selected image, using the set image model (i.e., using artificial intelligence technology) and the user-defined generation mode and parameters.

[0074] In some embodiments of this disclosure, it is not supported to select multiple storyboard videos in batches and then display a storyboard generation component to batch edit the generated multiple storyboard videos. In other words, it only supports selecting a single storyboard video and then using the storyboard generation component in follow mode to edit the selected single storyboard video individually. Specifically, when a user selects multiple storyboard videos in batches, the following can be displayed: Figure 9 The "Batch Regenerate or Delete" selection control shown allows users to regenerate the storyboard videos using the original generation parameters, essentially re-executing the generation process for multiple storyboard videos sequentially. Alternatively, users can click the "Delete" button to remove the selected storyboard videos from the storyboard canvas. Furthermore, when multiple types of footage are selected simultaneously (e.g., storyboard videos and storyboard diagrams), the storyboard generation component is not displayed (i.e., batch editing of different types of footage is not supported). In this case, the following display may also be available: Figure 9 The selection control shown is for "Batch Regenerate or Delete".

[0075] For the sake of brevity, since the settings for the generation mode and corresponding generation parameters in the storyboard generation component of the global mode and the following follow mode are similar, the specific settings for different generation modes and corresponding generation parameters in the storyboard generation component will be described in detail later.

[0076] In some embodiments, when a storyboard sketch or a storyboard image is selected, the storyboard generation component is in follow mode. The storyboard generation component in follow mode includes settings controls for the generation mode and corresponding generation parameters, as well as settings controls for at least one image editing function. The storyboard generation component in follow mode is displayed around the selected storyboard sketch or storyboard image. The image editing function is used to generate a new storyboard by editing the selected storyboard sketch or storyboard image. For example, Figure 10 Images (a) and (b) show schematic diagrams of the storyboard generation component in follow mode when a storyboard sketch or storyboard is selected. Figure 10 As shown, the storyboard growth component in Follow mode can specifically include a toolbar displayed above the selected image and an input box displayed below it. The toolbar displays various editing functions and selection buttons for two generation modes. The input box is used to assist in setting the corresponding editing parameters for the editing functions or the corresponding generation parameters for the generation modes. Therefore, the storyboard generation component in Follow mode can not only generate new storyboard drafts by setting the generation mode and corresponding generation parameters, but also generate new storyboard drafts by editing the selected storyboard sketch or storyboard draft.

[0077] like Figure 10 As shown, when a storyboard sketch or a storyboard image is selected, the storyboard generation component in follow mode includes at least one of the following image editing functions: partial redrawing, image expansion, multi-angle generation, and image re-enhancing. Among them, the partial redrawing function is used to redraw a part of the image (that is, to redraw a portion of the image), the image expansion function (i.e., image expansion) is used to expand the image content (that is, to add image content based on the selected image), the multi-angle generation function is used to generate images from multiple perspectives (that is, to generate images from multiple camera positions based on the selected image), and the image re-enhancing function is used to optimize the image for high definition (that is, to enhance the image quality of the selected image).

[0078] It should be understood that the toolbar of the storyboard generation component in Follow Mode may also include selection buttons for other auxiliary functions required for the selected storyboard sketch or storyboard diagram; for example, it may also include... Figure 10The download function selection button shown allows users to download the selected storyboard sketch or storyboard image to local storage. Since the storyboard image is generated based on the storyboard sketch, the toolbar can also include a regenerate function selection button when the storyboard image is selected. This regenerate function allows users to regenerate the storyboard image with a single click, based on the original generation parameters used to generate the selected storyboard image. It should be understood that the various auxiliary and editing functions provided in the above embodiments of this disclosure can be customized according to actual needs. The storyboard generation component provided in this embodiment supports the integration of various functional controls, and this embodiment does not impose any limitations on this.

[0079] in, Figure 10 The input box located below the image can be a collapsed input box. In the collapsed input box, the corresponding prompt for the selected storyboard sketch or storyboard in the image-raw mode can be displayed by default (e.g., ...). Figure 10 The input box (@original image@character) or generation parameters are not limited in this embodiment. When the user clicks the collapsed input box or clicks the selection button for any editing function or any generation mode in the toolbar, the input box can be switched to the expanded state and the corresponding function controls can be displayed. It should be understood that the content displayed in the expanded input box is different for different editing functions and different generation modes. In particular, the style of the expanded input box in the generation mode is similar to the style of the storyboard generation component in the global mode, and the specific details will be described later.

[0080] For example, when a user clicks the selection button for the partial redraw function, the following can be displayed: Figure 11The redraw component shown includes an expanded input box at the bottom of the image and a redraw function control at the top of the image. The redraw component supports multiple selection methods for the redraw area, such as: a box selection control representing the box selection method, a brush control representing the brush smearing method, and an eraser control representing the "eraser" erasing method. Among them, the box selection method and brush smearing are used to mark the area selected or smeared by the brush, while the "eraser" erasing method is used to remove the area marked by the box selection or brush smearing in the image. The brush smearing method can adjust the thickness of the brush smearing using the brush thickness adjustment control. It can also include a back control representing the back step function, a forward control representing the previous step function, and a clear control representing the clear function. The back step function is used to return to the previous operation (such as returning to the previous brush smearing state), the forward step function is used to return to the next operation (such as returning to the next brush smearing state), and the clear function is used to clear all operations performed on the image (i.e., clear all selected areas). Once the user has marked the areas that need to be redrawn, they can enter their redrawing requirements for the marked areas in the input box. For example, they can enter "remove the lights" in the input box. Then, they can click the "Generate Now" button to regenerate the storyboard based on the user-marked areas and the entered redrawing requirements.

[0081] like Figure 11 As shown, the prompt editing area also provides "reverse prompt" and "text optimization" functions. The reverse prompt function is used to deduce prompts based on the image after the user marks the redrawn area. The deduced prompts can include redrawing requirements. The text optimization function can be used to expand and optimize the prompts (i.e., redrawing requirements) input by the user. For example, a large language model can be used to analyze the image of the marked redrawn area to determine the possible operations that can be performed on the marked area, such as removing or replacing the redrawn area, to obtain the reverse prompts. A large language model can also be used to optimize the text based on the prompts input by the user to achieve the aforementioned cultural optimization function. It should be understood that users can use the above-mentioned reverse prompt function and text optimization function to generate more complete and higher-quality redrawing requirements, and this embodiment of the disclosure does not limit this. Figure 11 As shown, the input box can also display the default local redraw model (such as the model indicated by "Image Local Redraw - MJ-6.1") and the corresponding generation parameters (such as "4 images" representing the default generation of 4 storyboard images after local redrawing). It should be understood that if there are multiple local redraw models, the user can switch between them by clicking on the area where "Image Local Redraw - MJ-6.1" is located; this embodiment of the disclosure does not limit this. The local redraw model can achieve the local redrawing described above based on the prompt words and the selected redrawing area.

[0082] For example, when a user clicks the selection button for the image expansion function, the following can be displayed: Figure 12The expanded image control shown includes an expanded input box below the image and an expanded aspect ratio setting control above the image. The bottom layer of the image also displays an expanded area indicated by a checkerboard pattern. This expanded area can be displayed according to the user-set aspect ratio to indicate the area of ​​the expanded image. When the user clicks the expanded aspect ratio setting control, the following can be displayed: Figure 13 The aspect ratio selection list shown allows users to choose their desired aspect ratio. However, when a free aspect ratio is selected, users can adjust the size of the expanded image area by adjusting the boundary position of the checkerboard area. Users can also move the image's position or scale its size within the expanded image area indicated by the checkerboard. Figure 12 Move image to Figure 13 The locations shown are not limited in this embodiment. Similar to the local redrawing function described above, users can also input their expansion requirements (i.e., prompts) for the corresponding expansion area in the input box, or generate high-quality expansion requirements by reverse-engineering prompts or text optimization functions. When the user clicks "Generate Now," the expanded storyboard can be generated according to the expansion area indicated by the user and the input expansion requirements. It should be understood that if there are multiple image expansion models, users can switch image expansion models by clicking the area where "Image Expansion - MJ-6.1" is located, and this embodiment is not limited in this respect.

[0083] It should be understood that the multi-angle generation and high-definition image functions can be designed with reference to the aforementioned partial redrawing and image expansion functions, respectively, to implement the multi-angle generation and high-definition image functions. For example, the multi-angle function control can provide two methods: template generation or custom camera position generation. Users can generate storyboards from multiple camera angles using a multi-camera template, or generate storyboards from multiple camera angles using a custom camera position. Similarly, it can also provide... Figure 11 and Figure 12 The input boxes shown allow users to input their specific requirements for generating multi-camera storyboards. For example, the high-definition function controls allow users to set high-definition optimization requirements (such as detail enhancement, shadow enhancement, and other requirements that improve image quality), thereby generating higher-quality storyboards according to the user's settings. Specifically, image-based models can be used to generate multi-camera storyboards based on prompts, multiple camera templates (i.e., templates describing different shooting angles), and the original image (such as a storyboard or storyboard sketch). Furthermore, various image high-definition optimization models (such as detail enhancement models and shadow enhancement models) can be used to generate high-quality storyboards based on prompts and the original image.

[0084] In some embodiments, when a storyboard video is selected, the storyboard generation component is in follow mode, and the storyboard generation component in follow mode includes settings controls for at least one video editing function. The video editing function is used to generate a new storyboard by editing the selected storyboard video, and the storyboard generation component in follow mode is displayed around the selected storyboard video. Specifically, when a storyboard video is selected, the storyboard generation component in follow mode includes at least one of the following editing functions: lip-syncing function, video HD function, and frame capture function. The lip-syncing function is used to adjust the synchronization between the characters' lip movements and the dubbing audio in the storyboard video. The dubbing audio includes dubbing audio generated based on user-set text content and selected timbre, or uploaded local dubbing audio. The video HD function is used to perform HD optimization on the video (e.g., enhancing video clarity, enhancing details in the video frame, shadows, etc.). The frame capture function is used to capture video frames from the storyboard video (wherein, the captured video frames can be used as storyboard diagrams, storyboard sketches, or directly saved locally).

[0085] For example, Figure 14 This diagram illustrates a storyboard generation component in follow mode, displayed when a storyboard video is selected. Figure 14 As shown, the storyboard growth component in Follow mode can specifically include a toolbar displayed above the selected video and an input box displayed below it. The toolbar displays various video editing functions, and the input box is used to assist in setting the corresponding editing parameters (i.e., setting the corresponding prompts) for these functions. It should be understood that the toolbar of the storyboard generation component in Follow mode can also contain selection buttons for other auxiliary functions required for the selected storyboard video; for example, it can also include... Figure 14 The toolbar includes download and regenerate function buttons. Users can download the selected storyboard video to local storage using the download function. Since the storyboard video is generated based on storyboard sketches or diagrams, a regenerate function button can also be included when the storyboard video is selected. This regenerate function allows users to regenerate the storyboard video with a single click, based on the original generation parameters used to generate the selected storyboard video. It should be understood that the various auxiliary functions and video editing functions provided in the above embodiments of this disclosure can be customized according to actual needs. The storyboard generation component provided in this embodiment supports the integration of various functional controls, and this embodiment does not impose any limitations on this.

[0086] in, Figure 14The input box shown can be a collapsed input box, which may display a default prompt message (such as "Enter the prompt to modify" in the figure) or display the generation parameters of the storyboard video, etc. This embodiment does not limit this. When the user clicks the collapsed input box or clicks any video editing function selection button in the toolbar, it can switch to an expanded input box, displaying the corresponding function space. It should be understood that the content displayed in the expanded input box varies depending on the video editing function, and the corresponding function controls also differ. Those skilled in the art can design corresponding function controls to implement the corresponding video editing functions according to the actual needs of different video editing functions.

[0087] For example, when a user clicks the selection button for the lip-sync function, the following can be displayed: Figure 15 The lip-sync control shown may not be able to perform lip-sync processing in some storyboard videos where there may not be a consistently visible face, or the face may be far from the frame. Therefore, it is not displayed... Figure 15 The lip-sync control shown can also automatically perform facial recognition on the storyboard video. If facial recognition fails, it can display something like... Figure 16 The message shown in (a) reads, "The video does not contain a continuously and stably visible face, or the face is too far from the frame to match the lip movements," and the "Select Voice" and "Local Dubbing" buttons are unclickable. If face recognition passes, you can proceed as follows: Figure 16 As shown in (b), the storyboard video is displayed in the face recognition area. The "Select Voice" and "Local Dubbing" buttons are clickable. Users can enter the text they want the character to speak in the "Speech Content" field and generate dubbing audio by clicking the "Select Voice" button, thus synchronizing the character's lip movements in the storyboard video with the generated dubbing audio. Alternatively, users can click the "Local Dubbing" button to upload local dubbing audio from local storage, further synchronizing the character's lip movements in the storyboard video with the local dubbing audio. If multiple lip-sync models are available, users can... Figure 17 The example shown illustrates how clicking on the currently displayed model name (e.g., "Keling" in the image) displays a list of selectable models, allowing users to switch model types. This embodiment of the disclosure does not impose any limitations on this approach. For instance, a lip-sync model (i.e., speech-driven lip-syncing) can be used to generate videos where the character's lip movements are synchronized with the dubbing audio based on the dubbing audio and storyboard video. This embodiment of the disclosure does not impose any limitations on this approach.

[0088] For example, when a user enters text and clicks the "Select Tone" button, the following can be displayed: Figure 18The lip-syncing page shown displays pre-entered text in the input box under "Text Reading" (e.g., "This is my resume" in the image). Users can edit the text they want the character to speak in this input box. Users can select existing voices from the voice library using the controls in "Voice Selection," or search for voices by entering keywords in the search box in "Search Voices," or by selecting keywords using the controls in "Age," "Gender," and "Voice." The lip-syncing interface also displays the playback track of the storyboard video and can play the storyboard video simultaneously. After selecting a voice and setting the text content, users can click the "Add Voiceover" button to generate the voiceover audio. The generated voiceover audio is displayed on the audio track indicated by "Please add voiceover to the character from the right side of the face" below the video playback track. Users can also play the generated voiceover audio and regenerate it if they are not satisfied. After generating the voiceover audio, users can click the "Generate Now" button to synchronize the character's lip movements in the storyboard video with the voiceover audio; or, when the user clicks... Figure 16 After clicking the "Local Dubbing" button, the following will appear: Figure 19 The lip-syncing page shown above can also be found on the page above. Figure 18 Switch to the lip-syncing page shown Figure 19 The lip-syncing page shown is in Figure 19 The lip-syncing page shows that users can upload local audio using the "Upload Local Audio" control. After uploading local audio and clicking the "Add Voice-over" button, the local audio will be imported into the audio track indicated by "Please add voice-over to the character from the right side of the face" below the video playback track. Users can also play the uploaded audio and re-upload it if they are not satisfied. After clicking the "Generate Now" button, the lip movements in the storyboard video will be synchronized with the local audio to generate a lip-synced video (i.e., a lip-synced storyboard video). Figure 18 or Figure 19 After clicking the "Generate Now" button, you can return to the storyboard canvas and add a generated frame, which will then be displayed as shown below. Figure 20 The transition state shown is part of the lip-syncing video generation process; after generating the lip-syncing video, it can be done as follows: Figure 21 The generated lip-sync video is displayed and automatically focused on. The lip-sync video can then be played automatically. The storyboard generation component can also display the original storyboard video, timbre information, speech content, and generated dubbing audio from the lip-sync video, or display uploaded local dubbing audio.

[0089] It should be understood that the corresponding functional controls for both the high-definition video function and the frame capture function can be designed with reference to the various editing functions mentioned above, so as to realize the multi-angle generation function and the high-definition image function respectively. For example, the high-definition video function control allows users to set high-definition optimization requirements (such as detail enhancement, shadow enhancement, and other requirements that can enhance video quality), and then generate higher-quality storyboard videos according to the high-definition optimization requirements set by the user; the frame capture function control can, for example, display a video playback track and provide a control to select video frames to capture video frames from the storyboard video. The captured video frames can be used as storyboard diagrams or storyboard sketches or saved locally, and this disclosure embodiment does not limit this.

[0090] In some embodiments, during the process of selecting multiple storyboard sketches or multiple storyboard drafts in batches to queue for the generation of storyboard drafts, the generation grid of the storyboard canvas can display, as shown below. Figure 22 The queuing state shown in (a) can be displayed as follows during the generation of any storyboard or storyboard video. Figure 22 The generated state is shown in (b) above; if the storyboard or storyboard video is successfully generated, it can be displayed in the corresponding generated grid. If the storyboard or storyboard video fails to be generated, the corresponding generated grid can be displayed as shown below. Figure 22 The failure state shown in (c) indicates the failure to generate the storyboard; when the mouse focus moves to the generation cell where the storyboard generation failed (e.g., Figure 22 In (c)) it can be like Figure 22 As shown in (d), a delete button is displayed in the upper right corner of the generated cell, and the user can delete the generated cell by clicking the delete button.

[0091] Considering the potentially large number of generated storyboard images or videos, some of which users deem of high quality and wish to retain, the storyboard images or videos can be marked to facilitate user retention. Furthermore, to allow users to easily view the marked storyboard images or videos, in some embodiments, the storyboard creation interface may also include a view switching control. This control allows switching between the creation view, static storyboard view, and dynamic storyboard view. The storyboard creation interface defaults to the creation view (i.e., the aforementioned view). Figure 3 The method, as shown in the diagram, includes a creation view for generating storyboard drafts, a static storyboard view for displaying marked storyboard images, and a dynamic storyboard view for displaying marked storyboard videos. Users can switch between the creation view, static storyboard view, and dynamic storyboard view for quick and easy viewing of marked storyboard images or videos. Therefore, the method may further include:

[0092] In response to the focus moving to the storyboard or the operation of selecting the storyboard, a marker button is displayed on the storyboard; or, in response to the focus moving to the storyboard video or the operation of selecting the storyboard video, a marker button is displayed on the storyboard video and a video playback control is displayed while the storyboard video is played automatically.

[0093] In response to the user marking storyboards with the mark button and switching to the static storyboard view, the marked storyboards are displayed in the storyboard canvas under the static storyboard view in the order of the storyboard sketches. One storyboard sketch corresponds to one storyboard grid. When multiple storyboards are marked under a storyboard sketch, the storyboards are displayed in a carousel or page-turning format in the storyboard grid corresponding to the storyboard sketch. When no storyboards are marked under a storyboard sketch, a default unmarked prompt is displayed in the storyboard grid corresponding to the storyboard sketch.

[0094] In response to marking storyboard videos via the mark button and switching to the dynamic storyboard view, the marked storyboard videos are displayed in the storyboard canvas under the dynamic storyboard view in the order of the storyboard sketches. One storyboard sketch corresponds to one storyboard video cell. When multiple storyboard videos are marked under a certain storyboard sketch, the storyboard video cells corresponding to that storyboard sketch are displayed in a carousel or page-turning format. When no storyboard videos are marked under a certain storyboard sketch, the storyboard video cells corresponding to that storyboard sketch display a default unmarked message.

[0095] For example, when the mouse focus moves over the storyboard, it can display something like this: Figure 23 The marker button shown in (a) (in this case, the marker button is displayed as unmarked) will display the following when the user selects a storyboard (e.g., clicks on the storyboard): Figure 23 (b) shows the storyboard generation component in follow mode, simultaneously displaying a marker button; when the user marks the storyboard with this marker button, the marker button can switch as follows: Figure 23 The storyboard is marked as shown in (c); you can also unmark it by clicking the marked button. If the user has marked the storyboard, they can switch to the view using the view switching controls described above. Figure 24 The static storyboard view shown allows you to arrange the marked storyboard images in the order of the storyboard and adapt them to the screen rules. It also supports page zooming similar to a storyboard canvas, displaying only the marked storyboard images. This allows storyboard creators to get a more comprehensive overview of all marked storyboard images. If multiple storyboard images are marked under a single storyboard, they can be viewed in a carousel.

[0096] For example, such as Figure 24As shown, the static storyboard view displays marked storyboard images in the order of the storyboard sketches (i.e., the order of the scenes, such as SC01 to SC17). Each storyboard sketch corresponds to one storyboard grid. When multiple storyboard images are marked under a single storyboard sketch, moving the mouse focus to the storyboard grid corresponding to that sketch will display images as shown below. Figure 24 The selected state shown in ① indicates that the storyboards corresponding to the same storyboard sketch can be displayed in a carousel format within the storyboard grid corresponding to that storyboard sketch, or they can be displayed through a carousel format. Figure 24 The page turner shown in ① displays the storyboard images corresponding to the same storyboard sketch by turning pages; where only one storyboard image is marked under a certain storyboard sketch, clicking on the storyboard image in a storyboard grid will display as follows. Figure 24 In the selected state shown in ②, there is no page turner and no slideshow; when there is no storyboard under a certain storyboard sketch, the corresponding storyboard grid can display as shown in ②. Figure 24 The unlabeled prompt shown in ③ (such as "Images that have not yet been labeled") represents the storyboard images that are not labeled under this storyboard.

[0097] In some embodiments, if the current situation is as follows Figure 24 If you select a storyboard grid as shown in ① or ②, and then move the mouse focus to another storyboard grid, you will see... Figure 24 The selected state shown in ① or ② disappears, and the last viewed storyboard image is displayed. For example, assuming there are 4 storyboard images under storyboard SC01-2, clicking on the storyboard image corresponding to that storyboard will display as follows: Figure 25 ① As shown in the image, the selected state is displayed, and the first storyboard image is shown. At this time, you can use the page turner to turn to the next page, such as... Figure 25 The second storyboard shown in section ②, if the mouse focus is moved to... Figure 25 The storyboard diagram corresponding to SC02 shown in ③ is as follows: The storyboard diagram corresponding to SC01-2 is as follows: Figure 25 The selected state shown in ④ disappears and the second storyboard (i.e., the last storyboard viewed) is displayed.

[0098] In some embodiments, considering that users may need to modify a particular storyboard while browsing marked storyboard images, when a storyboard image needs modification, the user can double-click the storyboard image to return to the creation view and select it. Specifically, the method further includes: if the user performs a jump operation on any storyboard image grid in the static storyboard view and the storyboard image grid displays a storyboard image, the user jumps back to the creation view and selects the corresponding storyboard image; if the user performs a jump operation on any storyboard image grid in the static storyboard view and the storyboard image grid does not display a storyboard image, the user jumps back to the creation view and selects the corresponding storyboard sketch.

[0099] For example, when a user first moves the focus (e.g., mouse cursor) over any storyboard, it can display something like this: Figure 24 The guidance prompt shown in ④ (such as "Double-click to return to creation mode, where you can edit the image") means that users can return to the creation view and anchor the selected storyboard image by performing a jump operation on the storyboard grid (such as double-clicking the storyboard grid containing the storyboard image). When storyboard images are displayed in the same storyboard sketch, the user can double-click the storyboard grid that displays the storyboard image (such as...). Figure 26 The storyboard diagram indicated by ① in (a) can be switched back to, as shown in the image. Figure 26 In the creation view shown in (b) with the corresponding storyboard selected (i.e., displaying the storyboard generation component and marker button in follow mode), switching back to the creation view maintains the zoom level of the storyboard canvas before switching views. When the user executes a storyboard frame that does not display a storyboard (i.e., does not have a marked storyboard), such as... Figure 26 When performing a jump operation (such as double-clicking) on ​​the storyboard grid indicated in (a) ②, you can switch back to the previous screen. Figure 26 The creation view shown in (c) has the corresponding storyboard sketch selected (i.e., the storyboard generation component and marker button in follow mode are displayed). This allows for easy switching between the creation view and the static storyboard view.

[0100] For example, when the mouse focus moves over the storyboard video, it can display something like this: Figure 27 The marker button shown in (a) (in this case, the marker button is displayed as unmarked) can simultaneously play the storyboard video and display a playback progress bar. When the mouse clicks on the storyboard video, it can display the following: Figure 27 (b) shows the storyboard generation component in follow mode, simultaneously displaying marker buttons, and automatically playing the storyboard video while displaying player controls (including a draggable progress bar, pause and play buttons, a sound off or on button, and a full-screen button); when the mouse focus moves out of the player, it can be displayed as follows: Figure 27(c) shows a simplified progress bar; after the user marks the storyboard video using the mark button, the mark button can be switched as follows: Figure 27 The marked state is shown in (d); you can also unmark the storyboard video by clicking the marked button. If the user has marked the storyboard video, they can switch to the view using the view switching control. Figure 28 The dynamic storyboard view shown allows you to arrange the marked storyboard videos in the order of the storyboard and adapt them to the screen rules. It also supports page zooming similar to a storyboard canvas, displaying only the marked storyboard videos in the storyboard, which makes it easier for storyboard creators to have a more comprehensive overview of all marked storyboard videos. If multiple storyboard videos are marked under a single storyboard, they can be viewed in a carousel.

[0101] For example, such as Figure 28 As shown, the dynamic storyboard view displays marked storyboard videos in the order of the storyboard sketches (i.e., the order of the scenes, such as SC01 to SC17). One storyboard sketch corresponds to one storyboard video cell. When multiple storyboard videos are marked under a single storyboard sketch, moving the mouse focus to the storyboard video cell corresponding to that sketch will display... Figure 28 The selected state shown in ① (at which point the storyboard video can be played automatically) allows for the display of various storyboard videos corresponding to the same storyboard sketch in a carousel format within the storyboard video grid. Alternatively, it can be displayed via... Figure 28 The page turner shown in ① displays the storyboard videos corresponding to the same storyboard sketch by turning pages; where only one storyboard video is marked under a certain storyboard sketch, clicking on the storyboard video in a certain storyboard video cell will display as follows. Figure 28 In the selected state shown in ②, there is no page turner and no slideshow; when there are no storyboard videos marked under a certain storyboard sketch, the corresponding storyboard video cell can display as shown in ②. Figure 28 The unlabeled prompt shown in ③ (such as "videos that have not yet been labeled") represents the unlabeled video segments under that segment.

[0102] In some embodiments, similar to the static storyboard video described above, if the current is as follows... Figure 28 If you select a specific video frame as shown in ① or ②, and then move the mouse focus to another video frame, the selection will be affected. Figure 28When the selected state in ① or ② disappears, the last viewed storyboard video is displayed. For example, assuming there are 4 storyboard videos under storyboard SC01-2, clicking the storyboard video cell corresponding to that storyboard will display the selected state and show the first storyboard video. At this time, you can use the page turner to turn to the second storyboard video. If the mouse focus moves to the storyboard video cell corresponding to storyboard SC02, the selected state of the storyboard video cell corresponding to SC01-2 will disappear and the second storyboard video (i.e., the last viewed storyboard video) will be displayed.

[0103] In some embodiments, similar to the static storyboard view described above, considering that users may need to modify a particular storyboard video while browsing marked storyboard videos, when a storyboard video needs modification, the user can double-click the storyboard video to return to the creation view and select the storyboard video. Specifically, the method further includes: if the user performs a jump operation on any storyboard video cell in the dynamic storyboard view and the storyboard video cell displays a storyboard video, the user jumps back to the creation view and selects the corresponding storyboard video; if the user performs a jump operation on any storyboard video cell in the dynamic storyboard view and the storyboard video cell does not display a storyboard video, the user jumps back to the creation view and selects the corresponding storyboard sketch.

[0104] For example, when a user first moves the focus to any scene in the video, the following can be displayed: Figure 28 The guidance prompt shown in ④ (such as "Double-click to return to creation mode, where you can edit the video") means that users can return to the creation view and anchor the selected storyboard video by performing a jump operation on the storyboard video frame (such as double-clicking the storyboard video frame containing the storyboard video). When there are multiple storyboard videos in the same storyboard sketch, the storyboard video selected by the user when double-clicking the storyboard video frame is the storyboard video selected in the returned creation view. For example, when the user double-clicks... Figure 29 When the SC02-2 video frame indicated by ① in (a) is switched back, it can be switched back. Figure 29 In the creator view shown in (b), the double-clicked storyboard video is selected (i.e., the storyboard generation component and marker button in follow mode are displayed). Switching back to the creator view maintains the zoom level of the storyboard canvas before switching views. When the user... Figure 29 When performing a jump operation (such as double-clicking) on ​​a storyboard video cell (as shown in ② of (a)) that does not display storyboard video (i.e., does not have a storyboard video label), you can switch back to the previous screen. Figure 29 The creation view shown in (c) is displayed, and the storyboard sketch corresponding to the storyboard video frame is selected (i.e., the storyboard generation component in follow mode is displayed). In this way, it is easy to switch between the creation view and the dynamic storyboard view.

[0105] When generating storyboard drafts (such as image-to-image and image-to-video) using artificial intelligence technology, the storyboard generation component can prioritize using character images, scene images, or style images preset by the user in the asset management library as the basis for generation. This effectively constrains the randomness of the AI ​​mode's output, ensuring that the generated storyboard drafts maintain consistency and uniformity in character appearance, scene setting, and art style, which is a key technology for producing long-form narrative content. Therefore, the storyboard creation interface also includes: a character library management component, a scene library management component, a style library management component, and an inspiration library management component. The character library management component stores and manages preset character images, where the same character has at least one character image from a different perspective; the scene library management component stores and manages scene images; the style library management component stores and manages preset style images; and the inspiration library management component stores and manages preset storyboard sketches.

[0106] For example, such as Figure 3 The "Character," "Scene," "Style," and "Inspiration Library" shown can be the entry points for the aforementioned Character Library Management Component, Scene Library Management Component, Style Library Management Component, and Inspiration Library Management Component, respectively. When the user clicks the "Character" entry point, the following will be displayed: Figure 30 The character library management component shown allows users to import character images (e.g., upload images from local storage or drag and drop storyboard images from the storyboard canvas), delete existing images, and upload multiple images of the same character from different angles. Users can select different angles to ensure the quality of the raw image, and the character names can be edited. The characters in the character images can be used as the basis for generating characters in the storyboard. When the user clicks the "Scene" entry, the following will be displayed: Figure 31 The scene library management component shown allows users to import scene images (e.g., upload scene images from local storage or drag and drop storyboard images from the storyboard canvas) and delete existing scene images. Scenes in these scene images serve as the basis for generating scenes in storyboard drafts. When the user clicks the "Style" entry, the following will be displayed: Figure 32 The style library management component shown allows users to import style images (e.g., upload style images from local storage or drag and drop storyboard images from the storyboard canvas) and delete existing style images. The style of the style images serves as the basis for the style of the generated storyboard drafts. When a user clicks the "Inspiration Library" entry, the corresponding inspiration library management component is displayed. The style of this component can be referenced as described above. Figure 31 or Figure 32 The various component styles shown can be used to import storyboard sketches (for example, you can upload storyboard sketches from local storage or drag and drop storyboard sketches from the storyboard canvas), and you can also delete stored storyboard sketches.

[0107] In some embodiments, such as Figure 3 The storyboard creation interface shown may also include: a download component (i.e., the component indicated by "Download" in the figure) and an image upload component (i.e., the component indicated by "Image Upload" in the figure). The download component is used to download the storyboard sketches and storyboard drafts on the storyboard canvas to the local machine in the form of a storyboard table. The image upload component is used to upload the storyboard sketches from the local machine (i.e., users can upload storyboard sketches through this image upload component).

[0108] In some embodiments, the storyboard creation interface also includes an editor entry (such as...). Figure 3 The method further includes: in response to a user navigating to the editor's editing interface via the editor entry, importing the marked storyboard images and storyboard videos as storyboard materials into the editor, so as to use the storyboard materials for video editing in the editing interface. That is, both the marked storyboard images and storyboard videos can be placed into the editor as material packages and adjusted according to modifications in the storyboard creation module, wherein the storyboard videos can be automatically placed into the editor track. For example, Figure 33 This document illustrates an editing interface for a video editor. It should be understood that those skilled in the art can design the functional controls and interface layout included in the editing interface by referring to known video editor software, and this disclosure does not limit such designation. This approach allows users to conveniently edit videos using storyboard footage generated by the storyboard creation interface, eliminating the need for switching between different software and improving the user experience.

[0109] According to the method of this disclosure, by replicating the professional storyboard format commonly used by users and combining it with a new artificial intelligence generation mode, a multi-modal creative canvas is formed, which is more in line with the usage habits of relevant practitioners. In addition, by using artificial intelligence generation, the workload of hand-drawn storyboards and video production is reduced, creating an integrated canvas that is "manageable, generative, and previewable," realizing a closed loop from idea to visual result.

[0110] In practical applications, such as Figure 34 The framework structure of the intelligent canvas system shown is used to implement the above-mentioned storyboard creation method, such as... Figure 34As shown, this intelligent canvas system consists of multiple layers. The bottom layer is actually an infinite canvas modality (supporting the movement and scaling of the storyboard canvas). Above it are the storyboard canvas, the input layer (such as the storyboard generation component), the asset management layer (i.e., the asset management library, a database used for centralized storage, management, and retrieval of reusable creative elements, such as the character library management component, scene library management component, style library management component, inspiration library management component, etc.), the view switching layer (i.e., the aforementioned view switching control), and the top layer is the tool layer (supporting AI generation tools (i.e., the functions implemented in the storyboard creation interface of this embodiment) and video editing tools (i.e., an editor with editing capabilities). Through this canvas system, multiple modalities can be integrated to form an integrated canvas-based intelligent storyboard creation method that is "manageable, generative, and previewable".

[0111] In this embodiment, AI generation capabilities are integrated as the underlying core through the storyboard canvas, deeply embedded in the user interface's workspace, providing an intelligent creative environment with real-time generation, management, and preview functions. The structured storyboard table embodied in the storyboard canvas allows for the organization and presentation of storyboard content in a tabular format. In this solution, each row (one storyboard) is linked to the preview in the generation area on the right, and AI generation or content upload can be directly triggered within the table, achieving a combination of standardized storyboard management and operational flexibility.

[0112] There is no similar AI storyboard canvas format based on a storyboard pattern in related technologies. This disclosure replicates the professional storyboard pattern commonly used by users, while combining it with a novel AI generation mode to form a multi-modal storyboard creation canvas. Furthermore, during AI generation (e.g., image-generated images, image-generated videos), it prioritizes using characters, scenes, or styles preset by the user in the asset management library as the generation basis. This effectively constrains the randomness of AI output, ensuring that the generated storyboard images maintain consistency and uniformity in character design, scene setting, and artistic style. No similar solutions exist in related technologies.

[0113] The generation modes in the storyboard generation component of this disclosure will be described in detail below.

[0114] As mentioned above, the generation modes can include image-to-image mode and image-to-video mode. Image-to-image mode can be used to generate storyboard sketches, while image-to-video mode can be used to generate storyboard videos. Therefore, the generation mode determines the type of storyboard generated based on the storyboard sketch (i.e., whether to generate a storyboard sketch or a storyboard video). Generation parameters can include model-customized parameters and common parameters. These parameters determine the quality of the generated storyboard. The required model-customized parameters and common parameters differ depending on the generation mode.

[0115] In some embodiments, the common parameters in the raw image mode include: raw image prompts and raw image basis. The raw image basis includes at least one of the following: original images, reference images, style images, character images, and scene images required for generating storyboards. The original images (also called pad images) include selected storyboard sketches or storyboards. Reference images allow the raw image model to reference certain elements or the overall composition of the reference images. Style images allow the raw image model to reference the style. Character images allow the raw image model to reference character images. Scene images allow the raw image model to reference scenes. Raw image prompts are used to specify the raw image basis in the settings. Based on the description of the storyboard generation requirements, that is, the storyboard model can be instructed to generate storyboards according to the storyboard prompts input into the storyboard model. For example, if the user sets the storyboard type to style transfer and sets the original image, scene image, and character image, the storyboard prompt can be described as "while keeping the composition of the original image unchanged, replace the character on the right side of the original image with the character image in the character image and refer to the scene in the scene image". It should be understood that the specific content of the storyboard prompt can be edited according to the user's actual storyboard creation requirements and the storyboard basis set by the user. This embodiment of the disclosure does not limit this. The model customization parameters in the raw image mode include: raw image type, the model type of raw image available under different raw image types, and the corresponding raw image parameters; raw image parameters include: the aspect ratio of the storyboard to be generated (e.g., the user can set the aspect ratio of the original image, 16:9, or 4:3, etc.), the number (e.g., the user can set to generate 1, 2, 3, or 4 storyboards, etc.), and the resolution (e.g., the user can set the resolution of the generated storyboard to 4K, 1080P, 720P, etc.); raw image types include direct raw image type or style transfer type, style The transfer drawing type is used to generate corresponding storyboards according to common parameters while keeping the composition of the original image unchanged; the direct image generation type is used to generate corresponding storyboards according to common parameters without restricting the composition of the original image; the original image includes the selected storyboard sketch, that is, the image generation mode provides two types of image generation to facilitate users to choose according to actual needs; the above image generation model can be any known artificial intelligence model in the art that generates images based on images, and the embodiments of this disclosure do not limit the model type and number of image generation models that can be selected under different image generation types.

[0116] In some embodiments, the generation parameters in the image-generated video mode include: video generation type, model types of video generation models selectable under different video generation types and corresponding video generation parameters, video generation criteria and prompts to be set for different video generation types; the video generation type includes at least one of the following: first and last frame type, multi-frame type, camera movement type, and motion imitation type; the first and last frame type is used to generate a storyboard video by setting the first frame image and / or the last frame image, the multi-frame type is used to generate a storyboard video by setting multiple frame images, the camera movement type is used to generate a storyboard video based on images that meets the camera movement template and / or camera movement requirements set by the user; the motion imitation type is used to generate a storyboard video based on images. The generated video is a storyboard video that meets the user's settings for reference action videos. The parameters of the generated video include: the duration of the storyboard video to be generated (e.g., the user can set the duration to 5 seconds, 10 seconds, etc.), and the resolution (e.g., the user can set the resolution of the generated storyboard video to be 4K, 1080P, 720P, etc.). The criteria for generating the video include: a selected storyboard sketch or storyboard diagram, or, further, at least one of: a camera movement template (e.g., a video describing camera movement), camera movement requirements (e.g., textually described camera movement requirements, which can be reflected in the generated video prompts), and a reference action video (i.e., a video describing action). The above video generation model can be any known AI model for generating video based on images in the art. This disclosure does not limit the type or number of video generation models that can be selected for different video generation types. The selected storyboard image or storyboard sketch can be used as the first and / or last frame in the first and last frame class, or directly used for setting multiple frames and their order in the multi-frame class. Camera movement templates and requirements are used in the camera movement class, and reference motion videos are used in the motion template class. The generated video prompts describe the generation requirements of the storyboard video based on the set generated video criteria. That is, the generated video prompts input to the video generation model instruct the model to generate the storyboard video according to the prompts. For example, if the user sets the video generation type to the first and last frame class and sets the first and last frame images, the generated video prompts can be described as "Generate a video using the first and last frame images." It should be understood that the specific content of the generated video prompts can be edited according to the user's actual storyboard creation needs and the generated video criteria set by the user; this embodiment does not limit this.

[0117] As mentioned above, the storyboard generation component includes a follow mode and a global mode. The main difference between follow mode and global mode is that follow mode includes more editing functions and auxiliary functions than global mode. Furthermore, in follow mode, the button for selecting the generation mode is placed in the toolbar, as described above. Figure 7 and Figure 8 This demonstrates a storyboard generation component in global mode when multiple storyboard sketches or multiple storyboard images are selected in batches. Figure 14This shows the storyboard generation component in follow mode when the user clicks on a storyboard sketch or storyboard diagram. Figure 35 Images (a) and (b) show the storyboard generation component in Follow Mode, displayed when a user selects a storyboard sketch and clicks on either the Graph-to-Graphic or Graph-to-Video mode, respectively. Figure 36 Images (a) and (b) show the storyboard generation component in Follow Mode, displayed when a user selects a storyboard sketch and clicks on either the image-to-image mode or the image-to-video mode, respectively. As can be seen from the images above, Figure 35 and Figure 36 Compared to Figure 7 and Figure 8 Aside from the different selection buttons for the generation mode, other aspects (such as the method of setting the generation parameters corresponding to the generation mode) are basically the same. This disclosure mainly uses the storyboard generation component in global mode to introduce the setting method of generation parameters in different generation modes. For the storyboard generation component in follow mode, the setting method in global mode can be referred to, and will not be repeated.

[0118] In some embodiments, the global mode has a collapsed state and an expanded state, such as Figure 3 The input box indicated by "Please enter your idea" at the bottom of the interface shown is a storyboard generation component in collapsed global mode. For example, Figure 37 The image shows the storyboard generation component in the collapsed state when the user clicks it (e.g., ...). Figure 3 The method may further include: displaying the expanded global mode of the storyboard generation component when the collapsed version is not shown at the bottom of the interface (in which case the user has not selected any storyboard sketches or storyboard images in batches). Figure 3 The image shows the collapsed global mode storyboard generation component. Clicking the collapsed global mode storyboard generation component switches to the expanded global mode storyboard generation component as shown in image 37. When the user selects multiple storyboard sketches or multiple storyboard images in batches, the collapsed global mode storyboard generation component is displayed as shown in image 37. Figure 38 The image shown is a storyboard generation component in global mode in an expanded state.

[0119] Among them, in the display as Figure 37 In the case of the storyboard generation component in the image-to-image mode shown, users can switch generation modes by clicking "Image Generation" and "Video Generation" at the top of the component. It should be understood that, due to... Figure 37 This is the storyboard generation component displayed in global mode when the user has not selected storyboard sketches in bulk. Therefore, this storyboard generation component does not display any storyboard sketches or storyboard images. Figure 37(a) shows the storyboard generation component under the direct image generation class. Figure 37 (b) shows the storyboard generation component under the Style Transfer category, which users can click to generate a storyboard. Figure 37 The reference diagram setting control indicated by "Reference" shown in (a) or Figure 37 In (b) above, the original image upload control indicated by "Original Image" is used to import storyboard sketches or storyboard images. Additionally, character images can be set using the character setting control represented by "Character," scene images can be set using the scene setting control represented by "Scene," and style images can be set using the style setting control represented by "Style." This embodiment of the disclosure does not impose limitations on these aspects. Regardless of whether the user imports storyboard sketches or storyboard images, the default model customization parameters can still be displayed in the storyboard generation component in the expanded global mode. The method for adjusting the model customization parameters can be referred to later. Figure 39 The implementation method shown will not be elaborated here.

[0120] As mentioned above, the storyboard generation component can display the raw image mode by default. Users can switch the generation mode by clicking "Batch Generate Images" and "Video Generation" at the top of the component. When a user selects multiple storyboard sketches or multiple storyboard images in batches, the selected sketches or images can be displayed in the storyboard generation component, along with the default model customization parameters. For example, such as... Figure 38 As shown in (a), multiple storyboard sketches or multiple storyboard images selected by the user can be displayed as original images in the storyboard generation component. The default image type is "Style Transfer," and the input box can automatically fill in the prompt "Keep the original composition unchanged." Users can switch to "Style Transfer" by clicking "Style Transfer" at the bottom of the component. Figure 38 (b) shows the "raw image mode (representing direct raw image type)". In this mode, multiple storyboard sketches or storyboard images selected by the user can be displayed as reference images in the storyboard generation component. The bottom of the component can also display the default raw image model (such as Seedream 4.0) and the corresponding raw image parameters, such as... Figure 38 As shown in (a), the default raw image model under the Style Transfer category is "Seedream 4.0", the default aspect ratio is "Original (i.e., the aspect ratio of the original image)" and the grayed-out "Original" indicates that the aspect ratio under the Style Transfer category cannot be adjusted, the default resolution is "4K", and the default number is "4 images". Figure 38As shown in (b), the default image model for direct-generated images is "Seedream 4.0", with a default aspect ratio of "16:9", a default resolution of "4K", and a default number of "4 images". If the user does not want to use the default model parameters, except for the non-adjustable parameters (such as the grayed-out "original image" mentioned above), all the highlighted parameter areas can be adjusted by clicking on them. For example, clicking on "Seedream 4.0" will display... Figure 39 The model selection control shown in (a) allows users to select a raw image model by clicking its name. Clicking "Raw Image Mode" will display the following... Figure 39 The image type switching control shown in (b) allows users to switch between image types. It should be understood that the above information can be used to adjust aspect ratio, sharpness, and quantity. Figure 39 The selection controls shown are used to make adjustments, but this embodiment of the disclosure does not limit this. The types of raw image models that can be selected under different raw image types may differ. For example, some raw image models can be used for direct image generation but not for style transfer. Therefore, the names of raw image models that cannot be used under the currently selected raw image type can be grayed out in the model selection control (representing that they are not selectable), or the raw image models that cannot be used under the currently selected raw image type can be directly not displayed. This embodiment of the disclosure does not limit this.

[0121] As mentioned above, the storyboard generation component may include reference image setting controls (such as...). Figure 37 or Figure 38 Controls in the "Reference" section), character image setting controls (such as...) Figure 37 or Figure 38 Controls in the "Character" section), scene graph setting controls (such as...) Figure 37 or Figure 38 Controls in the "Scene" section and style map setting controls (such as...) Figure 37 or Figure 38 The controls in the "Style" section allow users to set reference images, character images, scene images, and style images, and also import original images (such as storyboard sketches). The reference image setting control is used to select at least one reference image from the imported reference images; the selected reference image is displayed next to the control. It is also used to import or delete reference images, which can be any image. The scene image setting control is used to select one or more scene images from the preset scene library; it is also used to import or delete scene images; the selected scene image is displayed next to the scene setting control. The style image setting control is used to select one or more style images from the preset style library; it is also used to import or delete style images; the selected style image is displayed next to the scene setting control.

[0122] For example, Figure 40 A schematic diagram of the control settings is shown in the reference figure; wherein, Figure 40 (a) shows the reference image setting control when there is no historical reference image. Users can import reference images by "uploading local images" or "dragging" storyboard sketches or generated storyboard images from the canvas. Imported reference images are used as historical reference images. Figure 40 (b) shows the reference map setting controls when a historical reference map is available. Users can select any reference map from the displayed historical reference maps, or they can click on the historical reference map. Figure 1 The side import control allows importing reference images from local uploads or by dragging and dropping from the storyboard canvas; in some embodiments, a maximum number of reference images can be set, for example, the maximum number of reference images can be set to 1 (that is, only one reference image can be selected). Figure 40 (c) shows the style of a historical reference image selected by the user (e.g., by clicking the mouse to select the reference image). Figure 40 (d) shows the style when a user selects multiple historical reference images. The selected historical reference images are displayed sequentially on the reference image settings control side according to the selection order. Selected reference images can be identified by a highlighted border or a selection icon (such as the "√" in the upper right corner of the second historical reference image in the first row). The selected reference images are then displayed on the reference image settings control side. The reference image settings control also supports deleting historical reference images. When a user wants to delete a historical reference image, they can do so by right-clicking or moving the mouse focus to the image, which will display a delete button (as indicated by the "-" in the upper right corner of the second historical reference image in the first row). The user can then click this delete button to delete the historical reference image.

[0123] In some embodiments, the maximum number of reference images can also be set to multiple (i.e., multiple reference images can be selected), such as... Figure 41 As shown in (a), if the maximum number of reference images is set to 5, when a user selects 5 historical reference images, the system will display the message "Maximum 5 images can be added" and simultaneously gray out other historical reference images (i.e., unselectable). When the user deselects any historical reference image (e.g., by clicking on the already selected historical reference image to deselect it), the other historical reference images will become selectable again. Specifically, when a user sets a reference image and exits the reference image setting control (e.g., by clicking on a blank area of ​​the storyboard generation component), the system will display the following: Figure 41 The storyboard generation component shown in (b) can display the following when the user sets multiple reference images, uploads them after reaching the required number, and exits the reference image settings control: Figure 41The storyboard generation component shown in (c) can display the following on the reference image after right-clicking or moving the mouse focus to the reference image displayed in the storyboard generation component: Figure 41 The delete button shown in (d) allows users to delete a set reference image. It should be understood that reference images are not a mandatory parameter; that is, storyboards can be generated without referencing elements or compositions in a reference image.

[0124] In practical applications, the implementation methods for style image setting controls and scene image setting controls can refer to the specific implementation methods of the reference image setting controls mentioned above. The similarities will not be elaborated upon. The difference lies in that the style image setting control displays a preset style image from the style library by default, while the scene image setting control displays a preset scene image from the scene library by default. Of course, style image setting controls and scene image setting controls can also import style images or scene images from local uploads or drag-and-drop from the storyboard canvas. Imported style images or scene images can be simultaneously saved to the style library or scene library. It should be understood that style images or scene images are not mandatory parameters; that is, storyboards can be generated without referencing styles in the style images or scenes in the scene images.

[0125] Considering that users may also want to control the proportion of reference images, style images, or scene images referenced by the raw image model when generating storyboards, and the greater the proportion, the more content the model references when generating storyboards, in some embodiments, the method may further include: if the user has set any one of the reference images, style images, and scene images, displaying the reference weight of any one of the set reference images, style images, and scene images, as well as a weight adjustment control, so as to adjust the reference weight through the weight adjustment control. The reference weight is used to determine the proportion of any one of the reference images, storyboards, and scene images referenced when generating storyboards using the raw image model.

[0126] For example, taking a reference image as an example, when the user has already set the reference image using the aforementioned reference image setting control, the following can be displayed: Figure 42 The storyboard growth component shown in (a) has a percentage bar (i.e., a weight adjustment control) displayed on the side of the reference image "Reference 1". The size of the white percentage in the percentage bar indicates the size of the default reference weight set for that reference image; a larger white percentage indicates a larger reference weight. When the mouse focus moves over the percentage bar, it can be highlighted (e.g., by highlighting or adding a border). When the user clicks the percentage bar, the following can be displayed: Figure 42 The adjustment layer shown in (b) allows users to adjust the reference weights by moving the position of the solid circle within it. It should be understood that the reference weights for both style maps and scene maps can be set in the same way as described above. Figure 42The settings shown for the reference diagram will not be elaborated upon here.

[0127] The character image settings control is used to select one or more character images from a preset character library. Different characters can have character images from one or more angles. It is also used to import character images of any character, delete or add character images of existing characters, and edit character names. The selected character images are displayed next to the character image settings control. For example, Figure 43 The diagram illustrates the character creation settings controls. When a user clicks the control at the "Character" position in the storyboard generation component, the following can be displayed first: Figure 43 (a) shows representative images of several preset characters in the current character library (these representative images can be thumbnails of character images from a default angle (e.g., a frontal angle)) to display selectable characters to the user. Character images can also be imported via an import control, either by uploading from the local machine or by dragging and dropping from the storyboard canvas. Any character image can be used to describe the character's appearance; a character image can also be called a character illustration. When the mouse focus moves over any character's representative image, it can be viewed as follows: Figure 43 As shown in (b), multiple selectable angles of the character's image are displayed. A delete button is also displayed in the upper right corner of the character's representative image. When the user clicks the delete button on any character's representative image, the character and all its character images can be deleted from the character image settings control. When the mouse focus moves to any character image of any character, the following can be done: Figure 43 As shown in (c), a delete button is displayed in the upper right corner of the character illustration. Users can delete the illustration for any character by clicking the delete button displayed on that character illustration, or they can add a new illustration for that character by clicking the "Add" button; for example... Figure 43 As shown in (b) and (c), when displaying the character's illustration, a name editing control can also be displayed at the character's name (as shown in "Character 4" in the figure). When the user clicks on the name editing control, the following can be displayed: Figure 43 The name editing box shown in (d) allows users to edit character names. In some embodiments, if there are many characters or many character images of the same character, the character image setting control can also support a sliding operation to switch between the representative image or character image displayed in the control. When a user selects a character image or multiple character images in the character image setting control, the selected character images can be displayed as follows: Figure 44 As shown in (a) and (b), the selected character images are marked with a highlight border or a selection icon. The selected character images can also be displayed sequentially in the storyboard generation component.

[0128] In practical applications, there can be no upper limit to the number of characters that can be set, meaning users can set any number of characters. However, there can be an upper limit to the number of character images that can be selected for the same character. For example, it can be set that only one character image can be selected for each character; of course, it can also be set to multiple character images, or there can be no upper limit on the number of character images that can be selected. This embodiment of the present disclosure does not impose any restrictions on this. After exiting the character image setting control (e.g., clicking on a blank area of ​​the storyboard generation component) and returning to the storyboard generation component, if the user wants to delete a set character image, they can do so by right-clicking or moving the mouse focus to the displayed character image, as shown above. Figure 41 The delete button shown in (d) allows users to delete previously set character illustrations. It should be understood that character illustrations are not a mandatory parameter; that is, storyboards can be generated without referencing character images from existing illustrations.

[0129] Considering that users may also have the need to crop the pre-set raw image references, such as wanting the reference image to have a specified aspect ratio, or wanting to use only a portion of the scene in the scene image, therefore, in some embodiments, the method may further include:

[0130] When the focus moves to a reference image, character image, scene image, or style image displayed in the storyboard generation component, a corresponding magnified image is displayed, which includes an image cropping button.

[0131] In response to the user triggering the image cropping button, a cropping page for the enlarged image is displayed. The cropping page displays cropping components that can be cropped according to a specified aspect ratio or a free aspect ratio to crop the enlarged image.

[0132] In response to the cropping page completing the cropping operation on the enlarged image and exiting the cropping page, the image corresponding to the enlarged image in the storyboard generation component is updated to the cropped image.

[0133] For example, taking a character image as an example, when the mouse focus moves to the character image displayed in the storyboard generation component, it can display something like this: Figure 45 The image shown is an enlarged view of the character. A cropping button can be displayed in the lower left corner of this enlarged view. When the user triggers (e.g., clicks) this cropping button, the following will be displayed: Figure 46 The cropped page for the enlarged image shown in (a) may include an aspect ratio setting button (as shown at "16:9" in the image). Users can click this aspect ratio setting button to display... Figure 13 The image shows a list of aspect ratios to select the cropping ratio. Selectable cropping ratios can include multiple specified aspect ratios or a free aspect ratio. For example, when the user selects a free aspect ratio, it can be displayed as shown below. Figure 46In the cropping page shown in (b), the cropping frame under free aspect ratio has no fixed aspect ratio, while the cropping frame under specified aspect ratio has a fixed aspect ratio. Users can select the area to be cropped by adjusting the position and size of the cropping frame. Users can then click the OK button to perform the cropping process, obtain the cropped image, and exit the cropping page. At this time, the storyboard generation component displays the cropped character image. Alternatively, users can click the exit button (the button shown by "×" in the figure) to exit the cropping page. At this time, the storyboard generation component still displays the original character image. In some embodiments, the cropping page can also display a "Re-upload" button, which users can use to reset the character image (this process can refer to the implementation method of the character image setting control mentioned above, and other interaction methods can also be set according to actual needs to reset the character image. This disclosure embodiment does not limit this).

[0134] It should be understood that the aspect ratio setting button, cropping box, and confirmation button in the above-mentioned cropping interface can together constitute a cropping component to achieve cropping processing of the set character image. The same applies to reference images, scene images, and style images. Figures 45 to 46 The implementation shown is used for trimming, but this embodiment does not limit the scope of the invention.

[0135] In one possible implementation, such as Figure 37 As shown, the storyboard generation component can also include an original image settings control (as shown in the "Original Image" section of the image). The original image settings control can be used to import or delete original images (such as storyboard sketches or storyboard diagrams). Imported original images are displayed in the original image settings control. The maximum number of original images that can be imported can be set, and it also supports deleting imported original images. For example, a delete button can be displayed when the mouse focus moves over an imported original image (see above). Figure 41 (See the delete button shown in (d)). Users can delete imported original sketches by clicking this button. If a user has selected multiple storyboard sketches or multiple storyboard images as original sketches, they can upload more storyboard sketches or storyboard images as original sketches through the original sketch settings control, or they can delete the selected storyboard sketches. When a user imports a new storyboard sketch or storyboard image as an original sketch to generate a new storyboard image through the original sketch settings control, the imported storyboard sketch or storyboard image can be arranged after the storyboard sketches already displayed in the storyboard screen as a newly added storyboard sketch.

[0136] In some embodiments, Figure 47This demonstrates the storyboard generation component in the image-to-video mode. When a user selects multiple storyboard images or sketches and switches to image-to-video mode, the default video generation type and required settings are determined based on the number of images selected. For example, if the number of selected images is less than a specified number (e.g., 5 images, where the specified number can be the upper limit for the number of images that can be selected in a multi-frame format), the default display can be as follows. Figure 47 The multi-frame storyboard generation component shown in (a) displays multiple images selected by the user (frames 1, 2, and 3 in the image). The user can also import a fourth frame using the "+" control. If the number of selected images exceeds a specified number (e.g., 5 images), the following can be displayed: Figure 47 The default video generation type shown in (b) is a storyboard generation component with first and last frames. In this case, multiple selected images can be used as the first frame to generate separate storyboard videos. For example, if 6 images are selected, each of the 6 images can be used as the first frame to generate 6 separate storyboard videos. Furthermore, by clicking the video generation type settings button, when the number of selected images is less than a specified number, the following can be displayed: Figure 47 The toggle control shown in (c) allows you to switch to other video generation types. For example, if the user selects fewer than the specified number of images but more than two images, they can switch to the first and last frame type (and...). Figure 47 (same as (b) except for the number of images). If the user selects fewer than the specified number of images but equal to 2 images, the function can be switched to... Figure 47 (d) shows the first and last frame class (in this case, the two selected images are used as the first and last frames to generate a storyboard video, and the first and last frames can interact with each other), and if the number of images selected by the user is less than the specified number, it can also switch to such Figure 48 The camera movement category shown in (a) (where the selected image is used as the base image, and a storyboard video with the same camera movement as the template is generated by uploading a camera movement template, and camera movement requirements, i.e., camera movement prompts, can also be entered in the prompt editing area to generate a storyboard video that meets the camera movement requirements), or you can switch to, as shown in (a) Figure 48 (b) shows the action imitation class (in this case, the selected image is used as the base image, and a storyboard video with the same actions as the reference action video is generated by uploading a reference action video); if the number of images selected by the user is greater than the specified number, then as shown in the example. Figure 48 In the switch control shown in (c), the "Smart Multi-Frame" option represents a multi-frame class that cannot be selected. In this case, you can switch to the camera movement class (as described above). Figure 48 (same as (a) above, only the number of images is different), or you can switch to the action mimicry category (same as above). Figure 48 (b) is the same, except the number of images is different.

[0137] In some embodiments, when a user selects a storyboard sketch or a storyboard image, the default video generation type can be set to the first and last frame type, for example, it can display as follows: Figure 35 The storyboard generation component shown in (a) defaults to using the selected image as the first frame. This selected storyboard sketch can be used as the first frame to generate the storyboard video (the first frame under the first and last frame categories is required, while the last frame is optional). Users can then import the last frame using the "+" import control (which can then be displayed as described above). Figure 47 (d)). Of course, when the user selects a storyboard sketch or a storyboard image, the default video generation type can be multi-frame. In this case, the selected image can be used as the first frame, and the user can use the "+" control to import the second frame, third frame, etc. (which can then be displayed as described above). Figure 47 (a) in the middle.

[0138] In some embodiments, if a camera movement template or reference motion video is imported into a storyboard generation component of a camera movement or motion imitation type, when the mouse focus moves over the camera movement template or reference motion video, it can be displayed as follows: Figure 49 As shown, the video content of the motion template or reference action video is enlarged and played, and a delete button represented by "-" can also be displayed so that users can view the video content of the motion template or reference action video at any time. Users can also delete the imported motion template or reference action video by clicking the delete button. This embodiment of the disclosure does not limit this.

[0139] As mentioned above, when storyboard sketches or storyboard images are not selected in bulk, clicking the collapsed storyboard generation component will display the following by default: Figure 37 The storyboard generation component shown in the image-to-image mode allows users to switch generation modes by clicking "Image Generation" and "Video Generation" at the top of the component. For example, from... Figure 37 When the image-to-image mode is switched to the image-to-video mode, the default display can be as shown below. Figure 50 The first frame class shown in (a) can be added using the import control represented by "+", and the last frame can also be displayed by default. Figure 50 As shown in (b) above, for multi-frame classes, the first frame, second frame, etc., can be imported sequentially using the import control represented by "+". Additionally, it can be imported using... Figure 47 The video generation type switching control shown in (c) allows you to switch the video generation type, for example, you can switch from... Figure 50 (a) in the middle switches to, for example Figure 50 In (b), you can also switch to, for example... Figure 50As shown in (c) above, the camera movement class allows you to import at least one image and a camera movement template. Alternatively, you can switch to a different mode such as... Figure 50 The action imitation class shown in (d) allows you to import at least one image and a reference action video. It should be understood that when a user uses the above... Figure 51 When importing images or videos into the storyboard generation component shown, you can switch to the above-mentioned... Figures 47 to 48 The storyboard generation components shown are not limited to the embodiments disclosed herein.

[0140] As shown in the figure above, the storyboard generation component also includes a prompt editing area. This prompt editing area can be used to edit, for example, the prompts entered in the image-to-image mode, the prompts entered in the image-to-video mode, the camera movement requirements (i.e., camera movement prompts) that can be entered under the camera movement category, the local redrawing requirements entered under the local redrawing function, the image expansion requirements entered under the image expansion function, and the text content entered under the lip-syncing function. In order to generate prompts more conveniently and quickly, in some embodiments, the method may also include: when the user sets the image basis, automatically filling the prompt editing area with node identifiers associated with the set image basis, or, by entering special symbols in the prompt editing area and selecting the set image basis, filling in the node identifiers associated with the selected image basis, so that the obtained image prompts use node identifiers to associate with the image basis, thereby allowing the associated image basis to be called using node identifiers when generating images based on the image prompts.

[0141] The styles of node identifiers and special symbols can be customized according to the actual situation. For example, they can be combined with image names, highlights, different colors, borders, etc., and special symbols can be any symbols such as "@" or "#". This disclosure does not limit these aspects.

[0142] For example, when a storyboard or storyboard sketch is selected, it can be done as follows: Figure 35 As shown in (a), the node identifier "@original image" is automatically entered in the prompt word editing area, or it can be done as follows: Figure 51 As shown in (a), the node identifier "Reference 1" on a white background is entered in the prompt word editing area. After the user imports a reference image, character image, or style image, it can be displayed as follows: Figure 36 As shown in (a), the prompt text "@original image@character4@style" will be automatically entered in the prompt text editing area. Figure 1 The node identifier is "", or it can be like this: Figure 52 As described in (b), the node identifiers "Reference 1" and "Reference 2" with a green background and "Role 1" and "Role 2" with a blue background are automatically added; or, the user can also... Figure 51Entering the special symbol "@" as shown in (c) will display a selection list of pre-set image criteria. Users can select any image criterion from this list to obtain the node identifier for "@image criterion". Node identifiers can be copied and deleted. Deleting a node identifier does not delete the image criterion; however, deleting the image criterion deletes the corresponding node identifier. It should be understood that users can edit and refine the image prompts based on their actual image needs and the node identifiers of the filled image criteria. For example, the image prompts can be edited to something like... Figure 51 The image prompt shown in (d) is "Character 1 is standing with his left side facing right, ...". For example, the above... Figure 38 The text in (a) shows "Keep the original composition unchanged, ...".

[0143] In some embodiments, the prompt editing area can also employ a structured prompt input method, which is used to input structured prompts according to preset structured prompt information, such as the one described above. Figure 16 The prompt editing area, "Please enter the words you want the character to say," can be a structured prompt input method, allowing users to enter the corresponding text content based on the prompt information.

[0144] In some embodiments, the prompt editing area also provides intelligent prompting functions such as text optimization and reverse prompting, as described above. Figures 11 to 13 The prompt editing area shown in the image includes a text optimization function for expanding and optimizing the text of the entered prompts, and a reverse prompt function for deducing prompts based on the image input by the user.

[0145] As mentioned above, in existing technologies, users need to frequently switch between multiple independent applications or interfaces when performing different storyboard generation tasks, resulting in fragmented operation processes, high learning costs, and poor functional scalability. To solve these problems, this disclosure provides a storyboard generation component based on a unified input interface. Its core interaction lies in building a unified intelligent interaction interface as the entry point for scheduling various AI generation tasks. It can support multiple generation modes and functions such as image generation, video generation, and lip-syncing. On this basis, it innovatively achieves decoupling between modes and models: that is, in a single mode, it supports users to flexibly select multiple underlying AI models and adjust key parameters. Currently, there is no similar fully standardized structure in business that can adapt to multiple scenarios, modes, and models to create a platform-based generative super input box. Similar solutions are mostly single-point generation, and the input items are unstructured, which leads to more time being spent on adjusting the input items and optimizing the corresponding prompt words.

[0146] In this embodiment, the storyboard generation component is an innovative and unified user interaction component. It allows users to access various AI generation capabilities through simple mode switching within an input box. In different modes, users can select different underlying AI models, and the component adaptively calls the corresponding input items and parameters to ensure the accuracy of the generated parameters. Users can select a desired mode, and the storyboard generation component will determine the availability of the mode or function based on the currently selected material type. If unavailable, it will be grayed out or hidden. For example, if the user selects video A, the storyboard generation component can display video regeneration and lip-syncing functions, while image generation can be grayed out (or hidden). Different models in different modes correspond to different structured input items. These structured input items are divided into common parameters and model-customized parameters, facilitating unified calling and personalized adaptation. Multiple models can be accessed through an API, and the component adapts to changes in input items and parameters based on different models.

[0147] In this embodiment, the data source in the storyboard generation component can come from raw image parameters, asset management library, historical data, output results of artificial intelligence models, etc. The data source can be dynamically adjusted according to user behavior. For example, it is initially empty. After inputting a raw image, the raw image parameters can be displayed here. At the same time, the current image can be placed into the input box as a pad image through user operation. The switching between models can maintain the stability of the unified data source, take the intersection, and discard the different items. That is, when the user switches the model type, the intersection between the parameters and basis before the switch is retained as the parameters and basis of the model after the switch.

[0148] In this embodiment, the storyboard generation component can serve as the core entry point for global generation, and can also be displayed alongside selected images (under the canvas model), embedding raw image parameters and other information for user viewing and correction; it can be understood as a super input box. Within the canvas, it is displayed globally by default; if the user clicks on a canvas, the global input box collapses downwards; the storyboard generation component is displayed following the storyboard canvas grid. Multiple selections of similar materials generate images: Selected images are batch-padded with reference or original images; the user inputs a unified item according to the model, generating multiple corresponding records. Selecting 2 images generates a video: The first and last frames are padded; the user inputs a unified item according to the model, generating 1 record. Selecting ≥3 images generates a video: The first frame is batch-padded; the user inputs a unified item according to the model, generating multiple corresponding records. When the user's focus is not on the storyboard generation component, the component can be collapsed to reduce obstruction and impact on the user's current operation. For example, when the user zooms the canvas to browse, the storyboard generation component defaults to a collapsed state; if the user selects multiple images, the storyboard generation component defaults to an expanded state for easy editing.

[0149] In this embodiment, the storyboard generation component adopts a more flexible and comprehensive input box hierarchy, flexibly integrating various generative models and adapting to existing components during integration, reducing development and user learning costs. Structurally, the clear hierarchy allows the storyboard generation component to adapt to multiple scenarios, modes, and models, becoming a platform-based generative super input box. From an interactive experience perspective, users can switch between different models on the same page, greatly avoiding the frequent downloading and uploading work required for reusing materials across multiple platforms. Moreover, users do not need to master the use of multiple independent software programs; they can explore and compare the generation effects of different models within a single interface, achieving simplified interaction and a leap in efficiency. Through pattern and structured input guidance, users are helped to express their generation intentions more clearly and accurately, thereby reducing the randomness of the generation results and improving the "hit rate." As a standardized "socket," the storyboard generation component can easily integrate new AI models and capabilities in the future.

[0150] In this embodiment, the storyboard generation component is an innovative, unified interactive component that serves as the core entry point for all AI generation tasks. It can intelligently support multiple generation modes, including raw images, raw videos, and lip-syncing, and supports multiple model selection and parameter adjustment in each mode. This reduces the learning cost for users, making it simple and intuitive to explore and compare the effects of different models, and providing users with extremely flexible and powerful generation control. The "super input box," as a standardized structure, can integrate new AI models and capabilities in the future, reducing the cost of future product feature expansion.

[0151] In this embodiment, the storyboard generation component acts as a powerful command center, allowing users to invoke various AI generation capabilities (such as raw images, raw videos, lip-syncing, etc.) within a single input box through simple mode switching, thus avoiding the cumbersome switching between different function pages. The interface elements of the storyboard generation component (such as parameter controls, upload area, and prompt examples) dynamically change according to the user's currently selected generation mode. The storyboard generation component can support processing and generating different types of content (i.e., "modals"), such as generating images from images (image-to-image), generating videos from images (image-to-video), and generating lip-syncing animations from audio and video. The core value of the "super input box" lies in its ability to serve as a front-end interface, carrying and scheduling these different generation modals.

[0152] Based on the above-described intelligent storyboard creation method, this disclosure also provides an intelligent storyboard creation device, comprising:

[0153] An import module is used to display at least one imported storyboard sketch in the storyboard canvas of the storyboard creation interface in response to a user importing at least one storyboard sketch into the storyboard creation interface.

[0154] The component module is used to respond to the user selecting one or multiple storyboard sketches from the displayed storyboard sketches, and to display the storyboard generation component corresponding to the storyboard sketches. The storyboard generation component is used to set the generation mode and corresponding generation parameters for the storyboard sketches. The generation mode is used to determine the type of storyboard generated based on the storyboard sketches, and the generation parameters are used to determine the effect of the storyboard generated based on the storyboard sketches.

[0155] The generation module is used to respond to the user's corresponding setting operation through the storyboard generation component, and to generate the storyboard draft corresponding to each selected storyboard sketch according to the user's set generation mode and corresponding generation parameters using artificial intelligence technology and display it in the storyboard canvas. The storyboard draft corresponding to any selected storyboard sketch is displayed on one side of the selected storyboard sketch. The storyboard draft is a storyboard image or a storyboard video.

[0156] In one possible implementation, the generation mode is further used for:

[0157] In response to a user selecting one or more storyboards from the displayed storyboards, a storyboard generation component corresponding to the selected storyboard is displayed, so as to generate a new storyboard based on the selected storyboard using the storyboard generation component.

[0158] The storyboard generation component includes a follow mode and a global mode. When multiple storyboard sketches or multiple storyboard images are selected in batches, the storyboard generation component is in global mode. The storyboard generation component in global mode includes settings controls for generation modes and corresponding generation parameters. The generation modes include image-to-image mode and image-to-video mode. The image-to-image mode is used to generate images based on images, and the image-to-video mode is used to generate videos based on images.

[0159] When a storyboard sketch or a storyboard image is selected, the storyboard generation component is in follow mode. The storyboard generation component in follow mode includes a setting control for the generation mode and corresponding generation parameters, as well as a setting control for at least one image editing function. The image editing function is used to generate a new storyboard by editing the selected storyboard sketch or storyboard image.

[0160] When a storyboard video is selected, the storyboard generation component is in follow mode and the storyboard generation component in follow mode includes settings controls for at least one video editing function, which is used to generate a new storyboard video by editing the selected storyboard video.

[0161] In one possible implementation, when a storyboard sketch or a storyboard image is selected, the storyboard generation component in follow mode includes at least one of the following image editing functions: partial redrawing, image expansion, multi-angle generation, and image high-definition; wherein, the partial redrawing function is used to partially redraw the image, the image expansion function is used to expand the image content, the multi-angle generation function is used to generate images from multiple perspectives, and the image high-definition function is used to optimize the image for high definition; when a storyboard video is selected, the storyboard generation component in follow mode includes at least one of the following video editing functions: lip-syncing and video high-definition; wherein, the lip-syncing function is used to adjust the synchronization between the character's lip movements and the dubbing audio in the storyboard video, the dubbing audio including dubbing audio generated based on user-set text content and selected timbre, or uploaded local dubbing audio; the video high-definition function is used to optimize the video for high definition.

[0162] In one possible implementation, the generation mode includes a graph-based image generation mode, and the generation parameters include model customization parameters and common parameters. The common parameters in the graph-based image generation mode include: image generation prompts and image generation criteria. The image generation criteria include at least one of the following: original images, reference images, style images, character images, and scene images required for generating storyboards. The image generation prompts are used to describe the generation requirements of storyboards based on the set image generation criteria. The original image includes the selected storyboard sketch or storyboard. The model customization parameters in the graph-based image generation mode include: image generation type, the model type of the image generation model that can be selected under different image generation types, and the corresponding image generation parameters. The image generation parameters include: the aspect ratio, number, and clarity of the storyboard to be generated. The image generation type includes direct image generation or style transfer image generation. The style transfer image generation type is used to generate corresponding storyboards according to common parameters while keeping the composition of the original image unchanged. The direct image generation type is used to generate corresponding storyboards according to common parameters without restricting the composition of the original image. The original image is the selected storyboard sketch or storyboard.

[0163] In one possible implementation, the storyboard generation component includes: a reference image setting control, a character image setting control, a scene image setting control, and a style image setting control; wherein, the reference image setting control is used to select at least one reference image from historically imported reference images, and is also used to import or delete reference images, with the selected reference image displayed on one side of the reference image setting control; wherein, the character image setting control is used to select character images of one or more characters from a preset character library, with different characters having character images from one or more angles, and is also used to import character images of any character, delete or add character images of existing characters, and edit character names; the selected character images are displayed on one side of the character image setting control; wherein, the scene image setting control is used to select one or more scene images from a preset scene library, and is also used to import or delete scene images; the selected scene images are displayed on one side of the scene setting control; wherein, the style image setting control is used to select one or more style images from a preset style library, and is also used to import or delete style images; the selected style images are displayed on one side of the scene setting control.

[0164] In one possible implementation, the component module is further configured to: display the reference weight of any one of the set reference image, style image, and scene image, as well as a weight adjustment control, when the user has set any one of the reference image, style image, and scene image, so as to adjust the reference weight through the weight adjustment control. The reference weight is used to determine the proportion of any one of the reference image, style image, and scene image referenced when generating storyboards using the raw image model.

[0165] In one possible implementation, the device further includes a cropping module, configured to: display a corresponding magnified image in response to the focus moving to a reference image, character image, scene image, or style image displayed in the storyboard generation component, the magnified image containing an image cropping button; display a cropping page for the magnified image in response to the user triggering the image cropping button, the cropping page displaying a cropping component according to a specified aspect ratio or a free aspect ratio to crop the magnified image; and update the image corresponding to the magnified image in the storyboard generation component to the cropped image in response to the cropping page completing the cropping operation for the magnified image and exiting the cropping page.

[0166] In one possible implementation, the storyboard generation component further includes a prompt word editing area. The component module is also used to: automatically fill the prompt word editing area with node identifiers associated with the set raw image basis when the user sets the raw image basis; or, by entering a special symbol in the prompt word editing area and selecting the set raw image basis, fill in the node identifiers associated with the selected raw image basis, so that the obtained raw image prompt word uses node identifiers to associate with the raw image basis.

[0167] In one possible implementation, the generation mode includes an image-to-video mode. The generation parameters in this mode include: video generation type, the model type of the video generation model selectable under different video generation types, the corresponding video generation parameters, the video generation criteria to be set for different video generation types, and video generation prompts. The video generation type includes at least one of the following: first and last frame type, multi-frame type, camera movement type, and motion imitation type. The first and last frame type is used to generate a storyboard video by setting a first frame image and / or a last frame image; the multi-frame type is used to generate a storyboard video by setting multiple frames; the camera movement type is used to generate a storyboard video based on images that meets the camera movement template and / or camera movement requirements set by the user; the motion imitation type is used to generate a storyboard video based on images that meets the reference motion video set by the user. The video generation parameters include: the duration and resolution of the storyboard video to be generated. The video generation criteria include: the selected storyboard sketch or storyboard image, or, further, at least one of: camera movement template, camera movement requirements, and reference motion video. The video generation prompts are used to describe the generation requirements of the storyboard video based on the set video generation criteria.

[0168] In one possible implementation, the storyboard creation interface further includes a view switching control for switching between a creation view, a static storyboard view, and a dynamic storyboard view. The storyboard creation interface defaults to the creation view, which is used to generate storyboard drafts. The static storyboard view displays marked storyboard images, and the dynamic storyboard view displays marked storyboard videos. The device also includes:

[0169] The marking module is used to: display a marking button on the storyboard in response to the focus moving to the storyboard or the operation of selecting the storyboard; or, display a marking button on the storyboard video and display video playback controls while automatically playing the storyboard video in response to the focus moving to the storyboard video or the operation of selecting the storyboard video.

[0170] The static display module is used to: respond to the user marking storyboards with the marking button and switching to the static storyboard view, display the marked storyboards in the storyboard canvas under the static storyboard view according to the order of the storyboard sketches. Here, one storyboard sketch corresponds to one storyboard grid. When multiple storyboards are marked under a certain storyboard sketch, each storyboard is displayed in a carousel or page-turning format in the storyboard grid corresponding to the storyboard sketch. When no storyboards are marked under a certain storyboard sketch, a default unmarked prompt is displayed in the storyboard grid corresponding to the storyboard sketch.

[0171] The dynamic display module is used to: respond to the user marking storyboard videos with the marking button and switching to the dynamic storyboard view; display the marked storyboard videos in the storyboard canvas under the dynamic storyboard view according to the arrangement order of the storyboard sketches; wherein, one storyboard sketch corresponds to one storyboard video cell; when multiple storyboard videos are marked under a certain storyboard sketch, the storyboard video cells corresponding to that storyboard sketch are displayed in a carousel or page-turning format; when no storyboard videos are marked under a certain storyboard sketch, the storyboard video cells corresponding to that storyboard sketch are displayed with a default unmarked prompt.

[0172] In one possible implementation, the static display module is further configured to: when the user performs a jump operation on any storyboard frame in the static storyboard view and the storyboard frame displays a storyboard image, jump back to the creation view and select the corresponding storyboard image; and when the user performs a jump operation on any storyboard frame in the static storyboard view and the storyboard frame does not display a storyboard image, jump back to the creation view and select the corresponding storyboard sketch.

[0173] The dynamic display module is further configured to: when the user performs a jump operation on any storyboard video cell in the dynamic storyboard view and the storyboard video cell displays a storyboard video, jump back to the creation view and select the corresponding storyboard video; and when the user performs a jump operation on any storyboard video cell in the dynamic storyboard view and the storyboard video cell does not display a storyboard video, jump back to the creation view and select the corresponding storyboard sketch.

[0174] In one possible implementation, the storyboard creation interface further includes an editor entry point, and the device further includes a storyboard jump module, used to: in response to a user jumping to the editor's editing interface through the editor entry point, import the marked storyboard diagram and storyboard video as storyboard materials into the editor, so as to use the storyboard materials for video editing in the editing interface.

[0175] In one possible implementation, the storyboard creation interface further includes: a character library management component, a scene library management component, a style library management component, an inspiration library management component, a download component, and an image upload component; the character library management component is used to store and manage preset character images, wherein the same character has at least one character image from a different perspective; the scene library management component is used to store and manage scene images; the style library management component is used to store and manage preset style images; the inspiration library management component is used to store and manage preset storyboard sketches; the download component is used to download the storyboard sketches and storyboard drafts on the storyboard canvas to the local machine in the form of a storyboard table; and the image upload component is used to upload storyboard sketches from the local machine.

[0176] In one possible implementation, the component module is further configured to: display the storyboard generation component in a collapsed global mode by default when no storyboard sketch or storyboard draft is selected; switch the storyboard generation component in a collapsed global mode to an expanded global mode when the user clicks on the storyboard generation component in a collapsed global mode, or when the user selects multiple storyboard sketches or multiple storyboard drafts in batches; and switch the storyboard generation component in a follow mode when the user clicks on any storyboard sketch or storyboard draft.

[0177] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0178] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0179] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0180] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0181] Figure 52 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a terminal device. (Refer to...) Figure 52 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0182] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0183] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0184] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0185] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0186] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0187] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0188] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0189] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0190] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0191] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An intelligent storyboard creation method, characterized in that, include: In response to a user importing at least one storyboard sketch into the storyboard creation interface, the imported at least one storyboard sketch is displayed in the storyboard canvas of the storyboard creation interface. In response to a user selecting one or more storyboard sketches from the displayed storyboard sketches, a storyboard generation component corresponding to the storyboard sketch is displayed. The storyboard generation component is used to set the generation mode and corresponding generation parameters for the storyboard sketch. The generation mode is used to determine the type of storyboard generated based on the storyboard sketch, and the generation parameters are used to determine the effect of the storyboard generated based on the storyboard sketch. In response to the user completing the corresponding settings operation through the storyboard generation component, artificial intelligence technology is used to generate the storyboard corresponding to each selected storyboard sketch according to the user-set generation mode and corresponding generation parameters, and the storyboard is displayed in the storyboard canvas. The storyboard corresponding to any selected storyboard sketch is displayed on one side of the selected storyboard sketch. The storyboard is a storyboard or a storyboard video. The storyboard generation component includes a follow mode and a global mode. When multiple storyboard sketches are selected in batches, the storyboard generation component is in global mode. The storyboard generation component in global mode includes settings controls for the generation mode and corresponding generation parameters. When a storyboard sketch is selected, the storyboard generation component is in follow mode. The storyboard generation component in follow mode includes a setting control for the generation mode and corresponding generation parameters, as well as a setting control for at least one image editing function. The image editing function is used to generate a new storyboard by editing the selected storyboard sketch.

2. The method according to claim 1, characterized in that, The method further includes: In response to a user selecting one or more storyboards from the displayed storyboards, a storyboard generation component corresponding to the selected storyboard is displayed, so as to generate a new storyboard based on the selected storyboard using the storyboard generation component. When multiple storyboard images are selected in batches, the storyboard generation component is in global mode; the generation mode includes image-to-image mode and image-to-video mode, the image-to-image mode is used to generate images based on images, and the image-to-video mode is used to generate videos based on images; When a storyboard image is selected, the storyboard generation component is in follow mode. The storyboard generation component in follow mode includes a setting control for the generation mode and corresponding generation parameters, as well as a setting control for at least one image editing function. The image editing function is also used to generate a new storyboard by editing the selected storyboard image. When a storyboard video is selected, the storyboard generation component is in follow mode and the storyboard generation component in follow mode includes settings controls for at least one video editing function, which is used to generate a new storyboard video by editing the selected storyboard video.

3. The method according to claim 2, characterized in that, When a storyboard sketch or a storyboard image is selected, the storyboard generation component in follow mode includes at least one of the following image editing functions: partial redrawing, image expansion, multi-angle generation, and image high-definition; wherein, the partial redrawing function is used to redraw a part of the image, the image expansion function is used to expand the image content, the multi-angle generation function is used to generate images from multiple perspectives, and the image high-definition function is used to optimize the image for high definition; When a storyboard video is selected, the storyboard generation component in follow mode includes at least one of the following video editing functions: lip-syncing function and high-definition video function; wherein, the lip-syncing function is used to adjust the synchronization between the characters' lip movements and the dubbing audio in the storyboard video, and the dubbing audio includes dubbing audio generated based on the text content set by the user and the selected timbre, or uploaded local dubbing audio; the high-definition video function is used to optimize the video in high definition.

4. The method according to claim 2, characterized in that, The generation mode includes graph-to-graph mode, and the generation parameters include model-customized parameters and common parameters; The common parameters in the raw image mode include: raw image prompts and raw image basis. The raw image basis includes at least one of the following: original images, reference images, style images, character images, and scene images that need to be referenced for generating storyboard images. The raw image prompts are used to describe the generation requirements of storyboard images based on the set raw image basis. The original images include the selected storyboard sketches or storyboard images. The model customization parameters in the raw image mode include: raw image type, the model type of raw image model that can be selected under different raw image types, and the corresponding raw image parameters; the raw image parameters include: aspect ratio, number, and resolution of the storyboard images to be generated; The types of generated images include direct generated images and style-transfer images. The style-transfer images are used to generate corresponding storyboard images according to common parameters while keeping the composition of the original image unchanged. The direct generated images are used to generate corresponding storyboard images according to common parameters without restricting the composition of the original image. The original image is the selected storyboard sketch or storyboard image.

5. The method according to claim 4, characterized in that, The storyboard generation component includes: reference image setting control, character image setting control, scene image setting control, and style image setting control; The reference image setting control is used to select at least one reference image from historically imported reference images, and is also used to import or delete reference images. The selected reference image is displayed on one side of the reference image setting control. The character image setting control is used to select one or more character images from a preset character library. Different characters have character images from one or more angles. It is also used to import character images of any character, delete or add character images of existing characters, and edit character names. The selected character images are displayed on one side of the character image setting control. The scene image setting control is used to select one or more scene images from a preset scene library, and is also used to import or delete scene images; the selected scene image is displayed on one side of the scene setting control. The style map setting control is used to select one or more style maps from a preset style library, and also to import or delete style maps; the selected style map is displayed on one side of the scene setting control.

6. The method according to claim 4 or 5, characterized in that, The method further includes: If the user has set any one of the reference image, style image, and scene image, the reference weight of any one of the set reference image, style image, and scene image, as well as the weight adjustment control, is displayed so that the reference weight can be adjusted through the weight adjustment control. The reference weight is used to determine the proportion of any one of the reference image, style image, and scene image referenced when generating storyboards using the raw image model.

7. The method according to claim 4 or 5, characterized in that, The method further includes: When the focus moves to a reference image, character image, scene image, or style image displayed in the storyboard generation component, a corresponding magnified image is displayed, which includes an image cropping button. In response to the user triggering the image cropping button, a cropping page for the enlarged image is displayed. The cropping page displays cropping components according to a specified aspect ratio or a free aspect ratio to crop the enlarged image. In response to the cropping page completing the cropping operation on the enlarged image and exiting the cropping page, the image corresponding to the enlarged image in the storyboard generation component is updated to the cropped image.

8. The method according to claim 4, characterized in that, The storyboard generation component also includes a prompt word editing area, and the method further includes: When the user sets the basis for the image, the node identifier associated with the set basis for the image is automatically filled in the prompt word editing area. Alternatively, by entering a special symbol in the prompt word editing area and selecting the set basis for the image, the node identifier associated with the selected basis for the image is filled in, so that the resulting prompt word for the image uses the node identifier to associate with the basis for the image.

9. The method according to claim 1 or 4, characterized in that, The generation mode includes a graph-generated video mode. The generation parameters in the graph-generated video mode include: video generation type, the model type of the video generation model that can be selected under different video generation types and the corresponding generated video parameters, the generated video basis and generated video prompt words that need to be set for different video generation types; The video generation type includes at least one of the following: first and last frame type, multi-frame type, camera movement type, and action imitation type; the first and last frame type is used to generate a storyboard video by setting a first frame image and / or a last frame image; the multi-frame type is used to generate a storyboard video by setting multiple frame images; the camera movement type is used to generate a storyboard video based on images that meets the camera movement template and / or camera movement requirements set by the user; the action imitation type is used to generate a storyboard video based on images that meets the reference action video set by the user. The generated video parameters include: the duration and resolution of the storyboard video to be generated; the generated video criteria include: the selected storyboard sketch or storyboard diagram, or, further include: camera movement template, camera movement requirements, and reference action video, at least one of the following; the generated video prompts are used to describe the generation requirements of the storyboard video based on the set generated video criteria.

10. The method according to claim 1, characterized in that, The storyboard creation interface also includes a view switching control, which is used to switch between the creation view, the static storyboard view, and the dynamic storyboard view. The storyboard creation interface is in the creation view by default. The creation view is used to generate storyboard drafts. The static storyboard view is used to display the marked storyboard images. The dynamic storyboard view is used to display the marked storyboard videos. The method also includes: In response to the focus moving to the storyboard or the operation of selecting the storyboard, a marker button is displayed on the storyboard; or, in response to the focus moving to the storyboard video or the operation of selecting the storyboard video, a marker button is displayed on the storyboard video and a video playback control is displayed while the storyboard video is played automatically. In response to the user marking storyboards with the mark button and switching to the static storyboard view, the marked storyboards are displayed in the storyboard canvas under the static storyboard view in the order of the storyboard sketches. One storyboard sketch corresponds to one storyboard grid. When multiple storyboards are marked under a storyboard sketch, the storyboards are displayed in a carousel or page-turning format in the storyboard grid corresponding to the storyboard sketch. When no storyboards are marked under a storyboard sketch, a default unmarked prompt is displayed in the storyboard grid corresponding to the storyboard sketch. In response to the user marking storyboard videos with the mark button and switching to the dynamic storyboard view, the marked storyboard videos are displayed in the storyboard canvas under the dynamic storyboard view in the order of the storyboard sketches. One storyboard sketch corresponds to one storyboard video cell. When multiple storyboard videos are marked under a certain storyboard sketch, the storyboard video cells corresponding to that storyboard sketch are displayed in a carousel or page-turning format. When no storyboard videos are marked under a certain storyboard sketch, the storyboard video cells corresponding to that storyboard sketch display a default unmarked prompt.

11. The method according to claim 10, characterized in that, The method further includes: If the user performs a jump operation on any storyboard grid in the static storyboard view and the storyboard grid displays a storyboard image, the user is redirected back to the creation view and the corresponding storyboard image is selected. If the user performs a jump operation on any storyboard frame in the static storyboard view and no storyboard is displayed in that storyboard frame, the user is redirected back to the creation view and the corresponding storyboard sketch is selected. If the user performs a jump operation on any storyboard video cell in the dynamic storyboard view and the storyboard video cell displays a storyboard video, the user is redirected back to the creation view and the corresponding storyboard video is selected. If the user jumps to any storyboard video frame in the dynamic storyboard view and no storyboard video is displayed in that frame, the user is redirected back to the creation view and the corresponding storyboard sketch is selected.

12. The method according to claim 10 or 11, characterized in that, The storyboard creation interface also includes an editor entry point, and the method further includes: In response to the user jumping to the editing interface of the editor through the editor entry, the marked storyboard and storyboard video are imported into the editor as storyboard materials, so that the video can be edited using the storyboard materials in the editing interface.

13. The method according to claim 1, characterized in that, The storyboard creation interface also includes: a character library management component, a scene library management component, a style library management component, an inspiration library management component, a download component, and an image upload component; The character library management component is used to store and manage preset character images, wherein the same character has at least one character image from a different perspective; the scene library management component is used to store and manage scene images; the style library management component is used to store and manage preset style images; and the inspiration library management component is used to store and manage preset storyboard sketches. The download component is used to download the storyboard sketches and storyboard drafts on the storyboard canvas to the local machine in the form of a storyboard table, and the image transfer component is used to upload the storyboard sketches from the local machine.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 13.

15. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

16. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images