Video generation method and apparatus, electronic device, storage medium, and program product

Abstract

The method comprises: obtaining video description information and a reference image; obtaining scene environment parameters corresponding to a target video and a shot scheme of at least one shot based on the video description information and the reference image; constructing a target prompt word corresponding to the shot based on the video description information, the reference image, the scene environment parameters and the shot scheme of the shot, and generating a shot video based on the target prompt word by using a video generation model; and splicing the at least one shot video to obtain the target video. According to the technical scheme, the environment parameter extraction and the shot planning are automatically completed based on the multi-modal input of the text and the image, the prompt word is automatically constructed, and the shot video is generated, so that the creation threshold and the artificial cost are reduced, and the consistency of the overall video style, the vision and the environment is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of multimodal generation technology, script generation, storyboard breakdown, and image / video generation have become important methods of content creation.

[0003] In related technologies, the common approach to video generation is to input text prompts for each shot, with video shot segmentation and camera planning entirely dependent on manual work by the user. This type of video generation method suffers from technical problems such as easy style drift across multiple shots, limited precision in text style constraints, and the lack of a unified and fixed benchmark for the global environment. Furthermore, it requires a high level of expertise and has certain professional limitations, making it difficult for non-professional users to create coherent and stylistically consistent complete video content. Summary of the Invention

[0004] This disclosure provides a video generation method, apparatus, electronic device, storage medium, and program product to solve the problems of video generation relying on manual storyboard planning, requiring manual input of prompts for each shot, low efficiency, easy style drift in multiple shots, lack of a unified and fixed benchmark for the global environment, and high professional threshold for creation.

[0005] According to one aspect of this disclosure, a video generation method is provided, the method comprising: Obtain video description information and reference images. The video description information includes style description information and content description information of the target video to be generated. The reference images are used to define the visual information presented by the target video. Based on the video description information and the reference image, obtain the scene environment parameters corresponding to the target video and at least one storyboard scheme. For each of the aforementioned storyboards, based on the video description information, the reference image, the scene environment parameters, and the storyboard scheme, target prompt words corresponding to the storyboard are constructed, and a video generation model is used to generate storyboard videos based on the target prompt words; At least one of the storyboard videos is stitched together to obtain the target video.

[0006] According to another aspect of this disclosure, a video generation apparatus is provided, the apparatus comprising: The first module is used to acquire video description information and reference images. The video description information includes style description information and content description information of the target video to be generated, and the reference images are used to limit the visual information presented by the target video. The second module is used to obtain scene environment parameters and at least one storyboard scheme corresponding to the target video based on the video description information and the reference image. The third module is used to construct target prompt words corresponding to each of the storyboards based on the video description information, the reference image, the scene environment parameters and the storyboard scheme, and to generate storyboard videos based on the target prompt words using a video generation model. The fourth module is used to stitch together at least one of the storyboard videos to obtain the target video.

[0007] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement a video generation method as described in any of the embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement any of the video generation methods described in the embodiments of this disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, which, when executed by a processor, implements any of the video generation methods described in the embodiments of the present disclosure.

[0010] The technical solution of this disclosure, by acquiring unified video description information and reference images as creative input, automatically derives and generates global scene environment parameters and storyboard schemes for the target video. This transforms fragmented creative needs into standardized global environmental benchmarks and structured shot planning, replacing the traditional manual storyboard design mode. It provides a unified execution basis for multi-shot video generation, effectively lowering the professional threshold for video creation. Furthermore, by constructing exclusive target prompts for each storyboard based on video description information, reference images, scene environment parameters, and storyboard schemes, it can layer and integrate style constraints, visual constraints, environmental constraints, and shot content, solidifying a shared style tone, visual characteristics, and environmental attribute benchmark for the entire film. This effectively avoids style drift and inconsistent environmental representation that occur during the independent generation of multiple shots, ensuring the coherence and unity of visual expression across shots. Furthermore, by employing a video generation model to generate segmented videos one by one based on the target prompts corresponding to each shot, and then integrating them to obtain the complete target video, the entire chain of automated generation from creative requirement input to complete video output is achieved. This eliminates the need for users to manually write input prompts for each shot, significantly simplifying the creation process of multi-shot videos. Simultaneously, it ensures that each shot not only meets the single-shot narrative design requirements but also strictly adheres to a globally unified creative standard. The technical solution of this disclosure addresses the problems in related technologies where video generation relies on manual shot planning, requires manual input of prompts for each shot (low efficiency), suffers from easy style drift in multi-shot videos, lacks a unified and fixed global environment standard, and has a high professional threshold for creation. It can automatically complete global environment settings and shot planning based on user input, and ensures consistency in style and environment across multiple shots through layered constraint on target prompts. This achieves automated and standardized generation of complete videos, effectively reducing the operational costs and professional limitations of video creation, improving the stylistic uniformity and narrative coherence of multi-shot generated videos, and ultimately enhancing the overall efficiency and quality of intelligent video creation.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a video generation method provided in an embodiment of this disclosure; Figure 2A schematic diagram of a video editing interface provided in an embodiment of this disclosure; Figure 3 A flowchart illustrating another video generation method provided in this embodiment of the disclosure; Figure 4 A flowchart illustrating another video generation method provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0018] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0019] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this disclosure. This embodiment is applicable to scenarios where a target video is generated based on text input. The method can be executed by a video generation device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 1 As shown, the method in this embodiment includes: S110. Obtain video description information and reference images.

[0022] The video description information can be understood as user-inputted instructions that characterize the user's needs for creating the target video. The video description information can be a description of the user's overall creative requirements for the target video, serving as the basis for the system to execute subsequent video generation processes. The video description information can be used to define the video style and / or content of the target video to be generated. Optionally, the information type of the video description information can include at least one of the following: text, voice, options, and tags. In some cases, in the provided solution, the video description information may include style description information and content description information of the target video to be generated.

[0023] Style description information can be understood as attribute-type information in video description information used to define the style characteristics of the target video, serving as the foundation for overall style control. Style description information can clarify the artistic style, visual texture, and visual tone of the target video, limiting only the expressive form of the video visuals and not involving specific narrative content such as characters, plot, or scenes. Style description information can indicate the visual style of the target video to be generated. Style description information can define the global style benchmark of the target video, ensuring that the visual style of the target video does not deviate directionally. Optionally, the information type of style description information can include at least one of the following: tags, options, text, and voice. For example, tag-type style description information could be the two style tags "Chinese style" and "ink wash" selected by the user from the preset style library; option-type style description information could be the "realistic" option selected by the user from multiple style options provided by the system, such as "realistic," "cartoon," and "cyberpunk"; text-type style description information could be the text entered by the user in natural language, such as "adopting an elegant and freehand ink wash texture, with an overall cool tone and an atmosphere of Eastern classical aesthetics."

[0024] Content description information can be understood as entity-based information used to carry the narrative subject, scene events, and specific visual elements of the target video. It serves as the core material basis for subsequent storyboard design and visual content generation. Content description information can be used to clarify the specific narrative content of the target video, encompassing entity elements such as the main character, scene space, actions, plot development, and shot scale. Content description information can also be used to indicate the visual content to be presented in the target video. Optionally, the information type of content description information can include at least one of the following: text, voice, tags, and options. For example, text-based content description information could be "A woman in Hanfu is dancing in a bamboo forest" entered by the user using natural language; tag-based content description information could be three sets of content tags selected by the user from a preset content library: "Hanfu woman," "dense bamboo forest," and "dance actions"; option-based content description information could be the user selecting the "bamboo forest" scene from the system's scene dropdown and the "ancient style dancer" subject from the character dropdown.

[0025] Reference images can be understood as image-based reference materials used to concretize and constrain the visual presentation effect of the target video. Reference images can be used to limit the visual information presented in the target video. Reference images can be visual pixel-based frames used as a reference benchmark for style and content, defining the image quality, detail, and element forms of the target video. Optionally, there can be one or more reference images. In one scenario, in the provided solution, reference images can be used to define at least one of the following aspects of the target video: main character, scene, color tone, props, special effects, and key visual elements.

[0026] The main character can be a visual element of a person or anthropomorphic subject that carries the core narrative function in the target video. Reference images corresponding to the main character can be used to define the visual attributes of that main character, such as appearance, styling details, clothing, and body proportions, ensuring consistency in the character's image across different shots and avoiding any shifts in appearance, clothing, or styling. For example, a user can upload a reference image of a woman in Hanfu (traditional Han clothing) to define the facial features, hairstyle, Hanfu style, accessory details, and body proportions of the dancing woman in the target video, ensuring that the character in each shot has the same visual image.

[0027] In this context, a scene can refer to the spatial environment and background elements within the target video that carry narrative events and provide space for character activities. A scene can also serve as the spatial carrier for character actions and plot development. Reference images corresponding to the scene can be used to define the scene's spatial structure, composition, material texture, environmental layers, and other visual characteristics, ensuring consistency in the spatial logic and visual presentation of the entire scene environment. For example, a user can upload a reference image of a dense bamboo forest scene to define the density of the bamboo forest, the height of bamboo nodes, ground vegetation, fog distribution, and other environmental characteristics in the target video, avoiding significant differences in the spatial form of the bamboo forest in different shots.

[0028] Hue can refer to the overall color scheme, color tendency, and lighting tone of the target video. Hue determines the overall color atmosphere and visual experience of the image. Reference images corresponding to the hue can be used to limit the proportion of primary and secondary colors, color saturation, contrast, color temperature, and other color attributes of the image, serving as a core constraint to ensure color unity throughout the target video. For example, a user can upload a hue reference image to limit the overall color scheme of the target video to dark green as the primary color, off-white as the secondary color, and light gold accents, as well as a cool hue tone with low saturation and soft contrast.

[0029] Props can be visual elements in the target video that assist in the narrative and complement the characters' actions. Props can also be physical objects that enrich the details of the scene and support the plot. Reference images corresponding to props can be used to define the visual characteristics of props, such as their shape, material texture, size proportions, and appearance details, ensuring that the same prop appears consistently in different shots. For example, a user can upload a reference image of an ancient-style folding fan to define the fan surface pattern, fan rib material, and size of the folding fan held by the dancer in the target video, avoiding discrepancies in the shape details of the folding fan in different shots.

[0030] Special effects can be non-physical dynamic visual elements used in the target video to enhance visual appeal. They can also enhance the atmosphere and artistic expression of the scene. Reference images corresponding to the effects can be used to define the visual characteristics of the effects, such as shape, color, intensity, and graininess, ensuring a consistent visual style throughout the target video for similar effects. For example, a user can upload a Tyndall effect reference image to define the shape, brightness, and halo texture of beams of light penetrating the leaves in a bamboo forest, preventing inconsistencies in the style and intensity of lighting effects across different shots.

[0031] Key visual elements can be iconic keyframe visual elements in the target video that carry core identification functions or core narrative scenes. Key visual elements can correspond to the landing frame or core promotional frame of the target video, and are the core images with the highest visual recognizability throughout the entire video. Reference images corresponding to key visual elements can be used to define the composition, element arrangement, visual focus, and overall texture of the key visual frame, ensuring high-fidelity reproduction of the core image. For example, a user can upload a reference image of a main visual poster depicting dancing in a bamboo forest to define the character positions, compositional proportions, visual focus, and overall visual effect of the ending shot of the target video, ensuring that the ending shot highly matches the core identification features of the reference image.

[0032] In one scenario, the provided solution may include at least one of the following methods for obtaining style description information: receiving a tag selection operation for an optional style tag and obtaining style description information based on the tag selection operation; or receiving a text editing operation for a text editing box and obtaining style description information.

[0033] In one scenario, the provided solution may include at least one of the following methods for obtaining content description information: receiving a tag selection operation for optional content tags and obtaining content description information based on the tag selection operation; or receiving a text editing operation for a text editing box and obtaining content description information.

[0034] The following is a detailed explanation of one method for obtaining video description information.

[0035] Optionally, obtaining video description information includes: displaying multiple optional style tags through the video editing interface, receiving the user's tag selection operation for the optional style tags, and obtaining style description information based on the selected optional style tags; displaying a text editing box through the video editing interface, receiving the user's text editing operation for the text editing box, and determining content description information; and obtaining video description information based on the style description information and content description information.

[0036] The video editing interface can be understood as a visual front-end operation platform that provides users with interactive video creation capabilities, serving as the core entry point for users to interact with the video generation system. The video editing interface can be a visual operation page integrating multiple video creation function modules, which may include at least one of the following: style selection, content input, material upload, parameter configuration, and result preview. The video editing interface can be used to support the user's entire creative interaction process, converting user commands into system-recognizable requirement information, while simultaneously providing feedback on the generation progress and results to the user. For example, the operation page that a user enters after opening the video generation tool, which includes functional modules such as a style selection area, text input area, reference image upload area, video preview area, and parameter configuration area, is the video editing interface.

[0037] The optional style tags can be understood as pre-defined, standardized style options presented in a tagged format, serving as interactive units for users to quickly select the style of a video. Each optional style tag corresponds to a fixed artistic style system and constraints, possessing the characteristics of standardization and reusability. Users do not need to manually write style descriptions; they can quickly determine the style direction of the target video by clicking and selecting an optional style tag, effectively reducing the input threshold and ambiguity of style requirements. For example, in the style selection area of ​​the video editing interface, the clickable style option buttons such as "Chinese Style," "3D Rendering," "Realistic," "Cartoon," "Stylized," "Cyberpunk," "Minimalist," "Oil Painting," "Ink Painting," and "Thick Paint" are optional style tags. Different optional style tags correspond to different style types.

[0038] The tag selection operation can be understood as a user's confirmation interaction with available style tags, a triggering action that allows the system to capture the user's style selection intent. Tag selection can refer to the interactive process where a user selects a style tag from multiple available options that matches their creative needs. It supports selecting a single tag or a combination of multiple tags. The system identifies this operation to determine the range of styles selected by the user. Optionally, the tag selection operation can include at least one of the following: click operation, voice-triggered operation, touch selection operation, shortcut key-triggered operation, or drag-and-drop selection operation.

[0039] In one scenario, style description information can be mapped from user-selected optional style tags, corresponding to standardized style constraint rules, which serve as the basis for style control of the entire target video.

[0040] The text editing box can be understood as an interactive control in the video editing interface used to receive free text input from the user, and it is the core carrier for obtaining customized content requirements from the user. The text editing box can be a reserved editable text input area within the video editing interface, supporting users to enter, modify, and supplement custom text content. It is used to receive customized descriptions of narrative elements such as video content, plot, and actions from the user, adapting to the diverse content creation needs of users. For example, in the video editing interface, there is an input text area marked with the prompt "Please enter a description of the video content," where the user can freely enter text; this input area is the text editing box.

[0041] Text editing operations can be understood as interactive operations performed by users within a text editing box, such as text input and adjustment. This process triggers the system to obtain content requirements. Text editing operations can encompass a series of text interactions, including user input, modification of descriptions, addition of details, and deletion / adjustment. By recognizing the final text result of these operations, the system extracts the user's video content creation needs.

[0042] In one scenario, a video editing interface is displayed, showing multiple selectable style tags and text editing boxes, each style tag corresponding to a style type. Further, upon receiving a user's tag selection operation for any selectable style tag, the selected style tag can be determined, and its corresponding style type can be defined as style description information. Further, upon receiving a text editing operation for a text editing box, the edited text can be obtained from the text editing box and displayed there. Further, in response to receiving an editing completion command, the text displayed in the text editing box is defined as content description information. Further, the determined style description information and content description information can be used as video description information for the target video.

[0043] In one scenario, the provided solution may include at least one of the following methods for acquiring the reference image: obtaining a reference image based on an image upload operation in response to receiving an image upload operation; or obtaining a reference image based on an image capture operation in response to receiving an image capture operation. One such image acquisition method is described below.

[0044] Optionally, obtaining a reference image includes: displaying multiple slots through a video editing interface, the slots being used to hold images; receiving an image upload operation for the slots; and determining a reference image based on the uploaded image.

[0045] In this context, a "slot" can be understood as a functional area within the video editing interface specifically designed for placing image materials. A slot can be a designated image storage location within the video editing interface, and the number of slots can be one or more. Slots are used to standardize the uploading, previewing, and management of reference images. Different slots can correspond to different material types, which may include at least one of the following: characters, scenes, color tones, props, special effects, and key visuals. That is, the material type corresponding to a slot can be pre-configured, or pre-configured and user-customizable, or user-editable. When the material type corresponding to a slot is pre-configured, the user can upload images of the corresponding type to the corresponding slot as reference images. When the material type corresponding to a slot is user-customizable, after uploading an image to the slot, multiple optional material types corresponding to that slot are displayed, and the user's selection of an optional material type is received. The selected material type is then confirmed as the material type corresponding to that slot.

[0046] Image upload can be understood as an interactive operation where users store local or cloud-based image materials into designated slots. Image upload can be triggered by the system retrieving external image materials, and can support uploading single or multiple images sequentially to different slots. For example, a user clicking the upload button in a slot, selecting an image from their local folder, and confirming the import constitutes an image upload operation.

[0047] In one scenario, multiple slots are displayed through a video editing interface. Further, for each of the multiple slots, an input control for the image upload control displayed in that slot is triggered, displaying an image display interface that includes multiple candidate images. Further, upon receiving an image selection operation for any candidate image, the selected image is uploaded to that slot and displayed in that slot as a reference image.

[0048] S120. Based on video description information and reference images, obtain scene environment parameters corresponding to the target video and a storyboard scheme for at least one storyboard.

[0049] Scene environment parameters can be understood as a standardized set of parameters characterizing the global environmental attributes of the target video. Scene environment parameters can be a structured description of environmental elements such as lighting conditions, time period, environmental space, color scheme, and atmosphere tone that are globally unified within the target video. This serves as a shared environmental benchmark for all shots in the target video, ensuring environmental consistency across shots. In other words, scene environment parameters can be used to define the environmental attributes and overall atmosphere tone of the target video. Optionally, scene environment parameters can include at least one of the following: lighting, time period, environment, color scheme, and atmosphere. Lighting can be a parameter used to describe the lighting conditions of the scene. Lighting can be used to record information such as light angle, intensity, color temperature, and lighting effects, unifying the lighting performance throughout the target video and avoiding discrepancies in lighting between different shots. Time period can be a parameter used to define the time interval of the video frame. Time period can specify the specific day / night time period and approximate time range, combining with lighting to shape the temporal characteristics of the video frame. Environment can be a parameter used to describe the spatial elements and real-world features of the scene. Environment can record the scale of the scene space, types of objects, materials, terrain, vegetation, and other physical characteristics, determining the basic scene appearance of the target video. A color scheme can be a parameter used to define the color system of a visual. It defines the rules for matching primary, secondary, and accent colors, and can include color names and values ​​to unify the overall tone and style of the target video. Atmosphere can be a parameter used to define the overall mood and artistic tone of a visual. It can be a concise keyword description of the mood, emotion, and style conveyed by the target video, to unify the sensory atmosphere of the entire video.

[0050] For example, continuing with the previous example, suppose the content description information is: a woman dressed in Hanfu dances in a bamboo forest; the style description information is: traditional Chinese style, ink painting; the reference images are: 3 images, namely: a front view of the character, a bamboo forest scene image, and an overall color tone reference image. The scene environment parameters determined based on the video description information and reference images can include: lighting: soft side-backlighting, with light filtering through the gaps in the bamboo forest, creating a Tyndall effect; time of day: early morning (within 30 minutes after sunrise); environment: a large bamboo forest, with tall bamboo trees and a thin mist covering the ground; color scheme: mainly dark green, with off-white as a secondary color, and light gold as an embellishment; atmosphere: ethereal, tranquil, and poetic.

[0051] In one scenario, the provided solution may use a set of global video environment parameters encapsulated in a standardized data format, serving as a structured data carrier for unifying the visual environment of the entire target video shot. Optionally, the data format may include at least one of the following: JSON, XML, YAML, plain text, or tabular format.

[0052] In this context, a storyboard can be understood as an independent visual unit formed by breaking down a complete video narrative according to shot boundaries. It serves as the basic execution unit for video content planning and segmented production. From a narrative perspective, each storyboard corresponds to a complete, independent shot, carrying a specific narrative function. Through differentiated design of shot size, camera movement, duration, and visual content, they are combined to form a complete video narrative logic. From a production perspective, a storyboard is the smallest execution unit for generating video segments. The system generates corresponding video clips based on individual storyboards, which are then spliced ​​together to obtain the complete target video. From a functional perspective, storyboard splitting enables refined control over video rhythm and visual perspective, making the overall narrative more layered and expressive. Optionally, the shot size of a storyboard can include at least one of the following: close-up, medium shot, long shot, wide shot, and extreme wide shot.

[0053] In this context, the storyboard scheme can be understood as a structured set of solutions for breaking down and designing shots in a target video. The storyboard scheme can be a summary of design information for each shot—including shot size, camera movement, duration, and content—after the complete video narrative has been broken down into multiple independent shots. It serves as the direct execution basis for subsequently generating target video frames shot by shot. Optionally, the content in the storyboard scheme may include at least one of the following: storyboard number, shot size, camera movement, shot duration, on-screen cues, and camera movement description.

[0054] For example, continuing with the previous example, based on the video description information and reference images, four scenes can be determined, and the scene breakdown scheme for each scene is as follows: Shot 1, Shot type: wide shot; Camera movement: slow zoom; Shot duration: 4 seconds; Visual cues: Deep in the bamboo forest, a woman's profile stands in the center of an open space, surrounded by a thin mist. The figure's height is about 30% of the video frame, emphasizing the macroscopic feel of the environment; Camera movement description: The camera slowly zooms in on the figure from a distance at a constant speed, with the focus gradually tightening, highlighting the process of the figure emerging from the environment.

[0055] Shot 2, Shot type: wide shot; Camera movement: fixed; Shot duration: 5 seconds; Visual cues: The woman's arms are outstretched, her water sleeves are flowing, she faces the left side of the frame, the figure occupies 60% of the frame height, the bamboo forest background is blurred, the figure is on the right in the rule of thirds composition; Camera movement description: Fixed camera position, recording the entire dance movement, the water sleeves are naturally flowing in the wind.

[0056] Scene 3, Shot type: Close-up; Camera movement: Slow horizontal movement; Shot duration: 4 seconds; On-screen cues: Female upper body, facial expression focused and graceful, water sleeves flowing into the frame from the left, hair moving slightly with the movement, bamboo sections in the background blurred into a green halo; Camera movement description: The camera slowly moves horizontally from right to left, following the rhythm of the flowing water sleeves.

[0057] Shot 4, Shot type: Close-up; Camera movement: Fixed; Shot duration: 3 seconds; On-screen cues: Details of the woman's hands and flowing sleeves, elegant finger posture, clearly visible silk texture of the flowing sleeves, completely blurred background; Camera movement description: Fixed close-up, emphasizing the texture of the clothing and details of the hand movements (this is a mandatory rule: a clothing close-up shot is automatically included when the number of shots is greater than or equal to 4).

[0058] It should be noted that when determining the storyboard scheme for at least one scene based on video description information and reference images, semantic analysis of the video description information can be performed first to determine the number of scenes. Furthermore, once the number of scenes is determined, the storyboard scheme for each scene can be determined based on the video description information and reference images.

[0059] It should also be noted that after determining the scene environment parameters of the target video, the scene environment parameters can be written into the database. Then, when generating each storyboard video in the future, the scene environment parameters can be directly read from the database to construct the target prompt words corresponding to the corresponding storyboard.

[0060] In one scenario, the provided solution, based on video description information and reference images, obtains a storyboard scheme for scene environment parameters and at least one scene corresponding to the target video. This can include at least one of the following: using a semantic reasoning model to generate a storyboard scheme for scene environment parameters and at least one scene corresponding to the target video based on scene task prompts, video description information, and reference images; using an intelligent system to generate a storyboard scheme for scene environment parameters and at least one scene corresponding to the target video based on video description information and reference images; using a preset template matching engine to match a corresponding baseline storyboard template based on video description information and reference images, and after adapting and adjusting the environmental parameter items in the baseline storyboard template to the scene content, generating a storyboard scheme for scene environment parameters and at least one scene corresponding to the target video; or using an architecture combining a rule engine and a reasoning model, first parsing structured creative elements from video description information and reference images using a rule engine, and then inputting the structured creative elements into a reasoning model to generate a storyboard scheme for scene environment parameters and at least one scene corresponding to the target video. Two of these determination methods are explained in detail below.

[0061] Optionally, based on video description information and reference images, a scene environment parameter and at least one scene segmentation scheme corresponding to the target video are obtained, including: using a semantic reasoning model, based on scene task prompts, video description information and reference images, to generate a scene environment parameter and at least one scene segmentation scheme corresponding to the target video.

[0062] The semantic reasoning model can be understood as an artificial intelligence model with multimodal semantic understanding and logical generation capabilities, serving as the core computational unit for parameter extraction and storyboard generation. The semantic reasoning model can parse input task instructions, text creation requirements, and image reference materials. Through semantic understanding, logical reasoning, and content generation capabilities, it outputs structured scene environment parameters and storyboard schemes according to rules. Optionally, the model structure of the semantic reasoning model can include at least one of the following: a Transformer structure containing only a decoder, an encoder-decoder Transformer structure, a multimodal fusion Transformer structure, a hybrid expert Transformer structure, or a visual-language cross-modal concatenation structure. For example, a multimodal large language model that can receive storyboard task prompts, video description information, and reference images, and automatically reason and output corresponding environmental parameters and storyboard schemes, is a semantic reasoning model.

[0063] The storyboard task prompt can be understood as an instruction text that gives the semantic reasoning model the task of generating storyboards and constrains the execution standards. It is the core rule basis for the model to execute tasks. The storyboard task prompt can unify and clarify the two core tasks that the semantic reasoning model needs to complete: extracting scene environment parameters and generating storyboard schemes. It also specifies the execution logic, content boundaries and output format of the two tasks, guiding the semantic reasoning model to output structured results according to the system's preset standards, avoiding output content that deviates from the requirements or has an unparseable format. For example, an instruction text that reads "Based on the input video description information and reference images, first extract global scene environment parameters, then generate coherent storyboards; the environment parameters need to cover five dimensions: lighting, time period, environment, color scheme, and atmosphere; the storyboards need to include shot size, camera movement, shot duration, and scene content; finally, output in JSON format" is a storyboard task prompt.

[0064] In one scenario, storyboard task prompts are obtained, and the storyboard task prompts, video description information, and reference images are input into a semantic reasoning model. The semantic reasoning model processes the video description information and reference images based on the storyboard task prompts to obtain the output scene environment parameters corresponding to the target video and the storyboard scheme for at least one storyboard.

[0065] Optionally, based on video description information and reference images, a storyboard scheme for scene environment parameters and at least one storyboard corresponding to the target video is obtained, including: using an intelligent system to generate a storyboard scheme for scene environment parameters and at least one storyboard corresponding to the target video based on video description information and reference images.

[0066] Intelligent systems can be understood as systems capable of autonomous control based on machine learning models. For example, an intelligent system can make decisions and autonomously execute actions based on machine learning models to achieve preset goals or complete preset tasks—a virtual object or physical entity. An intelligent system can be an automated program that understands user intent and can use models or invoke tools to complete various types of tasks. An intelligent system can be understood as an intelligent operating platform with autonomous analysis and task processing capabilities. An intelligent system can be a system carrier that receives external input instructions, relies on its own reasoning, computation, and invocation capabilities to independently process business and input corresponding response results. For example, an intelligent system can be an interactive service platform that can answer user queries, process task requests, and autonomously complete logical judgments. In some cases, examples of intelligent systems may include, but are not limited to: agents, robots, chatbots, digital avatars, intelligent customer service, digital assistants, etc. Alternatively, an intelligent system can also be an intelligent role implemented based on a machine learning model. Intelligent systems can process user requests based on generative models (e.g., language models, multimodal models) to execute specified types of tasks. In some cases, intelligent systems can also be associated with virtual accounts, which may have corresponding avatars or nicknames.

[0067] In one scenario, given video description information and reference images, the video description information and reference images can be provided to an intelligent system. The intelligent system then processes the video description information and reference images to obtain scene environment parameters corresponding to the target video and a storyboard scheme for at least one scene.

[0068] S130. For each storyboard, based on video description information, reference images, scene environment parameters and storyboard scheme, construct target prompts corresponding to the storyboard, and use a video generation model to generate storyboard videos based on the target prompts.

[0069] In this context, target cue words can be understood as instruction text used to drive the video generation model to produce corresponding shots. Target cue words can be structured cue text that integrates multi-dimensional information such as style constraints, environmental benchmarks, and shot content; they are the direct input content for the video generation model. Target cue words can be used to indicate the visual presentation effect of a single shot. They can be composed of scene environment parameters, shot composition schemes, video description information, and reference image features, clearly defining all generation requirements for a single shot, including shot content, visual style, lighting and color tone, and compositional details, allowing the video generation model to analyze and produce the corresponding shot.

[0070] The video generation model can be understood as an artificial intelligence model capable of generating video content from text and image inputs. It can be a type of multimodal generation model, generating dynamic video clips with corresponding content and style based on input target prompts and reference images. It is the core execution unit for achieving dynamic image generation. Optionally, the model structure of the video generation model can include at least one of the following: spatiotemporal diffusion model structure, autoregressive Transformer video generation structure, encoder-decoder video generation structure, cascaded step-by-step video generation structure, and multi-condition controllable video generation structure.

[0071] In this context, a storyboard video can be understood as an independent short video clip generated corresponding to a single storyboard scheme. A storyboard video can be a component of a complete target video; each storyboard video corresponds to a complete dynamic frame of a single shot, possessing independent duration, camera movement effects, and visual content. For example, continuing with the previous example, the 4-second dynamic video clip generated corresponding to the wide-angle zoom-in shot is the storyboard video for storyboard 1.

[0072] In one scenario, the provided solution constructs target cue words corresponding to the scene based on video description information, reference images, scene environment parameters, and scene segmentation schemes, including at least one of the following: Feature extraction is performed on the reference image to obtain its features; the video description information, reference image features, scene environment parameters, and shot composition scheme are then combined to obtain the target prompt words.

[0073] Style constraint information is generated based on style description information in video description information; visual constraint instructions corresponding to the reference image are generated based on the reference image and visual constraint instructions are determined based on the reference image and visual constraint instructions; environmental constraint information is generated based on scene environment parameters; shot content description information is generated based on the shot scheme; style constraint information, visual constraint information, environmental constraint information and shot content description information are concatenated in a preset priority order to obtain target cue words corresponding to the shot; style constraint information has the highest priority and shot content description information has the lowest priority.

[0074] In one scenario, the provided solution employs a video generation model to generate storyboard videos based on target prompts, which may include at least one of the following: inputting the target prompts and reference images into the video generation model to obtain the output storyboard video; using the image generation module in the video generation model to generate storyboard images corresponding to the storyboards based on the target prompts and reference images; or using the video generation module to generate storyboard videos based on the target prompts and storyboard images. The following provides a detailed explanation of one such storyboard video generation method.

[0075] Optionally, the video generation model includes an image generation module and a video generation module; using the video generation model to generate storyboard videos based on target prompts includes: using the image generation module to generate storyboard images corresponding to the storyboards based on the target prompts and reference images; and using the video generation module to generate storyboard videos based on the target prompts, storyboard images, and reference images.

[0076] In one scenario, the video generation model can be a generative model employing a two-tiered architecture of "static image generation - dynamic video extension," serving as the core computational vehicle for producing single-shot dynamic videos. The overall video generation model can consist of an image generation module and a video generation module connected in series, using distributed generation logic. First, a high-precision single-frame reference image is generated through a pre-processor module, and then the static image is extended into a continuous dynamic video that meets the requirements of camera movement and duration through a post-processor module, balancing image detail accuracy with inter-frame motion continuity.

[0077] The image generation module can be understood as a pre-processing unit of the video generation model, used to generate static reference frames corresponding to a single storyboard. Located in the first stage of the two-level generation chain, the image generation module receives textual constraints from target prompts and visual constraints from reference images, outputting high-definition static frames that meet the requirements of storyboard content, style, tone, and environmental parameters, providing precise visual anchor points for subsequent video generation.

[0078] The video generation module can be understood as a post-computation unit of the video generation model, used to expand static storyboard images into continuous dynamic video clips. Located in the second stage of the two-level generation chain, the video generation module uses the storyboard image as the starting reference frame and, combined with the requirements of camera movement, duration, and scene content in the target prompts, completes the inter-frame transitions and motion logic, outputting a coherent single-storyboard dynamic video.

[0079] In this context, storyboard images can be understood as static keyframes corresponding to a single storyboard shot, serving as an intermediate visual product between textual requirements and dynamic video. Storyboard images can be output by the image generation module, fully encompassing all visual elements of a single storyboard shot, including scene layout, character design, lighting, color tone, and artistic style. They precisely match the visual design of the storyboard scheme and serve as the starting reference frame for subsequent video generation modules. For example, continuing with the previous example, the high-definition static image generated for the panoramic storyboard shot, presenting the silhouette of a woman in a bamboo forest reminiscent of ink wash painting, is the storyboard image for storyboard shot 1.

[0080] In one context, a storyboard video can be understood as a dynamic video clip corresponding to a single storyboard shot, serving as an independent shot unit that constitutes the complete target video. Storyboard videos can be generated by a video generation module based on storyboard images, possessing the duration, camera movement effects, and dynamic performance set by the storyboard scheme. Each storyboard video segment corresponds to an independent shot, and multiple storyboard videos are spliced ​​together in narrative order (or storyboard number order) to form a complete target video. For example, continuing with the previous example, a 4-second, slowly advancing panoramic dynamic video clip of a bamboo forest in ink painting style is the storyboard video for storyboard shot 1.

[0081] In one scenario, for each scene in at least one storyboard, after obtaining the target cue word corresponding to the scene, the target cue word and reference image can be used as multimodal inputs to the video generation model. Further, the target cue word and reference image are received by the image generation module in the video generation model, and processed to obtain the scene image corresponding to the scene. Further still, the target cue word, reference image, and scene image can be input to the video generation module in the video generation model, and processed to obtain the output scene video corresponding to the scene.

[0082] In one scenario, to further enhance the interactivity and controllability of storyboard creation, enabling users to preview, verify, and flexibly adjust the storyboard content before video generation, thereby reducing the deviation between the final generated result and the creative expectation, the video generation method may further include steps of displaying and outputting the storyboard script, interactive editing, and synchronously updating the cue words. The specific process is as follows: Based on the target cue words, obtain the storyboard script for at least one storyboard scene and display at least one of the storyboard scripts; in response to receiving a script editing operation for the storyboard script, obtain the edited storyboard script and update the target cue words based on the edited storyboard script.

[0083] The storyboard can be understood as a visually interactive medium presenting a single-shot storyboard in a structured text format. It serves as an intermediate product for users to preview and verify the storyboard before video generation. The storyboard can be derived from target cues, intuitively displaying core design information such as shot size, camera movement, shot duration, and scene content for a single shot. This lowers the barrier to understanding for users, allowing them to confirm whether the storyboard design meets expectations before officially generating the target video. It acts as an intermediary between automatic generation logic and manual adjustment needs. In some cases, the storyboard may include image cues and / or video cues. The image cues can be used to generate the storyboard image for the corresponding shot, and the video cues can be used to generate the storyboard video for the corresponding shot. For example, the storyboard for a particular shot may include both image cues and video cues. The image cues are: 3D-to-2D style, symmetrical composition in the distance, frontal view of an indoor theater stage, a deep red velvet curtain dominating the frame, a warm golden spotlight shining through the curtain gaps, a faint golden light reflecting off the wooden stage floor, cool white edge lighting from footlights and chandelier frames, a light mist clinging to the ground, the audience seating shrouded in darkness with only outlines visible, the overall color palette warm gold / deep red / blue-purple / black / ivory white, creating a solemn atmosphere with a sense of anticipation for the opening, without any text elements. The video cues could be: Starting with a keyframe (i.e., a storyboard image), the camera maintains a steady, low-to-medium speed, the deep red curtain slowly opens from the center to both sides, the warm golden light from the gaps gradually widens and creates longer highlight bands on the floor, the stage mist is gently pushed aside by airflow to create layers, cool white fill light from the side enhances the outline and edges, the rhythm gradually brightens from restraint, creating a sense of opening ceremony. No subtitles or watermarks are included.

[0084] The script editing operation can be understood as an interactive operation by which the user modifies and adjusts the displayed storyboard script. It is a trigger action that the system uses to capture the user's customized optimization needs. The script editing operation can encompass a series of interactive operations such as modifying, adding, deleting, and replacing any information item in the storyboard script, including shot size, camera movement, duration, and scene description. By recognizing this operation, the system obtains the user's intention to adjust the original automatically generated storyboard plan, providing a basis for subsequent optimization and generation instructions. Optionally, the script editing operation can be triggered by at least one of the following: text editing trigger, parameter control adjustment trigger, voice command trigger, preset option click trigger, and shortcut key trigger.

[0085] In one scenario, after obtaining the target cue words corresponding to the storyboard, keyword extraction can be performed on the target cue words. Based on the extracted keywords, a storyboard script corresponding to the storyboard can be generated and displayed in the video editing interface, with a text editing box displayed at the relevant position of the storyboard script. Furthermore, upon receiving a script editing operation for the storyboard script through the text editing box, script adjustment information is obtained based on the script editing operation and displayed in the text editing box. Further, in response to receiving an editing completion instruction, the storyboard script is edited based on the script adjustment information displayed in the text editing box to obtain an edited storyboard script. Further still, the target cue words corresponding to the storyboard can be updated based on the edited storyboard script to obtain updated target cue words.

[0086] S140. At least one storyboard video is spliced ​​together to obtain the target video.

[0087] The target video can be understood as the final, complete video product that meets the user's creative needs. The target video can be composed of all storyboard videos integrated in a preset order, possessing continuous narrative logic, a unified video style, and a complete playback duration; it is the final output of the entire video generation process. Optionally, the preset order can include at least one of the following: storyboard number order, narrative order.

[0088] In one scenario, after obtaining at least one storyboard video, the at least one storyboard video can be spliced ​​together in sequence according to the storyboard number, and the resulting spliced ​​video can be identified as the target video. For example, continuing with the previous example, the complete dance short film formed by splicing the four storyboard videos of a woman in Hanfu dancing in a bamboo forest in sequence according to the storyboard number is the target video.

[0089] In one scenario, to achieve efficient iterative optimization of the target video, reduce the computational resource consumption and time cost of full regeneration, and support users in making targeted content adjustments based on the initial generation results, the video generation method can further incorporate a feedback-driven storyboard optimization mechanism. When the system receives video feedback information for the target video and determines that the feedback pertains to a local adjustment requirement that is not a full regeneration, it iteratively optimizes the storyboard scheme based on the feedback information, synchronously updates the generation instructions, and finally outputs the optimized target video. The specific process is as follows: In response to receiving video feedback information for the target video and the video feedback information meeting the first condition, the storyboard scheme is adjusted based on the video feedback information to obtain the adjusted storyboard scheme. Based on the adjusted storyboard scheme, the target prompt words are updated to obtain the updated target prompt words. Using a video generation model, the updated target video is generated based on the updated target prompt words.

[0090] Video feedback information can be understood as users' optimization and adjustment requests for the target video, serving as the core input driving the iteration of the storyboard design. Video feedback information can encompass user suggestions for modifications across multiple dimensions, including shot effects, pacing, duration, image details, and camera movement. Analyzing this information helps pinpoint specific adjustment points and their magnitude. For example, a user's suggestion to "change the camera movement of the second scene to a slow horizontal pan, and increase its duration by 1 second" constitutes video feedback information.

[0091] The first condition can be understood as a judgment rule used to classify and verify video feedback information, serving as a threshold for distinguishing iterative optimization modes. The first condition can define whether the current video feedback information belongs to a lightweight local adjustment scenario. In some cases, the first condition may include not regenerating the storyboard. When the video feedback information meets this first condition, an efficient iterative process of "adjusting the storyboard—updating the prompts—local regeneration" can be executed; if not, a complete process of fully re-extracting scene environment parameters and generating the storyboard and target video is triggered.

[0092] In one scenario, a text editing box for the target video is displayed through the video editing interface. Further, upon receiving video feedback information for the target video through the text editing box, it can be determined whether the video feedback information includes information on regenerating the storyboard. Further, if the video feedback information does not include information on regenerating the storyboard, it can be determined that the video feedback information satisfies a first condition. Then, based on the video feedback information, the storyboard scheme for the corresponding scene is adjusted to obtain the adjusted storyboard scheme. Further, the target cue words for the corresponding scene can be updated based on the adjusted storyboard scheme to obtain the updated target cue words. Further, the updated target cue words and reference images are provided to the video generation model, which processes the target cue words and reference images to obtain an output video, which is then used as the updated target video.

[0093] For example, Figure 2 This is a schematic diagram of a video editing interface provided in an embodiment of this disclosure. Figure 2As shown, the video generation process includes four steps: 1. Storyboard generation; 2. Keyframe generation; 3. Video generation; 4. Confirm download. In the storyboard generation step, the following content can be displayed through the video editing interface. Video generation modes can include three types: Quick Mode, Storyboard Mode, and Custom Mode. Entering a mode selection operation for Storyboard Mode updates it to the selected state. Selecting duration templates can include four types: 10-second template, 15-second template, 20-second template, and 30-second template. Entering a template selection operation for the 15-second template updates it to the selected state. The target video's aspect ratio is "9:16 vertical narrow screen," and the output resolution is "1080P ultra-high definition." Optional style tags can include ten types: Chinese Style, 3D to 2D, Realistic, Cartoon, Stylized, Minimalist, Cyberpunk, Oil Painting, Ink Painting, and Thick Paint. Entering a style selection operation for the 3D to 2D style tag updates it to the selected state. Furthermore, text information can be entered into the core content input box to obtain the core content, which is then displayed in the core content input box. Similarly, text information can be entered into the video content outline input box to obtain the video content outline, which is then displayed in the video content outline input box. The displayed core content and video content outline can be defined as content description information. Furthermore, reference description information for video style and / or camera movement can be entered through the form reference control, and reference videos can be uploaded through the reference video upload control. The entered reference description information and the uploaded reference video will serve as references for video style and camera movement. When a reference image has already been uploaded, the style of the generated image will strictly follow the reference image; the entered reference description information and the uploaded reference video will be added as "supplementary atmosphere descriptions," but will not overwrite the reference image. Furthermore, the reference image upload area in the video editing interface displays 6 slots. The first 5 slots sequentially upload 5 images, which are respectively... Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 Furthermore, upon receiving a control trigger operation for the "Generate Storyboard" control, a storyboard can be generated based on the selected style tag, the selected video duration, the input content description information, and the 5 uploaded reference images, and the generated storyboard will be displayed in the video editing interface. For example... Figure 2As shown, the storyboard includes five scenes: Scene 1, Scene 2, Scene 3, Scene 4, and Scene 5. Taking Scene 1 as an example, the storyboard can include image prompts and video prompts. A user feedback input box is also shown, allowing users to receive script editing operations for the storyboard, obtain the edited storyboard, and update the target prompts based on the edited storyboard. Furthermore, upon receiving a control trigger operation for the "Confirm Storyboard → Generate Keyframe" control, keyframes for the corresponding scene are generated based on the storyboard.

[0094] The technical solution of this disclosure, by acquiring unified video description information and reference images as creative input, automatically derives and generates global scene environment parameters and storyboard schemes for the target video. This transforms fragmented creative needs into standardized global environmental benchmarks and structured shot planning, replacing the traditional manual storyboard design mode. It provides a unified execution basis for multi-shot video generation, effectively lowering the professional threshold for video creation. Furthermore, by constructing exclusive target prompts for each storyboard based on video description information, reference images, scene environment parameters, and storyboard schemes, it can layer and integrate style constraints, visual constraints, environmental constraints, and shot content, solidifying a shared style tone, visual characteristics, and environmental attribute benchmark for the entire film. This effectively avoids style drift and inconsistent environmental representation that occur during the independent generation of multiple shots, ensuring the coherence and unity of visual expression across shots. Furthermore, by employing a video generation model to generate segmented videos one by one based on the target prompts corresponding to each shot, and then integrating them to obtain the complete target video, the entire chain of automated generation from creative requirement input to complete video output is achieved. This eliminates the need for users to manually write input prompts for each shot, significantly simplifying the creation process of multi-shot videos. Simultaneously, it ensures that each shot not only meets the single-shot narrative design requirements but also strictly adheres to a globally unified creative standard. The technical solution of this disclosure addresses the problems in related technologies where video generation relies on manual shot planning, requires manual input of prompts for each shot (low efficiency), suffers from easy style drift in multi-shot videos, lacks a unified and fixed global environment standard, and has a high professional threshold for creation. It can automatically complete global environment settings and shot planning based on user input, and ensures consistency in style and environment across multiple shots through layered constraint on target prompts. This achieves automated and standardized generation of complete videos, effectively reducing the operational costs and professional limitations of video creation, improving the stylistic uniformity and narrative coherence of multi-shot generated videos, and ultimately enhancing the overall efficiency and quality of intelligent video creation.

[0095] Figure 3 This is a schematic flowchart illustrating another video generation method provided in this embodiment. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 3As shown, the method in this embodiment may specifically include: S210. Obtain video description information and reference images.

[0096] S220, Obtain storyboard task prompts.

[0097] The storyboard task prompts can be understood as instructive text that issues storyboard generation tasks and constrains execution standards to the semantic reasoning model. They serve as the core rules for the model's task execution. The storyboard task prompts clearly define the two core tasks the semantic reasoning model needs to complete: extracting scene environment parameters and generating storyboard schemes. They also specify the execution logic, content boundaries, and output format of these two tasks, guiding the semantic reasoning model to output structured results according to the system's preset standards, preventing output content from deviating from requirements or becoming unparseable. The storyboard task prompts can be used to indicate the rules for extracting scene environment parameters, the specifications for generating storyboard schemes, and the format requirements for the output results.

[0098] The rules for extracting scene environment parameters can be understood as the execution standards and constraints that guide the model in extracting scene environment parameters from the input information. These extraction rules clearly define the dimensions of the scene environment parameters to be extracted, the basis for extracting each dimension, the standardized expression of the parameters, and their value boundaries. This ensures that the parameters extracted by the model are complete in dimensions and uniform in expression, serving as a shared environmental benchmark for all shots in the target video. For example, the rules may require the extraction of five types of parameters: lighting, time period, environment, color scheme, and atmosphere. These parameters can only be derived from video description information and reference images; no additional environmental settings beyond the input are permitted. This constraint standard constitutes the scene environment parameter extraction rules.

[0099] The storyboard generation guidelines can be understood as a set of rules that constrain the generation logic, content elements, and professional standards of the storyboard. These guidelines clearly define the narrative principles of storyboard breakdown, the information items a single storyboard must include, the logic of shot composition and camera movement, and duration requirements, ensuring that the generated storyboard is narratively coherent, rhythmically sound, and informationally complete, conforming to the professional logic of video creation. For example, the guidelines require the number of storyboards to be controlled between 3 and 5, designed according to a narrative rhythm of "environmental setup - main subject presentation - detail highlighting." Each storyboard must be labeled with its shot type, camera movement, duration, and content; this constraint standard constitutes the storyboard generation guidelines.

[0100] The output format requirements can be understood as standardized constraints on the data structure and presentation of the model's final output. Format requirements specify the data format, field naming rules, hierarchical structure, and data types used in the output, ensuring that the model outputs standardized structured data that can be directly read, parsed, and called by the system, avoiding format confusion that could prevent subsequent processing. For example, requiring the output to be in JSON format, with the top-level field containing "Scene Environment Parameters" and "Storyboard Scheme," the storyboards arranged in an array, and the duration field being a numeric type, constitutes the output format requirements.

[0101] In one scenario, storyboard task prompts, which specify the rules for extracting scene environment parameters, the specifications for storyboard generation, and the output format requirements, can be pre-stored as standardized execution templates. Furthermore, when the storyboard generation process begins, these pre-set storyboard task prompts can be directly retrieved without requiring manual configuration or dynamic assembly by the user. Once retrieved, they can be used as core instructions input into the semantic reasoning model to constrain the overall execution logic of subsequent scene environment parameter extraction and storyboard generation.

[0102] S230. Using a semantic reasoning model, based on the storyboard task prompts, video description information, and reference images, generate scene environment parameters corresponding to the target video and a storyboard scheme for at least one storyboard.

[0103] The semantic reasoning model can be understood as an artificial intelligence model with multimodal semantic understanding and logical generation capabilities, serving as the core computational unit for parameter extraction and storyboard generation. The semantic reasoning model can parse input task instructions, text creation requirements, and image reference materials, and through semantic understanding, logical reasoning, and content generation capabilities, output structured scene environment parameters and storyboard schemes according to rules.

[0104] In one scenario, the provided solution, in order to adapt to the diverse combinations of user input materials, can stably and accurately extract the global scene environment benchmark of the target video even when the scene-related input information has different levels of completeness. In the step of generating scene environment parameters using a semantic reasoning model, the system will select the corresponding valid input combination to input into the semantic reasoning model based on the judgment results of whether the reference image contains scene images and whether the content description information contains scene description information, thereby generating scene environment parameters that match the target video.

[0105] In one scenario, the provided solution employs a semantic reasoning model to generate scene environment parameters corresponding to the target video based on storyboard task prompts, video description information, and reference images. This can include at least one of the following: responding to a reference image including a scene image but the content description information not including scene description information, employing a semantic reasoning model to generate scene environment parameters corresponding to the target video based on storyboard task prompts and scene images; responding to a reference image not including a scene image but the content description information including scene description information, employing a semantic reasoning model to generate scene environment parameters corresponding to the target video based on storyboard task prompts and scene description information; responding to a reference image including a scene image and the content description information including scene description information, employing a semantic reasoning model to generate scene environment parameters corresponding to the target video based on storyboard task prompts, scene images, and scene description information.

[0106] Optionally, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on storyboard task prompts, video description information, and reference images. This includes: in response to the reference image including a scene image and the content description information not including scene description information, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on storyboard task prompts and scene images.

[0107] Scene images can be understood as reference images that present the environmental space as the main subject and carry the complete visual characteristics of the scene. They are the core visual materials for extracting scene environmental parameters. A scene image can focus on the overall environment, including complete visual information such as the scene's spatial layout, environmental elements, lighting conditions, color tone, texture, and atmosphere. Various environmental attributes of the scene can be directly extracted from it. For example, a panoramic environmental image uploaded by a user, showing a bamboo forest surrounded by a thin mist, is a scene image.

[0108] Scene description information can be understood as the text content within the content description information that describes the environment in which the target video is located. It serves as the textual basis for extracting scene environment parameters. Scene description information can specifically describe environmental attributes such as the location type, environmental features, weather conditions, lighting conditions, and overall atmosphere of the scene. For example, text specifically describing the environment, such as "deep in a bamboo forest shrouded in morning mist," belongs to scene description information.

[0109] In one scenario, given reference images and video description information, it can be determined whether the reference images include scene images based on the material type corresponding to each reference image. Furthermore, semantic analysis can be performed on the content description information to determine whether it includes scene description information. Further, if the reference images include scene images but the content description information does not, a semantic reasoning model can be used to extract scene environment parameters from the scene images based on storyboard task prompts, thereby obtaining scene environment parameters corresponding to the target video.

[0110] Optionally, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on storyboard task prompts, video description information, and reference images. This includes: in response to the reference image not including a scene image and the content description information including scene description information, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on storyboard task prompts and scene description information.

[0111] In one scenario, given reference images and video description information, it can be determined whether the reference images include scene images based on the material type corresponding to each reference image. Furthermore, semantic analysis can be performed on the content description information to determine whether it includes scene description information. Further, if the reference images do not include scene images but the content description information does include scene description information, a semantic reasoning model can be used to extract scene environment parameters from the scene description information based on storyboard task prompts, thereby obtaining scene environment parameters corresponding to the target video.

[0112] Optionally, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on storyboard task prompts, video description information, and reference images. This includes: in response to the reference image including a scene image and the content description information including scene description information, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on storyboard task prompts, scene images, and scene description information.

[0113] In one scenario, given reference images and video description information, it can be determined whether the reference images include scene images based on the material type corresponding to each reference image. Furthermore, semantic analysis can be performed on the content description information to determine whether it includes scene description information. Further, if both the reference images and content description information include scene description information, a semantic reasoning model can be used to extract scene environment parameters from the scene images and scene description information based on storyboard task prompts, yielding a first environment parameter and a second environment parameter. Finally, based on the first and second environment parameters, the scene environment parameters corresponding to the target video are determined.

[0114] In one scenario, storyboard task prompts are obtained, and the storyboard task prompts, video description information, and reference images are input into a semantic reasoning model. The semantic reasoning model processes the video description information and reference images based on the storyboard task prompts to obtain the output scene environment parameters corresponding to the target video and the storyboard scheme for at least one storyboard.

[0115] S240. For each storyboard, based on video description information, reference images, scene environment parameters and storyboard scheme, construct target prompts corresponding to the storyboard, and use a video generation model to generate storyboard videos based on the target prompts.

[0116] S250. At least one storyboard video is spliced ​​together to obtain the target video.

[0117] The technical solution of this disclosure introduces standardized storyboard task prompts as unified execution rules. Relying on the joint semantic reasoning of text and image multimodal inputs through a semantic reasoning model, it can automatically complete the structured extraction of global scene environment parameters and the automated generation of storyboard schemes, without requiring users to manually carry out professional storyboard design and environment settings. This effectively reduces the professional threshold and manual operation cost of video creation, and ensures the consistency of environmental parameter dimensions and the standardization of storyboard schemes through the constraints of unified rules. It establishes a globally unified environmental benchmark and shot design basis for subsequent multi-shot video generation, avoiding the style deviation and environmental disconnect problems that are prone to occur in independent multi-shot creation from the source of generation. At the same time, it significantly improves the execution efficiency of the storyboard planning stage and the standardization of the output results.

[0118] Figure 4 This is a schematic flowchart illustrating another video generation method provided in this embodiment. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 4 As shown, the method in this embodiment may specifically include: S310, Obtain video description information and reference images.

[0119] S320. Based on video description information and reference images, obtain scene environment parameters corresponding to the target video and a storyboard scheme for at least one storyboard.

[0120] S330. For each scene, generate style constraint information based on the style description information in the video description information.

[0121] Style constraint information can be understood as a top-level generation constraint fragment formed by the standardized transformation of style description information. It is the highest priority constraint fragment among the target prompts. Style constraint information can organize the scattered style descriptions input by the user into standardized style instructions that the video generation model can accurately recognize, clarifying the top-level visual requirements such as the artistic style, color tendency, and texture expression of the target video. Because it determines the overall tone of the video, it has the highest priority in the target prompts, ensuring the style consistency of the generated content.

[0122] In one scenario, style description information can be obtained from the video description information. Furthermore, the style description information is processed according to a preset standardized generation instruction template to obtain style constraint information.

[0123] S340. Based on the reference image, generate visual constraint instructions corresponding to the reference image, and determine visual constraint information based on the reference image and the visual constraint instructions.

[0124] Visual constraint instructions can be understood as rule-based instructions used to guide the extraction of visual features from reference images, serving as an intermediate rule carrier for generating visual constraint information. Visual constraint instructions can predefine the dimensions, expression standards, and constraint strength of visual features extracted from reference images. They guide the system to decompose visual elements such as character design, scene appearance, material details, and color matching according to standards, ensuring the completeness and consistency of the extracted feature dimensions, which can be directly used to generate visual constraint information. For example, the extraction rule "extract the Hanfu style, clothing color scheme, and hairstyle accessories of the character from the reference image, preserving consistency in design details" is a visual constraint instruction for the corresponding reference image.

[0125] Visual constraint information can be understood as concrete visual constraint fragments extracted and transformed from reference images and visual constraint instructions. It is the core fragment ensuring consistency between the generated content and the reference image. Visual constraint information can transform visual content such as character appearances, scene elements, material textures, and color characteristics from the reference image into text constraint instructions that the video generation model can recognize. This is used to lock in concrete visual details in the image, reduce visual deviations between the generated result and the reference material, and ensure the fidelity of core elements such as characters and scenes. Visual constraint information has the second highest priority in target prompts, second only to style constraint information.

[0126] For example, visual constraint information could be: strictly referencing a reference image to lock in the art style: reference Figure 1 (Character): The Hanfu style, hairstyle, and accessories must be consistent with the reference image; Reference Figure 2 (Scene): Bamboo forest density and light projection angle are shown in the reference diagram; Reference Figure 3(Color tone): The overall color scheme follows the reference image's dark green + off-white + light gold tones; no other style terms should be introduced into the image.

[0127] In one scenario, the input reference image is first subjected to image semantic classification and recognition to determine its material type (e.g., character image, scene space, style and texture). Based on the recognition result, feature extraction rules for the corresponding dimension are matched to generate visual constraint instructions that correspond one-to-one with the reference image. Further, the reference image and the corresponding visual constraint instructions are input into a multimodal feature parsing unit. The parsing unit, according to the extraction range and accuracy requirements defined by the instructions, parses concrete visual features such as character design, scene elements, color scheme, and material texture from the reference image. These visual features are then converted into standardized text representations that the video generation model can recognize, ultimately integrating them to form complete visual constraint information.

[0128] S350: Process the scene environment parameters according to the preset prompt information template to obtain environmental constraint information.

[0129] The prompt message template can be understood as a pre-configured, standardized format carrier used to translate structured parameters into constraint text, serving as the format specification for translating scene environment parameters. The prompt message template pre-defines the expression format of environmental constraint information, the order of parameter dimensions, the standardization of technical terminology, and semantic connection logic. When using it, users simply fill in the corresponding values ​​of each environmental parameter dimension into the template fields to quickly generate uniformly formatted and semantically standardized constraint text, ensuring complete consistency in the expression of environmental constraints across different storyboards.

[0130] Environmental constraint information can be understood as a global environmental constraint fragment generated from scene environmental parameters. It ensures that individual shots maintain consistency with the overall environmental baseline. Environmental constraint information can be generated by converting structured scene environmental parameters into natural language constraints, clearly defining the environmental attributes of each shot, such as lighting conditions, time of day, atmosphere, and spatial tone. This ensures that the environmental representation of a single shot remains consistent with the unified environmental baseline of the entire film, avoiding cross-shot style drift. For example, the instruction text "soft side-backlighting in the early morning, a bamboo forest space surrounded by mist, a tranquil and hazy atmosphere," generated from scene environmental parameters, is environmental constraint information.

[0131] In one scenario, scene environment parameters are acquired, and parameter values ​​for various dimensions such as lighting conditions, time period characteristics, spatial environment type, color tone, and overall atmosphere are analyzed sequentially. Further, based on preset environmental parameter translation rules, the discrete parameter values ​​of each dimension are transformed into standardized descriptive statements adapted to the semantic recognition habits of the video generation model, unifying the expression caliber and descriptive granularity. Further still, the environmental description statements of each dimension are logically integrated into a complete and coherent text, forming environmental constraint information that can directly participate in the concatenation of target prompt words. This serves as a global environmental benchmark to constrain the visual performance of a single shot, avoiding deviations in environmental attributes across shots.

[0132] S360 extracts information from the storyboard to obtain descriptions of the shot content.

[0133] The shot content description information can be understood as a fragment describing the content of a single shot generated from the storyboard, and it is the core content fragment in the target cues. The shot content description information integrates shot design information from the storyboard, such as shot size, camera movement, duration, main subject, action sequence, and compositional proportions, into a coherent descriptive text. It clarifies the specific narrative content and shot expression of a single storyboard, and serves as the direct content basis for generating the corresponding shot. For example, the descriptive text obtained from the conversion of a wide-angle storyboard, "Wide-angle shot, slowly advancing at a uniform speed, a woman's profile standing in the center of a bamboo forest clearing, her height occupying approximately 30% of the frame," is the shot content description information.

[0134] In one scenario, for each shot in at least one storyboard, the storyboard scheme for that shot is obtained, and the core design parameters included in the storyboard scheme, such as shot type, camera movement, duration, main subject action, and composition ratio, are analyzed one by one. Furthermore, according to a preset shot description transcription standard, the discrete parametric design information is transformed into coherent image description text that conforms to the semantic recognition of the video generation model, unifying the expression logic and granularity of description to ensure complete coverage of shot presentation and image content; ultimately, shot content description information corresponding one-to-one with that storyboard is formed, serving as the underlying content component of the target cue words in subsequent splicing.

[0135] S370. According to the preset priority order, the style constraint information, visual constraint information, environmental constraint information and shot content description information are spliced ​​together in sequence to obtain the target cue words corresponding to the storyboard. Then, a video generation model is used to generate the storyboard video based on the target cue words.

[0136] The preset priority order can be understood as the pre-defined arrangement rule of various constraint information in the target prompt, which determines the ranking of the weight of each constraint. The priority order can be based on the scope and importance of each constraint, specifying the order in which style, visual, environmental, and shot content information are concatenated. Constraints with higher priority appear earlier in the target prompt and have a greater impact on the generated model, thus ensuring that top-level style and consistency constraints take effect first, while lower-level content details do not interfere with the overall tone. In some cases, a pre-defined arrangement rule of "style constraint information > visual constraint information > environmental constraint information > shot content description information" is used; this rule is the preset priority order. That is, style constraint information has the highest priority, shot content description information has the lowest priority, and visual constraint information and environmental constraint information have decreasing priorities in that order.

[0137] In one scenario, a priority ranking rule for constraint information is pre-defined, specifying that style constraints have the highest priority, followed by visual constraints, then environmental constraints, and finally, shot content description information has the lowest priority. Furthermore, during the splicing operation, the generated style constraints, visual constraints, environmental constraints, and shot content description information are retrieved separately and concatenated sequentially according to their priority from highest to lowest. During the splicing process, appropriate semantic connectors are added to ensure the overall coherence of the instructions, placing higher-priority constraints at the beginning of the target cue words. This aligns with the video generation model's focus on pre-contextual content, ultimately integrating to form a complete target cue word corresponding to a single storyboard. Further, the target cue words, reference images, and storyboard images can be input into the video generation module of the video generation model. The video generation module processes the target cue words, reference images, and storyboard images to obtain the output storyboard video corresponding to the storyboard.

[0138] For example, continuing with the previous example, the target keywords corresponding to storyboard 2 could be: Main visual style: Chinese style, ink painting; Priority: highest at the project level; Strictly refer to reference images: Hanfu style / hairstyle / accessories and references. Figure 1 Consistent; Bamboo forest scene reference Figure 2 Color scheme follows reference Figure 3 Dark green + off-white + light gold; soft side backlighting in the early morning, thin mist in the dense bamboo forest, main colors #2D5A27 / #F5F0E8 / #D4A853, ethereal and tranquil atmosphere; panoramic view; the woman's arms are outstretched and her water sleeves are flowing, the figure occupies 60% of the height, and the composition follows the rule of thirds.

[0139] S380. At least one storyboard video is spliced ​​together to obtain the target video.

[0140] The technical solution of this disclosure decomposes the target prompt words into four independent constraint dimensions: style, visual, environment, and shot content. Standardized constraint information is generated for each dimension, and these are sequentially spliced ​​together according to a preset priority from high to low weight to form a complete generation instruction. On the one hand, this achieves layered control over the video generation effect, prioritizing the style constraints that determine the overall tone of the film. Simultaneously, visual constraints ensure the detail reproduction of reference materials, environmental constraints solidify the global environmental performance benchmark, and shot content clarifies the single-shot narrative design. The weight of each constraint layer is clear, and each layer performs its own function, effectively avoiding interference from the bottom-level shot details with the top-level style and overall consistency. On the other hand, the standardized constraint generation and sequential splicing mechanism improves the standardization and stability of the target prompt words. It avoids problems such as style drift and environmental attribute disconnect that are prone to occur when multiple shots are generated independently from the source of the generation instruction. This ensures that the generation instructions of each shot not only meet the single-shot image design requirements but also strictly follow the unified style, visual, and environmental benchmark of the entire film, significantly improving the accuracy of the shot video generation results and the visual coherence across shots.

[0141] Figure 5 This is a schematic diagram of the structure of a video generation apparatus provided in an embodiment of this disclosure. Figure 5 As shown, the video generation device includes: a first module 410, a second module 420, a third module 430, and a fourth module 440; wherein, the first module 410 is used to acquire video description information and reference images, the video description information including style description information and content description information of the target video to be generated, and the reference images are used to limit the visual information presented by the target video; the second module 420 is used to obtain scene environment parameters corresponding to the target video and a storyboard scheme of at least one storyboard based on the video description information and the reference images; the third module 430 is used to construct target prompt words corresponding to each storyboard based on the video description information, the reference images, the scene environment parameters, and the storyboard scheme of the storyboard, and to generate a storyboard video based on the target prompt words using a video generation model; the fourth module 440 is used to stitch together at least one storyboard video to obtain the target video.

[0142] The technical solution of this disclosure, by acquiring unified video description information and reference images as creative input, automatically derives and generates global scene environment parameters and storyboard schemes for the target video. This transforms fragmented creative needs into standardized global environmental benchmarks and structured shot planning, replacing the traditional manual storyboard design mode. It provides a unified execution basis for multi-shot video generation, effectively lowering the professional threshold for video creation. Furthermore, by constructing exclusive target prompts for each storyboard based on video description information, reference images, scene environment parameters, and storyboard schemes, it can layer and integrate style constraints, visual constraints, environmental constraints, and shot content, solidifying a shared style tone, visual characteristics, and environmental attribute benchmark for the entire film. This effectively avoids style drift and inconsistent environmental representation that occur during the independent generation of multiple shots, ensuring the coherence and unity of visual expression across shots. Furthermore, by employing a video generation model to generate segmented videos one by one based on the target prompts corresponding to each shot, and then integrating them to obtain the complete target video, the entire chain of automated generation from creative requirement input to complete video output is achieved. This eliminates the need for users to manually write input prompts for each shot, significantly simplifying the creation process of multi-shot videos. Simultaneously, it ensures that each shot not only meets the single-shot narrative design requirements but also strictly adheres to a globally unified creative standard. The technical solution of this disclosure addresses the problems in related technologies where video generation relies on manual shot planning, requires manual input of prompts for each shot (low efficiency), suffers from easy style drift in multi-shot videos, lacks a unified and fixed global environment standard, and has a high professional threshold for creation. It can automatically complete global environment settings and shot planning based on user input, and ensures consistency in style and environment across multiple shots through layered constraint on target prompts. This achieves automated and standardized generation of complete videos, effectively reducing the operational costs and professional limitations of video creation, improving the stylistic uniformity and narrative coherence of multi-shot generated videos, and ultimately enhancing the overall efficiency and quality of intelligent video creation.

[0143] In some embodiments of this disclosure, optionally, the second module 420 includes: a first unit and a second unit; wherein, the first unit is used to obtain storyboard task prompts, the storyboard task prompts being used to indicate the extraction rules of scene environment parameters, the generation specifications of storyboard schemes, and the format requirements of output results; the second unit is used to use a semantic reasoning model to generate scene environment parameters and at least one storyboard scheme corresponding to the target video based on the storyboard task prompts, the video description information, and the reference image.

[0144] In some embodiments of this disclosure, optionally, the second unit is specifically configured to, in response to the reference image including a scene image and the content description information not including scene description information, employ a semantic reasoning model to generate scene environment parameters corresponding to the target video based on the storyboard task prompt and the scene image; in response to the reference image not including a scene image and the content description information including scene description information, employ a semantic reasoning model to generate scene environment parameters corresponding to the target video based on the storyboard task prompt and the scene description information; and in response to the reference image including a scene image and the content description information including scene description information, employ a semantic reasoning model to generate scene environment parameters corresponding to the target video based on the storyboard task prompt, the scene image, and the scene description information.

[0145] In some embodiments of this disclosure, optionally, the third module 430 includes: a third unit, a fourth unit, a fifth unit, a sixth unit, and a seventh unit; wherein, the third unit is used to generate style constraint information based on the style description information in the video description information; the fourth unit is used to generate visual constraint instructions corresponding to the reference image based on the reference image, and determine visual constraint information based on the reference image and the visual constraint instructions; the fifth unit is used to process the scene environment parameters according to a preset prompt information template to obtain environmental constraint information; the sixth unit is used to extract information from the storyboard scheme to obtain shot content description information; the seventh unit is used to sequentially splice the style constraint information, the visual constraint information, the environmental constraint information, and the shot content description information according to a preset priority order to obtain target prompt words corresponding to the storyboard; the style constraint information has the highest priority and the shot content description information has the lowest priority.

[0146] In some embodiments of this disclosure, optionally, the video generation model includes an image generation module and a video generation module; the third module 530 further includes an eighth unit and a ninth unit; wherein, the eighth unit is used to use the image generation module to generate a storyboard image corresponding to the storyboard based on the target prompt word; the ninth unit is used to use the video generation module to generate a storyboard video based on the target prompt word and the storyboard image.

[0147] In some embodiments of this disclosure, optionally, the apparatus further includes: a fifth module and a sixth module; wherein the fifth module is configured to obtain a storyboard script for at least one storyboard based on the target prompt word, and display at least one of the storyboard scripts; the sixth module is configured to obtain an edited storyboard script in response to receiving a script editing operation for the storyboard script, and update the target prompt word based on the edited storyboard script.

[0148] In some embodiments of this disclosure, optionally, the first module 410 includes: a tenth unit, an eleventh unit, and a twelfth unit; wherein, the tenth unit is used to display multiple optional style tags through the video editing interface, receive a user's tag selection operation for the optional style tags, and obtain style description information based on the selected optional style tags; the eleventh unit is used to display a text editing box through the video editing interface, receive a user's text editing operation for the text editing box, and determine content description information; the twelfth unit is used to obtain video description information based on the style description information and the content description information.

[0149] In some embodiments of this disclosure, optionally, the first module 410 further includes a thirteenth unit and a fourteenth unit; wherein the thirteenth unit is used to display multiple slots through a video editing interface, the slots being used to carry images; and the fourteenth unit is used to receive image upload operations for the slots and determine a reference image based on the uploaded image.

[0150] In some embodiments of this disclosure, optionally, the apparatus further includes a seventh module and an eighth module; wherein the seventh module is configured to, in response to receiving video feedback information for the target video and the video feedback information satisfying a first condition, adjust the storyboard scheme based on the video feedback information to obtain an adjusted storyboard scheme, and update the target prompt word based on the adjusted storyboard scheme to obtain an updated target prompt word; the eighth module is configured to, using the video generation model, generate an updated target video based on the updated target prompt word.

[0151] The video generation apparatus provided in this disclosure can execute the video generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0152] It is worth noting that the various units and modules included in the above-mentioned video generation device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0153] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0154] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0155] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0156] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video generation methods.

[0157] In some embodiments, the video generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video generation method by any other suitable means (e.g., by means of firmware).

[0158] Various implementations of the systems and techniques described above herein can be implemented in digital circuit systems, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.

[0159] Computer programs used to implement the video generation method of this disclosure can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] This disclosure provides a computer-readable storage medium storing computer instructions for causing a processor to execute a video generation method, comprising: acquiring video description information and a reference image, wherein the video description information includes style description information and content description information of a target video to be generated, and the reference image is used to define the visual information presented by the target video; obtaining scene environment parameters corresponding to the target video and a storyboard scheme for at least one storyboard based on the video description information and the reference image; for each storyboard, constructing target prompts corresponding to the storyboard based on the video description information, the reference image, the scene environment parameters, and the storyboard scheme, and generating a storyboard video based on the target prompts using a video generation model; and splicing at least one storyboard video to obtain the target video.

[0161] In the context of this disclosure, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, also known as flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode-ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0163] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0164] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0165] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of embodiments of this disclosure.

[0166] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements a video generation method according to any embodiment of this disclosure.

[0167] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, characterized in that, include: Obtain video description information and reference images. The video description information includes style description information and content description information of the target video to be generated. The reference images are used to define the visual information presented by the target video. Based on the video description information and the reference image, obtain the scene environment parameters corresponding to the target video and at least one storyboard scheme. For each of the aforementioned storyboards, based on the video description information, the reference image, the scene environment parameters, and the storyboard scheme, target prompt words corresponding to the storyboard are constructed, and a video generation model is used to generate storyboard videos based on the target prompt words; At least one of the storyboard videos is stitched together to obtain the target video.

2. The video generation method according to claim 1, characterized in that, The step of obtaining a scene environment parameter and at least one storyboard based on the video description information and the reference image, includes: Obtain storyboard task prompts, which are used to indicate the extraction rules of scene environment parameters, the generation specifications of storyboard schemes, and the format requirements of output results; Using a semantic reasoning model, based on the storyboard task prompts, the video description information, and the reference image, a storyboard scheme for at least one scene corresponding to the target video is generated.

3. The video generation method according to claim 2, characterized in that, The step employs a semantic reasoning model to generate scene environment parameters corresponding to the target video based on the storyboard task prompts, the video description information, and the reference image, including: In response to the fact that the reference image includes a scene image and the content description information does not include scene description information, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on the storyboard task prompt words and the scene image; In response to the fact that the reference image does not include a scene image and the content description information includes scene description information, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on the storyboard task prompts and the scene description information; In response to the fact that the reference image includes a scene image and the content description information includes scene description information, a semantic reasoning model is used to generate scene environment parameters corresponding to the target video based on the storyboard task prompts, the scene image, and the scene description information.

4. The video generation method according to claim 1, characterized in that, The step of constructing target prompts corresponding to the scene based on the video description information, the reference image, the scene environment parameters, and the scene segmentation scheme includes: Based on the style description information in the video description information, style constraint information is generated; Based on the reference image, a visual constraint instruction corresponding to the reference image is generated, and visual constraint information is determined based on the reference image and the visual constraint instruction. The scene environment parameters are processed according to the preset prompt information template to obtain environmental constraint information; Information is extracted from the aforementioned storyboard scheme to obtain scene content description information; According to a preset priority order, the style constraint information, the visual constraint information, the environmental constraint information, and the shot content description information are sequentially spliced ​​together to obtain the target prompt words corresponding to the storyboard; the style constraint information has the highest priority and the shot content description information has the lowest priority.

5. The video generation method according to claim 1, characterized in that, The video generation model includes an image generation module and a video generation module; the step of generating a storyboard video based on the target prompt words using the video generation model includes: Using the image generation module, a storyboard image corresponding to the storyboard is generated based on the target prompt and the reference image; The video generation module is used to generate a storyboard video based on the target prompt, the storyboard image, and the reference image.

6. The video generation method according to claim 1, characterized in that, Also includes: Based on the target prompt words, obtain at least one storyboard script for a scene, and display at least one of the storyboard scripts; In response to receiving a script editing operation for the storyboard, the system obtains the edited storyboard and updates the target prompt based on the edited storyboard.

7. The video generation method according to claim 1, characterized in that, The acquisition of video description information includes: The video editing interface displays multiple optional style tags, receives user selection operations for the optional style tags, and obtains style description information based on the selected optional style tags; different optional style tags correspond to different style types; The video editing interface displays a text editing box, receives text editing operations from the user on the text editing box, and determines the content description information; Video description information is obtained based on the style description information and the content description information.

8. The video generation method according to claim 1, characterized in that, The method for acquiring the reference image includes: The video editing interface displays multiple slots, which are used to hold images; Receive an image upload operation for the slot, and determine a reference image based on the uploaded image.

9. The video generation method according to claim 1, characterized in that, Also includes: In response to receiving video feedback information for the target video and the video feedback information satisfying a first condition, the storyboard scheme is adjusted based on the video feedback information to obtain an adjusted storyboard scheme, and the target prompt word is updated based on the adjusted storyboard scheme to obtain an updated target prompt word; Using the aforementioned video generation model, an updated target video is generated based on the updated target prompts.

10. A video generation apparatus, characterized in that, include: The first module is used to acquire video description information and reference images. The video description information includes style description information and content description information of the target video to be generated, and the reference images are used to limit the visual information presented by the target video. The second module is used to obtain scene environment parameters and at least one storyboard scheme corresponding to the target video based on the video description information and the reference image. The third module is used to construct target prompt words corresponding to each of the storyboards based on the video description information, the reference image, the scene environment parameters and the storyboard scheme, and to generate storyboard videos based on the target prompt words using a video generation model. The fourth module is used to stitch together at least one of the storyboard videos to obtain the target video.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1-9.

Images