Video generation method and device, medium, electronic equipment and program product

Abstract

A video generation method and apparatus, a medium, an electronic device and a program product relate to the technical field of computers, and display a video script in response to video generation information, then display a split image of a first picture quality in response to a confirmation operation for the video script, and then display a split image of a second picture quality in response to a confirmation operation for the split image. According to the embodiment of the invention, the video with the second picture quality is displayed, and the second picture quality is greater than the first picture quality, so that the initial rendering time can be greatly reduced by generating the low-cost split image in advance, the waste of expensive computing resources and time cost before generating the video of the final version can be avoided, and the cost of video generation errors is greatly reduced. According to the method and the device, the transparent image generated by the video and the controllability can be ensured, the user is allowed to participate in the video creation process in the early stage, the participation sense and the satisfaction degree of the user are enhanced, and the situation that the finally generated video does not meet the requirements of the user is avoided.

Description

Technical Field

[0001] The technical solution relates to the field of computer technology, specifically to a video generation method, apparatus, medium, electronic device, and program product. Background Technology

[0002] With the rapid development of video processing technology, users' demand for video content creation is increasing. However, related video generation technologies still face many bottlenecks: on the one hand, traditional template-based solutions are rigid in terms of material adaptability, style consistency, and narrative logic, making it difficult to automatically synthesize high-quality finished products based on fragmented images, text, or spoken scripts provided by users; on the other hand, although deep learning-based generative models can produce more flexible camera language, users cannot intervene once they input their requirements until the final output video is released. If the script or storyboard is not satisfactory, a complete regeneration is required. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] Firstly, a video generation method is provided, including: In response to video generation information, a video script is displayed, the video script being generated based on the video generation information; In response to a confirmation operation on the video script, a storyboard image is displayed, the storyboard image being generated based on the video script and having a first picture quality; In response to a confirmation operation on the storyboard image, a video is displayed, the video being generated based on the storyboard image, the video having a second picture quality, and the second picture quality being greater than the first picture quality.

[0005] Secondly, a video generation apparatus is provided, comprising: A first display module is configured to display a video script in response to video generation information, wherein the video script is generated based on the video generation information. The second display module is configured to display a storyboard image in response to a confirmation operation on the video script. The storyboard image is generated based on the video script and has a first image quality. A third display module is configured to display a video in response to a confirmation operation on the storyboard image, the video being generated based on the storyboard image, the video having a second image quality, and the second image quality being greater than the first image quality.

[0006] Thirdly, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0007] Fourthly, an electronic device is provided, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.

[0008] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] Based on the above technical solution, by responding to video generation information, a video script is displayed; then, in response to a confirmation operation on the video script, a storyboard image of first-frame quality is displayed; and subsequently, in response to a confirmation operation on the storyboard image, a video of second-frame quality is displayed. The second-frame quality is higher than the first-frame quality. This not only significantly reduces initial rendering time by generating low-cost storyboard images first, but also avoids wasting expensive computing resources and time before generating the final version of the video, greatly reducing the cost of video generation errors. Secondly, by displaying the video script and storyboard images, initial script and visual references can be provided to the user, ensuring transparency and controllability in video generation. It also allows users to participate in the video creation process early in the process, enhancing user engagement and satisfaction, and preventing the final video from failing to meet user needs.

[0010] Other features and advantages of the technical solution will be described in detail in the following detailed implementation section. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the technical solution will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic diagram illustrating application scenarios of video generation methods based on certain situations.

[0012] Figure 2 This is a flowchart illustrating video generation methods under certain circumstances.

[0013] Figure 3 This is a schematic diagram of the video generation interface shown under certain circumstances.

[0014] Figure 4 This is a schematic diagram of the video generation interface shown under other circumstances.

[0015] Figure 5 This is a schematic diagram of the video generation interface shown under certain circumstances.

[0016] Figure 6 This is a schematic diagram of a video editing interface shown under certain circumstances.

[0017] Figure 7 This is a flowchart illustrating a video generation method under other circumstances.

[0018] Figure 8 This is a schematic diagram of the module connections of a video generation device provided under certain circumstances.

[0019] Figure 9 This is a schematic diagram of the module connections of an electronic device provided under certain circumstances. Detailed Implementation

[0020] The technical solution will now be described in more detail with reference to the accompanying drawings. Although certain scenarios are shown in the drawings, it should be understood that the technical solution can be implemented in various forms and should not be construed as limited to the scenarios described herein. Rather, these scenarios are provided to provide a more thorough and complete understanding of the technical solution. It should be understood that the accompanying drawings and the scenarios described are for illustrative purposes only and are not intended to limit the scope of protection of the technical solution.

[0021] It should be understood that the steps described in the method implementation may be performed in different orders and / or in parallel. Furthermore, the method implementation may include additional steps and / or omit the steps shown. The scope of the technical solution is not limited in this respect.

[0022] The term "comprising" and its variations as used herein can be open-ended, meaning "including but not limited to". The term "based on" can mean "at least partially based on". The term "one case" means "at least one case"; the term "another case" means "at least one additional case"; the term "some cases" means "at least some cases". Definitions of other terms will be given in the following description.

[0023] It should be noted that the concepts of "first" and "second" mentioned here are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "one" and "more" used here are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of messages or information exchanged between the multiple devices in the implementation are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] It is understandable that before using the technical solutions provided here, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in accordance with relevant laws and regulations, and their authorization should be obtained through appropriate means.

[0027] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations described herein.

[0028] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the technical solution. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the technical solution.

[0030] At the same time, it is understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws, regulations and related provisions.

[0031] In some cases, the video generation methods provided herein can be executed by electronic devices, which can be at least one of terminal devices and servers. Figure 1 These are schematic diagrams illustrating application scenarios of video generation methods under various conditions. For example... Figure 1 As shown, the application scenarios may include terminal device 101 and server 102.

[0032] The terminal device 101 may have a video generation program installed, which is used by users to generate videos. For example, the video generation program may be a computer program that runs through a Web (World Wide Web) browser; alternatively, it may be client software on the terminal device. The server 102 may be a backend server for the video generation program, providing backend services to the program.

[0033] In some cases, video generation information can be input into the graphical user interface (GUI) of terminal device 101. Terminal device 101 then displays a video script generated based on the video generation information through the GUI. For example, the video generation information input into the GUI of terminal device 101 can be sent to server 102. Server 102 generates a video script based on the video generation information and returns the video script to terminal device 101, which then displays the video script. When the user confirms the displayed video script, a storyboard image is displayed. This storyboard image is generated based on the video script and has a first image quality. For example, when the user confirms the video script, terminal device 101 sends a confirmation command to server 102. Server 102 generates a storyboard image based on the video script and returns it to terminal device 101, which then displays the storyboard image. When the user confirms the displayed storyboard image, terminal device 101 displays a video generated based on the storyboard image, which has a second image quality that is greater than the first image quality. For example, the user triggers the terminal device 101 to send a confirmation command to the server 102 by confirming the storyboard image. The server 102 generates a video based on the storyboard image and returns the generated video to the terminal device 101, which then displays the video.

[0034] Figure 2 This is a flowchart illustrating video generation methods under certain circumstances. For example... Figure 2 As shown, a video generation method is provided, which can be executed by a video generation device, which can be implemented by software and / or hardware. Figure 2 As shown, the method may include the following steps.

[0035] In step 210, in response to the video generation information, a video script is displayed, which is generated based on the video generation information.

[0036] Here, video generation information can be user-initiated information indicating a video generation request. This information can be one or more of text, image, audio, or video. The information included in the video generation information describes the user's video generation request. For example, the video generation information could include the text message "Generate a video of a stray cat's adventure," indicating the user's request. Of course, the video generation information can also be a multimodal video generation request. For example, the video generation information could include the text message "Generate a video of a stray cat's adventure" and a photo of a cat, indicating the user's request.

[0037] It should be understood that if the video generation information includes video information and supports uploading video information via video links, then the video generation agent can retrieve the corresponding video content based on the video link, and then analyze the video content using a large visual model to extract the corresponding video style and script structure as a reference for video generation. The video generation agent is an intelligent system based on artificial intelligence technology, capable of understanding user intent and autonomously executing video creation tasks, achieving full automation from conception to final output.

[0038] The video generation interface may display information input controls for users to input video generation information. The video generation interface can be a graphical user interface (GUI) for users to interact with a video proxy service. For example, the information input controls can be information input boxes for communicating with the video generation proxy. Alternatively, the video generation interface may also display a first confirmation control; when the user triggers the first confirmation control, the input video generation information is sent to the video generation proxy. For example, the first confirmation control can be an information sending control set in the information input box.

[0039] A video script can be the core planning document in the video generation process, detailing in text form the visuals, dialogue, actions, scene transitions, sound effects, and other elements to be presented in the video. The video script can be generated by a video generation agent based on user-inputted video generation information. For example, the video generation agent can generate a video script based on the video generation information by calling a large language model. The displayed video script can include the video title, video type, video style, characters, scenes, and scene descriptions corresponding to the scenes to be generated.

[0040] For example, the video title could be "AA and BB's Adventure," the video type could be an animated short, the video style could be cyberpunk, neon, or dynamic / static, and the characters could be "AA, a genius scientist, arrogant, who often takes BB on dangerous adventures" and "BB, AA's grandson, timid and cowardly, who is always forced to participate in AA's adventures." The scenes could be "city scenes, jungle scenes, tunnels, or snow scenes." The storyboard description information could include the storyboard duration and the segment description, such as the storyboard duration of storyboard 1 being 3 seconds, and the corresponding segment description being "the camera quickly pulls back to show the entire city and the aircraft shuttling through."

[0041] Figure 3 This is a schematic diagram illustrating the video generation interface under certain circumstances. For example... Figure 3 As shown, in the video generation interface 300, video generation information 2301 and video script 302 generated based on video generation information 2301 can be displayed.

[0042] It is worth noting that when the video generation interface displays the video script generated based on the video generation information, it can also display the video generation model used by the video to be generated, the estimated duration of the video to be generated, and the resources consumed by the video to be generated.

[0043] In step 220, in response to a confirmation operation for the video script, a storyboard image is displayed. The storyboard image is generated based on the video script and has a first picture quality.

[0044] Here, a second confirmation control can be displayed in the video generation interface. The confirmation operation for the video script can be triggered by this second confirmation control. For example, when the user clicks the second confirmation control, a confirmation operation for the video script is triggered.

[0045] Figure 4 This is a schematic diagram illustrating the video generation interface under other circumstances. For example... Figure 3 and Figure 4 As shown, a second confirmation control 303 is displayed in the video generation interface 300. When the user clicks the second confirmation control 303, the storyboard image 304 can be displayed.

[0046] It should be understood that by displaying the video script generated based on the video generation information, users can view the video script to be used. If users are not satisfied with the video script, they can modify the content of the video script by adjusting the video generation information or modifying the content of the video script.

[0047] Upon detecting a confirmation operation on the video script, a storyboard image of first-frame quality can be displayed in the video generation interface. This storyboard image can be generated by the video generation agent based on the video script. For example, the video generation agent can generate the storyboard image based on the video script by invoking a large language model. The storyboard image can refer to converting the textual descriptions in the video script into a series of static images arranged in chronological order, used to preview the shot composition, scene content, character actions, camera movements, and pacing of the video to be generated.

[0048] It's important to note that storyboard images can refer to the visual representation of a video script, with each storyboard image corresponding to a single shot within the script. Image quality refers to the perceived sharpness, detail, color accuracy, and contrast of an image or video, used to evaluate its visual appeal and determine its overall quality. In one scenario, a storyboard image of "first-quality" might have a resolution that meets the highest standards. The final video's resolution can be "second-quality," meaning the second quality is higher than the first. In other words, the cost of the final video is higher than the cost of the storyboard images. Therefore, before generating a high-quality, high-cost video, a low-quality, low-cost storyboard image is first shown to the user. Once the user confirms the storyboard image, the high-quality, high-cost video is then generated based on it.

[0049] By viewing the first-frame quality storyboard image, users can confirm whether the character designs and scene composition meet their requirements. Once the user confirms the displayed storyboard image, video generation will then proceed.

[0050] It's worth noting that the displayed video script and the video script used to generate storyboard images can have different formats. The video script used to generate storyboard images can be a data-driven script with fields, machine-readable, and capable of being automatically fed into a large language model pipeline. This data-driven script can convert the text-based video script through structured fields, allowing the large language model to directly generate video from the data-driven script without further translation.

[0051] A video script may include one or more story segments. Each story segment may include the segment number corresponding to the story segment, the reason for splitting the story segment, the pacing used by the story segment, the planned duration of the story segment, the video generation model used by the story segment, the design assets used by the story segment, a first cue word for indicating the still image frame in the generated story segment, and a second cue word for indicating the still image frame.

[0052] In the data-driven script, the field "shot_id:2" represents the field number, meaning the second story segment in the video script. The field "split_reason":" indicates the reason for splitting, for example, "split_reason: core character appearance, asset integration required, single subject display, matching B-level standard shot." The field "pacing_tier":"B-level" indicates the pacing level, meaning the pacing level is a pre-set B-level. The field "pipeline_method":"i2v_asset"""""" indicates the video generation model used, meaning the "i2v_asset" video generation model is used to generate the story segments. The field "character_ids":["char_baogong"]" indicates the design asset used, meaning the character "char_baogong" is used. The field "gen_image_prompt":" indicates the first prompt word. For example, "gen_image_prompt":" can be a medium close-up shot using assets to construct the image. The image depicts a stern, dark-style XX sitting upright behind a courtroom desk, his head slightly lowered, his gaze intently fixed on the documents. His black official robe and red belt appear even more solemn in the dim candlelight, while gold decorations shimmer in the shadows. The background is the solemn and dignified Kaifeng Prefecture courtroom, in a 16:9 aspect ratio." The second prompt can be represented by the field "gen_video_prompt":, for example, it can be represented as "gen_video_prompt": The camera slowly zooms in. The stern, dark-style XX slowly raises his head, his sharp eyes sweeping across the empty courtroom. He is a man around 40 years old, with a deep, authoritative voice, speaking slowly and solemnly, saying: "Bring him in."

[0053] It should be understood that the data-driven script is essentially a JSON (JavaScript Object Notation) file, which specifies the type and value range of each field in the JSON file, enabling the large language model to directly understand its meaning and automatically generate videos based on the data-driven script.

[0054] In step 230, in response to a confirmation operation on the storyboard image, a video is displayed. The video is generated based on the storyboard image and has a second picture quality, which is greater than the first picture quality.

[0055] Here, the video generation interface can display a third confirmation control. The confirmation operation for the storyboard image can be triggered by this third confirmation control. For example, when the user clicks the third confirmation control, a confirmation operation for the storyboard image is triggered.

[0056] Upon detecting a confirmation action on the storyboard image, the video generation agent can invoke a large language model to generate a second-quality video based on the storyboard image. Thus, before officially generating the second-quality video, the user is first shown a first-quality storyboard image, reducing the cost of video errors.

[0057] It should be understood that the methods for generating second-frame quality video based on storyboard images can vary depending on the video generation mode. As some examples, the generated storyboard images may be stitched together to obtain video frames, which are then stitched together with generated audio to obtain the final second-frame quality video. As other examples, a motion video may be generated based on the generated storyboard images, and then the motion video and generated audio may be stitched together to obtain the final second-frame quality video.

[0058] In some cases, scene features can be extracted from confirmed storyboard images, and then a second-quality video can be generated based on these features. These scene features can include character characteristics and scene style from the storyboard images. By generating video based on character features and scene style extracted from storyboard images, consistency in subsequent video generation processes can be ensured.

[0059] In other cases, the second-quality video comprises multiple video clips generated based on two adjacent storyboard images. Accordingly, an adjusted video, generated based on storyboard images with the adjusted arrangement, can also be displayed in response to an adjustment operation that indicates an adjustment to the arrangement of the storyboard images.

[0060] For example, the adjustment operation used to indicate the order of storyboard images can be a drag operation on the storyboard images. Users can adjust the order of storyboard images by dragging them.

[0061] The second-quality video can include multiple video clips, each generated based on two adjacent storyboard images. For example, a corresponding video clip can be generated by interpolating between two adjacent storyboard images. The multiple video clips in the second-quality video can be stitched together according to the arrangement order of the storyboard images. Accordingly, the user can update the playback order of the video clips in the video by using an adjustment operation to indicate the adjustment of the arrangement order of the storyboard images.

[0062] Figure 5 This is a schematic diagram illustrating the video generation interface based on certain scenarios. For example... Figure 5 As shown, a third confirmation control 305 is displayed in the video generation interface 300. When the user clicks the third confirmation control 305, the video 306 generated based on the confirmed storyboard image 304 can be displayed.

[0063] For example, a second-quality video generated based on storyboard images can be displayed in the video generation interface. This can include a cover image or thumbnail corresponding to the second-quality video, and the video can be played when the user triggers the cover image or thumbnail.

[0064] Therefore, by responding to video generation information, displaying the video script, then responding to a confirmation operation on the video script, displaying a storyboard image of first-frame quality, and then responding to a confirmation operation on the storyboard image, displaying the video of second-frame quality (higher than the first), this not only significantly reduces initial rendering time by generating low-cost storyboard images first, but also avoids wasting expensive computing resources and time before generating the final version of the video, greatly reducing the cost of video generation errors. Secondly, by displaying the video script and storyboard images, initial script and visual references can be provided to the user, ensuring transparency and controllability in video generation. This also allows users to participate in the video creation process early in the process, enhancing user engagement and satisfaction, and preventing the final video from failing to meet user needs.

[0065] In some cases, in step 230, in response to a confirmation operation on the storyboard image, a video may be displayed, the video being generated based on a first video generation mode and the storyboard image.

[0066] Here, different video generation logics can be used to generate second-quality videos based on storyboard images under different first video generation modes. In other words, each first video generation mode can correspond to a video generation logic. After determining the first video generation mode, the second-quality video can be generated based on the storyboard images using the video generation logic corresponding to the first video generation mode.

[0067] In some cases, a first video generation mode that matches the video type can be determined from multiple second video generation modes based on the video type indicated by the video generation information.

[0068] The video type can be a semantic tag or category identifier used to classify video content, purpose, structure, or presentation. For example, video types can include short film, documentary, adventure, coming-of-age, Japanese narrative, educational, product introduction, and so on. Different video types will have different narrative logic, visual style, pacing, and audio strategies.

[0069] The video generation service can identify the video type indicated by the video generation information, and then determine a first video generation mode that matches the video type from multiple candidate second video generation modes. Different second video generation modes can correspond to different video types.

[0070] For example, the first video generation mode includes one of the following: narration mode, audio-visual synchronization mode, and image dubbing mode. Different first video generation modes correspond to different video generation logic.

[0071] In narration mode, the main narrative of the video can be conveyed through narration (also known as voice-over). In narration mode, the visual elements of the video can include relevant images, video clips, infographics, etc., to supplement or enhance the story told through narration. The video types that can be matched with narration mode include documentaries, educational videos, and product introductions.

[0072] In narration mode, the corresponding video generation logic can be: generating narration audio based on the video script; generating video clips based on the storyboard images, with the duration of the video clips controlled by the duration of the narration audio; and generating a second-quality video based on the video clips and the narration audio.

[0073] It's important to note that the video generation agent can initiate a narration generation task, which, in response, generates narration audio based on the video script. The video generation task can also initiate a video clip generation task, which, in response, generates video clips based on storyboard images. Furthermore, during the generation of video clips from storyboard images, the video generation agent can calculate the duration of the narration audio and control the duration of the video clips output by the video generation model to ensure that the video clip duration matches the narration audio duration.

[0074] As examples, the narration generation task and the video clip generation task can be executed in parallel, that is, narration audio can be generated based on the video script and video clips can be generated based on the storyboard image at the same time.

[0075] After generating the narration audio and video clips, they can be combined based on the corresponding timeline information to generate a second-quality video. Alternatively, if a subtitle file exists, the narration audio, video clips, and subtitle file can also be combined to generate a second-quality video.

[0076] The audio-visual synchronization mode emphasizes the close coordination between audio and video, meaning that the actions of characters, lip movements, scene changes, and rhythm changes in the video are all related to the audio. The video types that can be matched with this mode include digital human narration and animated dialogue.

[0077] In audio-visual synchronization mode, the corresponding video generation logic can be: generate audio based on the video script, generate video clips based on the audio and storyboard images, and then generate a second-quality video based on the video clips and audio.

[0078] For example, audio can be generated based on a video script first, and then the generated audio and storyboard images can be input into a video generation model. The audio generated based on the video script can drive the video generation model to generate the corresponding video, so that the lip movements, actions, rhythms, etc. of the characters in the generated video can be synchronized with the audio.

[0079] After generating video clips, the generated audio and video clips can be combined based on their corresponding timeline information to create a second-quality video. Alternatively, if subtitle files exist, the generated audio, video clips, and subtitle files can also be combined to generate a second-quality video.

[0080] Image-with-dubbing mode is a creative mode that uses static images as a base and adds camera movement and audio (such as voice-over, narration, dialogue, etc.) to generate short videos or audio-visual content with narrative or expressiveness. The corresponding video types for image-with-dubbing mode can be educational demonstrations or social media marketing.

[0081] In the image-to-dubbing mode, the corresponding video generation logic can be as follows: generate dubbing audio based on the video script, generate multiple still images based on the storyboard images, generate camera movement video based on the still images and the corresponding camera movement method, and generate a second-quality video based on the camera movement video and the dubbing audio.

[0082] Generating multiple static images based on storyboard images can be achieved by using the generated storyboard images as static images, or by interpolating the generated storyboard images to generate multiple static images that include the storyboard images.

[0083] Camera movement can refer to the movement trajectory of a virtual camera, and can be determined based on the semantic rhythm and / or prosodic features of the dubbing audio.

[0084] It is possible to perform 2.5D reconstruction on static images and render the reconstructed images according to the camera movement to obtain dynamic camera movement video. Then, based on the timeline information corresponding to the camera movement video and the audio narration, the audio narration and the camera movement video are time-aligned and synthesized to output a video of second-view quality. Of course, if a subtitle file exists, the generated audio narration, camera movement video, and subtitle file can also be synthesized to generate a video of second-view quality. Here, 2.5D reconstruction refers to recovering an intermediate representation containing depth information and surface orientation from a single two-dimensional static image, forming a quasi-three-dimensional structure image that lies between two and three dimensions.

[0085] In other cases, multiple second video generation modes can be displayed, and then, in response to a selection operation for a second video generation mode, the selected second video generation mode can be determined as the first video generation mode.

[0086] When the user inputs video generation information, multiple candidate second video generation modes can be displayed; similarly, when displaying a video script, or when displaying a storyboard image. Alternatively, multiple candidate second video generation modes can be displayed when the user confirms a storyboard image or video script. In other words, there is no restriction on when multiple candidate second video generation modes are displayed; it can be set according to actual needs.

[0087] The selection operation for the second video generation mode can refer to the triggering operation of the fourth confirmation control corresponding to any of the displayed second video generation modes. For example, a user can select the first video generation mode by clicking the fourth confirmation control of the second video generation mode.

[0088] For example, a narration mode, an audio-visual synchronization mode, and an image dubbing mode can be displayed. When the user selects the narration mode, the narration mode can be determined as the first video generation mode. When the user selects the image dubbing mode, the image dubbing mode can be determined as the first video generation mode.

[0089] Therefore, through the above implementation method, a first video generation mode adapted to the video type can be automatically selected, allowing video generation to proceed according to different video generation modes. In narration mode, the duration of the video frame can be dynamically adjusted based on the duration of the narration audio; in audio-visual synchronization mode, audio can be used to drive the generation of the video frame, meeting the needs of different scenarios. Of course, users can also choose the video generation mode according to their needs.

[0090] In some cases, it can also respond to video editing operations by displaying the corresponding video editing track, and then respond to adjustments made to the video editing track by displaying the adjusted video.

[0091] Here, video editing tracks include the video track, audio track, and subtitle track. These tracks describe how different types of media elements (video, audio, and subtitles) are organized, overlaid, and played synchronously along the timeline. The video track carries the video frames, containing all the video frames that change over time. The audio track carries the sound content, and the subtitle track carries text information, displaying text content related to the audio or video frames, such as dialogue subtitles, explanatory text, titles, and keywords.

[0092] Adjusting a video editing track can refer to dragging or dropping the video editing track, such as dragging or dropping any track among the video track, audio track, and subtitle track.

[0093] Figure 6 This is a schematic diagram illustrating a video editing interface under certain circumstances. For example... Figure 6 As shown, the video editing interface 600 can display video 306 as well as audio track, subtitle track and video track. Users can adjust video 306 by adjusting the audio track, subtitle track and video track.

[0094] It should be understood that, in some cases, the generated second-quality video can be considered an editable video file. For example, the generated second-quality video can be an FCPXML (Final Cut Pro XML, an open format based on XML (Extensible Markup Language) used for exporting and importing editing projects) / RP (a file used in video editing software to save video editing project information). When an editing operation on the video is detected, the second-quality video can be displayed in the video editing interface. Furthermore, the video editing interface can display the corresponding video track, audio track, and subtitle track. Users can adjust the generated video by dragging and dropping the video track, audio track, and subtitle track, thereby displaying the adjusted video.

[0095] Therefore, by displaying video editing tracks, the barriers between video generation and professional post-production video editing can be broken down, thereby better assisting creators in video production.

[0096] In some cases, in response to an adjustment operation that indicates an extension of the video duration, the extended video can be displayed, wherein the extended video is generated based on the last video frame and the video itself.

[0097] Here, a new video segment can be generated based on the last frame of the video, and the new video segment can be spliced ​​with the video to obtain a video with extended duration.

[0098] After generating a second-quality video, if the total duration of the generated video does not meet the user's needs, the user can use the adjustment operation to indicate the duration of the extended video, which allows the video generation agent to delay the total duration of the generated video.

[0099] The adjustment operation used to indicate the duration of the extended video can be a trigger operation on a displayed duration adjustment control. For example, a duration adjustment control can be displayed in the video generation interface, and the user can trigger the adjustment operation to indicate the duration of the extended video by activating the duration adjustment control.

[0100] The video generation agent can respond to adjustment operations that indicate the duration of the extended video by generating a new video segment based on the last video frame, splicing the new video segment with the previously generated video to obtain the extended video, and then displaying the extended video.

[0101] It is important to note that users can configure the required extension duration according to their needs. Accordingly, a new video segment can be generated based on the last video frame and the configured extension duration, so that the duration of the generated new video segment is consistent with the indicated extension duration. Then, the new video segment is spliced ​​with the previously generated video to obtain the extended video, which is then displayed.

[0102] Therefore, through the above implementation method, users can extend the total duration of the generated video, and the consistency of the overall video style can be maintained by generating a new video segment from the last video frame.

[0103] In some cases, after displaying a storyboard image of first-frame quality generated based on a video script, a video clip generated based on the selected storyboard image can also be displayed in response to a preview operation on the selected storyboard image, so as to adjust the selected storyboard image and / or the video script corresponding to the selected storyboard image based on the video clip.

[0104] Here, the selected storyboard image can be one or more storyboard images selected by the user. For example, the user can determine the selected storyboard image through a selection operation. After selecting the storyboard image, the user can trigger a preview operation on the selected storyboard image, which can be a trigger operation on a preview control. For example, a preview control can be displayed in the video generation interface, and when the user clicks the preview control, a preview operation on the selected storyboard image is triggered.

[0105] The video generation agent responds to a preview operation on a selected storyboard image by displaying a video clip generated based on the selected storyboard image. Thus, after displaying a first-frame-quality storyboard image generated based on the video script, the user can select one or more storyboard images as selected storyboard images. Then, through the preview operation, the user instructs the video generation agent to first generate a video clip based on the selected storyboard images, allowing the user to preview whether the generated video clip meets their requirements. If it does not meet their requirements, the user can individually adjust the selected storyboard images and / or the corresponding video script, thereby making video generation more controllable and reducing the user's trial-and-error costs.

[0106] Therefore, through the above implementation method, users can preview whether the video segments generated from the selected storyboard images meet their requirements through the preview operation, support users to adjust the storyboard images segment by segment, ensure the controllability of video generation and reduce the user's trial and error costs.

[0107] In some cases, in step 210, multiple story templates may be displayed in response to video generation information; a story script may be displayed in response to a selection operation on a story template; modification suggestions corresponding to the story script may be displayed in response to an adjustment operation on the story script; the adjusted story script may be displayed in response to a confirmation operation on the modification suggestions; and a video script generated based on the adjusted story script may be displayed in response to a confirmation operation on the adjusted story script.

[0108] Here, a story script describes the content of the story itself, including plot, characters, conflict, and emotional arc. It focuses on narrative logic, theme, character motivations, and plot development. A video script, on the other hand, transforms the story into a filmable / generable audiovisual language, encompassing shots, visuals, dialogue, sound effects, and duration, focusing on how the story is presented.

[0109] When a user inputs video generation information, multiple candidate story templates can be displayed. These candidate story templates can be determined based on the video generation information. For example, if a user inputs the video generation information "An elderly person feeds stray cats every day until winter comes and the stray cats disappear," the video generation agent can recommend multiple story templates that match the story theme corresponding to that video generation information, such as adventure, short film, coming-of-age, Japanese narrative, educational, and so on. Different types of story templates have different narrative structures.

[0110] Then, upon detecting a selection action for a story template, the story script can be displayed. The story template selected by the user is the target story template, which can be chosen by the user from multiple displayed story templates through the selection action.

[0111] The story script is generated based on the selected story template and video generation information. Users can choose a story template as the target story template, and then the video generation agent generates the corresponding story script based on the selected story template and video generation information. It should be understood that generating a story script can be understood as using the selected story template to refine the user-input video generation information, resulting in a more rigorous narrative.

[0112] Next, in response to the adjustment operation on the story script, suggested modifications to the story script are displayed. For example, the adjustment operation on the story script can be automatically triggered. For instance, after displaying the story script, the video generation agent can automatically detect logical flaws or weaknesses in the story script and display suggested fixes to provide the user with corresponding suggestions. For example, if the story script as a whole lacks significant conflict, the suggested modification could be "whether to add conflict elements to enhance the story's pacing."

[0113] The confirmation action for the modification suggestions can be a trigger action on the displayed fifth confirmation control. Upon detecting a confirmation action on the fifth confirmation control, the adjusted story script can be displayed. It should be understood that the adjusted story script is generated based on the modifications suggested in the modification suggestions.

[0114] For example, the revised story script could be: "An old man often interacts with stray cats. Even as the weather gets colder, the stray cats still come to play with the old man. One day, a blizzard blocked the road, and the stray cats disappeared. After many hardships, the old man found that the stray cats had given birth to a litter of kittens in a pipe."

[0115] After adjusting the story script, in response to a confirmation action on the adjusted story script, a video script generated based on the adjusted story script can be displayed.

[0116] It should be understood that after the story script is finalized, the user can instruct the video generation agent to generate a video script based on the adjusted story script by confirming the revised story script. This confirmation action can be a trigger action on the sixth trigger control.

[0117] In some cases, when displaying a video script generated based on the adjusted story script, users can be allowed to revert to the original story script and further optimize it. For example, if a user finds that parts of the story script's plot are difficult to visualize in the displayed video script, they can revert and modify the story script's plot.

[0118] Therefore, by providing story templates, users can avoid building a story structure from scratch, reducing the video production skills required. Furthermore, by providing modification suggestions, interactive collaboration between the video generation agent and the user can be achieved. Figure 7 This is a flowchart illustrating video generation methods under other circumstances. For example... Figure 7 As shown, the user inputs video generation information, a video script is generated based on this information, and then the video script is displayed. The system checks if the user has modified the video script; if so, it is regenerated. If the user confirms the video script, a storyboard image is generated and displayed based on the confirmed script. Upon confirmation of the storyboard image, a video clip can be generated based on it. Alternatively, if a narration mode is used, narration audio can be generated. Correspondingly, a video clip can be generated based on the narration audio and the storyboard image. Background music can also be matched. Subtitles are then added to the video clips, and the video clips with subtitles and background music are combined to generate the final video.

[0119] It should be understood that if a user uploads audio, the video generation agent can extract the text and timestamps from the audio using Automatic Speech Recognition (ASR). If the audio is generated through the video generation agent, the text and timestamps from the audio can be obtained. Subtitle files are then generated using the extracted text and timestamps.

[0120] Secondly, the video generation agent can search the music library for background music that matches the emotional tags corresponding to the video script, and automatically perform volume ducking on the background music.

[0121] During the video compositing stage, video clips, audio tracks (narration audio, background music, etc.), and subtitle tracks can be combined.

[0122] Figure 8 This is a schematic diagram of the module connections of a video generation device, provided according to certain situations. For example... Figure 8 As shown, a video generation apparatus 800 is provided, which may include: The first display module 801 is used to display a video script in response to video generation information, wherein the video script is generated based on the video generation information. The second display module 802 is configured to display a storyboard image in response to a confirmation operation on the video script. The storyboard image is generated based on the video script and has a first image quality. The third display module 803 is configured to display a video in response to a confirmation operation on the storyboard image, the video being generated based on the storyboard image, the video having a second image quality, and the second image quality being greater than the first image quality.

[0123] In some cases, the third display module 803 is used for: In response to a confirmation operation on the storyboard image, the video is displayed, the video being generated based on a first video generation mode and the storyboard image.

[0124] In some cases, the video generation apparatus 800 may further include: The first determining module is configured to determine a first video generation mode that matches the video type from a plurality of second video generation modes, based on the video type corresponding to the video to be generated indicated by the video generation information; or The fourth display module is used to display multiple second video generation modes; The second determining module is configured to, in response to the selection operation for the second video generation mode, determine the selected second video generation mode as the first video generation mode. In some cases, the first video generation mode includes one of the following: narration mode, audio-visual synchronization mode, and image dubbing mode. Different first video generation modes correspond to different video generation logics.

[0125] In some cases, the video generation apparatus 800 may further include: The fifth display module is used to display the video editing track corresponding to the video in response to the editing operation on the video. The video editing track includes a picture track, an audio track, and a subtitle track. An adjustment module is used to adjust the video in response to an adjustment operation on the video editing track and display the adjusted video.

[0126] In some cases, the video generation apparatus 800 may further include: The sixth display module is configured to display the extended video in response to an adjustment operation indicating an extension of the video duration, wherein the extended video is generated based on the last video frame of the video and the video itself.

[0127] In some cases, the video generation apparatus 800 may further include: The seventh display module is used to display a video clip in response to a preview operation on a selected storyboard image, the video clip being generated based on the selected storyboard image.

[0128] In some cases, the first display module 801 is used for: In response to video generation information, multiple story templates are displayed; In response to a selection of a story template, a story script is displayed, which is generated based on the selected story template and the video generation information; In response to the adjustment operation on the story script, the corresponding modification suggestions for the story script are displayed; In response to the confirmation action for the proposed modifications, the revised story script is displayed; In response to a confirmation action for the adjusted story script, a video script, generated based on the adjusted story script, is displayed.

[0129] Regarding the video generation device 800 in the above technical solution, the method logic executed by each functional module has been described in detail in the section on methods, and will not be repeated here.

[0130] The following is for reference. Figure 9 It shows an electronic device suitable for implementing the above-mentioned technical solution (e.g. Figure 2The diagram below shows the structure of the terminal device or server (900). The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs (Televisions), desktop computers, etc. Figure 9 The electronic device shown is merely an example and should not be construed as limiting its functionality or scope of use.

[0131] like Figure 9 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, the ROM 902, and the RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0132] Typically, the following devices can be connected to the input / output interface 905: input devices 906 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 908 including, for example, magnetic tape, hard disk, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0133] In particular, depending on certain circumstances, the processes described in the flowchart above can be implemented as computer software programs. For example, a computer program product is provided, comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. This computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a read-only memory 902. When the computer program is executed by a processing device 901, it performs the functions defined in the above-described methods.

[0134] It should be noted that the aforementioned computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In one case, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In another case, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0135] In some implementations, terminal devices and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the internet (e.g., the Internet), and end-to-end networks (e.g., ad-hoc end-to-end networks), as well as any currently known or future-developed networks.

[0136] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0137] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: display a video script generated based on the video generation information in response to video generation information; display a storyboard image generated based on the video script in response to a confirmation operation on the video script, the storyboard image having a first picture quality; and display a video generated based on the storyboard image in response to a confirmation operation on the storyboard image, the video having a second picture quality, and the second picture quality being greater than the first picture quality.

[0138] Computer program code for performing the above operations can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages, as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules mentioned above can be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the functionality of that module.

[0141] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Parts (ASSPs), Systems on Chips (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0142] In this context, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] The above description is merely illustrative and explains the technical principles employed. Those skilled in the art should understand that the scope of the technical solution is not limited to specific combinations of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features provided herein that have similar functions.

[0144] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the technical solution. Certain features described in the context of a single example can also be implemented in combination in a single example. Conversely, various features described in the context of a single example can also be implemented individually or in any suitable sub-combination in multiple examples.

[0145] Although the technical solution has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims. Regarding the aforementioned apparatus, the specific manner in which each module performs its operation has already been described in detail in the section concerning the method, and will not be elaborated upon here.

Claims

1. A video generation method, comprising: In response to video generation information, a video script is displayed, the video script being generated based on the video generation information; In response to a confirmation operation on the video script, a storyboard image is displayed, the storyboard image being generated based on the video script and having a first picture quality; In response to a confirmation operation on the storyboard image, a video is displayed, the video being generated based on the storyboard image, the video having a second picture quality, and the second picture quality being greater than the first picture quality.

2. The method according to claim 1, wherein, The method further includes: In response to an editing operation on the video, the corresponding video editing track is displayed, the video editing track including a video track, an audio track, and a subtitle track; In response to the adjustment operation on the video editing track, the video is adjusted and the adjusted video is displayed.

3. The method according to claim 1, wherein, After displaying the storyboard image in response to a confirmation operation for the video script, the method further includes: In response to a preview operation on a selected storyboard image, a video clip is displayed, the video clip being generated based on the selected storyboard image.

4. The method according to claim 1, wherein, The step of displaying the video script in response to video generation information includes: In response to video generation information, multiple story templates are displayed; In response to a selection of a story template, a story script is displayed, which is generated based on the selected story template and the video generation information; In response to the adjustment operation on the story script, the corresponding modification suggestions for the story script are displayed; In response to the confirmation action for the proposed modifications, the revised story script is displayed; In response to a confirmation action for the adjusted story script, a video script, generated based on the adjusted story script, is displayed.

5. The method according to any one of claims 1-4, wherein, The step of displaying video in response to a confirmation operation on the storyboard image includes: In response to a confirmation operation on the storyboard image, the video is displayed, the video being generated based on a first video generation mode and the storyboard image.

6. The method according to claim 5, wherein, The first video generation mode is determined through the following steps: Based on the video type corresponding to the video to be generated indicated by the video generation information, a first video generation mode that matches the video type is determined from a plurality of second video generation modes; or Displays multiple secondary video generation modes; In response to the selection operation for the second video generation mode, the selected second video generation mode is determined as the first video generation mode.

7. The method according to claim 5, wherein, The first video generation mode includes one of the following: narration mode, audio-visual synchronization mode, and image dubbing mode. Different first video generation modes correspond to different video generation logics.

8. The method according to any one of claims 1-4, wherein, The method further includes: In response to an adjustment operation instructing the duration of the video to be extended, the extended video is displayed, wherein the extended video is generated based on the last video frame of the video and the video itself.

9. A video generation apparatus, comprising: A first display module is configured to display a video script in response to video generation information, wherein the video script is generated based on the video generation information. The second display module is configured to display a storyboard image in response to a confirmation operation on the video script. The storyboard image is generated based on the video script and has a first image quality. A third display module is configured to display a video in response to a confirmation operation on the storyboard image, the video being generated based on the storyboard image, the video having a second image quality, and the second image quality being greater than the first image quality.

10. A computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method according to any one of claims 1-8.

11. An electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images