Video generation method, device, electronic device and storage medium

Abstract

This disclosure provides a video generation method, apparatus, electronic device, and storage medium, belonging to the field of multimedia technology. The method includes: displaying a video editing interface, wherein the video editing interface displays material generation controls and a video track; responding to a trigger operation on the material generation controls, using artificial intelligence technology to generate first material, the first material including at least one of image material and video material; displaying the first material in the video track; and responding to a video generation operation, generating a video based on the first material in the video track. The above technical solution deeply integrates image / video material generation functions into the video editing scenario, allowing seamless connection between material generation and video generation, thereby improving video generation efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia technology, and in particular to a video generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of multimedia technology, video, with its advantages of combining text and images and being rich and vivid, has gradually become the main way for people to obtain information and entertainment. Various video editing applications have emerged, which usually provide convenient video editing functions, enabling users to quickly create their own videos.

[0003] When creating a video, if a large amount of image or video material is needed, users typically generate the image or video material in an AI-generated application and download it to their photo album. Then, they return to the video editing application, retrieve the AI-generated image or video material from their album, and edit it to create a new video.

[0004] However, the above technical solutions require cross-application operations, meaning that users need to generate materials in an application that supports AI generation, save them to their album, and then import them into the video editing application. This process is cumbersome and disjointed, affecting the efficiency of video generation. Summary of the Invention

[0005] This disclosure provides a video generation method, apparatus, electronic device, and storage medium, which deeply integrates image / video material generation functions into the video editing scenario. Material generation and video generation can be seamlessly connected, thereby improving video generation efficiency. The technical solution of this disclosure is as follows:

[0006] According to one aspect of the embodiments of this disclosure, a video generation method is provided, comprising:

[0007] The video editing interface is displayed, which includes material generation controls and a video track.

[0008] In response to a trigger operation on the material generation control, artificial intelligence technology is used to generate a first material, which includes at least one of image material and video material;

[0009] The first footage is displayed in the video track;

[0010] In response to the video generation operation, a video is generated based on the first material in the video track.

[0011] According to another aspect of the embodiments of this disclosure, a video generation apparatus is provided, comprising:

[0012] The display unit is configured to display a video editing interface, which displays material generation controls and a video track;

[0013] The first generation unit is configured to perform a trigger operation in response to the material generation control, and generate a first material using artificial intelligence technology, wherein the first material includes at least one of image material and video material;

[0014] The display unit is also configured to display the first material in the video track;

[0015] The second generation unit is configured to perform a video generation operation in response to the video generation operation, generating a video based on the first material in the video track.

[0016] In some embodiments, the first generating unit includes:

[0017] The display subunit is configured to display the material generation panel in response to a trigger operation on the material generation control;

[0018] The acquisition subunit is configured to acquire prompt information, the prompt information including at least one of prompt text and prompt image, the prompt information being used to indicate at least one of the content and style of the material to be generated;

[0019] The display subunit is also configured to display the prompt information in the material generation panel;

[0020] The generation subunit is configured to perform a response to a confirmation generation operation, using artificial intelligence technology to generate the first material that matches the prompt information.

[0021] In some embodiments, the material generation control includes at least one of an image generation control, a video generation control, and a batch generation control, and the material generation panel includes at least one of an image generation panel, a video generation panel, and a batch generation panel.

[0022] The generation subunit is configured to perform any of the following:

[0023] In response to the confirmation generation operation in the image generation panel, based on the prompt information in the image generation panel, artificial intelligence technology is used to generate image materials that match the prompt information;

[0024] In response to the confirmation and generation operation in the video generation panel, based on the prompt information in the video generation panel, artificial intelligence technology is used to generate video material that matches the prompt information;

[0025] In response to the confirmation and generation operation in the batch generation panel, based on multiple prompts in the batch generation panel, artificial intelligence technology is used to generate image or video materials that match each of the multiple prompts.

[0026] In some embodiments, the video editing interface also displays other tracks besides the video track;

[0027] The acquisition subunit is configured to determine the prompt information based on the materials in the other tracks, provided that corresponding materials have already been added to those other tracks.

[0028] In some embodiments, the material generation panel includes at least one of an image generation panel and a video generation panel;

[0029] The acquisition subunit is configured to, when corresponding material has been added to the other tracks, determine the second material located at the position indicated by the timeline in the other tracks; and determine the prompt information based on the second material.

[0030] In some embodiments, the display unit is configured to display the first material in the video track according to the position of the second material in the other track.

[0031] In some embodiments, the prompt information includes the second material, the length of the first material is equal to the length of the second material, and the length of the material is used to represent the display duration of the material.

[0032] In some embodiments, the material generation panel is a batch generation panel;

[0033] The acquisition subunit is configured to, when corresponding materials have been added to the other tracks, determine the prompt information corresponding to each of the multiple second materials in the other tracks.

[0034] In some embodiments, the generation subunit is configured to perform a response to a confirmation generation operation, based on the prompt information corresponding to each of the multiple second materials in the other tracks, and using artificial intelligence technology to generate the first material corresponding to each prompt information.

[0035] The display unit is configured to display, in the video track, the first material corresponding to each prompt message, in the order of multiple second materials in the other tracks.

[0036] In some embodiments, the video editing interface also displays other tracks besides the video track;

[0037] The display unit is further configured to execute prompt information for each second material in the other tracks of the material generation panel, display an editing control corresponding to the prompt information; and in response to a trigger operation on the editing control, display an editing panel for the prompt information, the editing panel being used to edit the prompt information.

[0038] In some embodiments, the video editing interface displays multiple other tracks in addition to the video track;

[0039] The acquisition subunit is configured to perform a response to the selection operation of any other track in the video editing interface, and to determine the prompt information corresponding to each of the second materials in the other tracks.

[0040] In some embodiments, the display unit is further configured to perform a response to any prompt message in the material generation panel, and in response to a type selection operation of the prompt message, display the material type corresponding to the prompt message in the material generation panel;

[0041] The generation subunit is configured to perform a response to the confirmation generation operation, using artificial intelligence technology to generate the first material that matches the prompt information and conforms to the corresponding material type.

[0042] In some embodiments, the display unit is configured to perform the following actions in response to any prompt in the material generation panel: display a material selection control for the prompt in the material generation panel; display an image type option and a video type option in response to a triggering operation on the material selection control; and display the material type indicated by the selected type option in the material generation panel when either the image type option or the video type option is selected.

[0043] In some embodiments, the display unit is configured to perform an operation in response to the selection of any plurality of prompts in the media generation panel, displaying an image type option and a video type option; when either the image type option or the video type option is selected, the media type indicated by the selected type option is displayed in the media generation panel.

[0044] In some embodiments, the material generation panel displays a template selection control;

[0045] The acquisition subunit is configured to perform a trigger operation on the template selection control to display multiple text templates; and to determine the text template as the prompt message in response to an application operation on any text template.

[0046] In some embodiments, the material generation panel displays an image acquisition control;

[0047] The acquisition subunit is configured to perform a trigger operation on the image acquisition control to display multiple images; and to determine the prompt information based on the at least one image in response to a selection operation on the multiple images.

[0048] In some embodiments, the acquisition subunit is configured to perform the following actions for any one of the plurality of images: in response to a selection operation on the image, display the image in a prompt editing interface; in response to a prompt selection operation on the image in the prompt editing interface, determine a prompt area in the image, and determine the image and the area prompt information used to indicate the prompt area as the prompt information.

[0049] In some embodiments, the display unit is further configured to perform a text input operation in response to the material generation panel, displaying input prompt text in the material generation panel;

[0050] The acquisition subunit is configured to perform any of the following:

[0051] In response to a trigger operation on the image acquisition control, the plurality of images are displayed based on the prompt text in the material generation panel, wherein at least one of the content and style of the plurality of images conforms to the prompt text;

[0052] In response to a trigger operation on the image acquisition control, the plurality of images are displayed based on the prompt text in the material generation panel, and at least one of the plurality of images whose content and style conform to the prompt text is highlighted.

[0053] In some embodiments, the material generation panel includes a video generation panel, the prompt information in the video generation panel includes multiple images, and the video generation panel displays a first and last frame mode control;

[0054] The generation subunit is configured to, in response to a confirmation generation operation, when the first and last frame mode control is triggered, process two adjacent images in any order among the plurality of images in the video generation panel using an interpolation algorithm to obtain a transition image between the two images; and generate the first material based on the plurality of images in the video generation panel and the transition image of the plurality of images.

[0055] In some embodiments, the display unit is further configured to display the first material in the material generation panel;

[0056] In response to the selection of the first clip, if the timeline is in an empty area of ​​the video track, the first clip is displayed after any clip already added to the video track, where the empty area is a region where no clip exists in the video track; or...

[0057] In response to the selection of the first clip, if the timeline is in a non-idle area of ​​the video track, the first clip is displayed between the two clips closest to the timeline in the video track.

[0058] In some embodiments, the display unit is further configured to perform the following actions during the generation of the first material: displaying a minimize control in the material generation panel; minimizing the material generation panel in response to a triggering operation of the minimize control; highlighting a first target area in the video track, the first target area indicating the position to be occupied by the first material in the video track; and displaying the first material in the first target area.

[0059] In some embodiments, the first generation unit is further configured to perform a third material regeneration in response to an update operation on the first material in the video track;

[0060] The display unit is also configured to replace the first material in the video track with the third material.

[0061] In some embodiments, the first generation unit is further configured to perform, for any image material in the video track, in response to a conversion operation on the image material, generate video material based on the content and length of the image material, wherein the content of the video material is related to the content of the image material, the length of the video material conforms to a preset proportional relationship with the length of the image material, and the length of the material is used to represent the display duration of the material;

[0062] The display unit is also configured to replace the image material in the video track with the video material.

[0063] In some embodiments, the display unit is further configured to perform a video extension panel in response to an extension operation on any video clip in the video track;

[0064] The display unit is further configured to perform an extension prompt message based on the video extension panel, displaying an incremental video clip matching the extension prompt message in the video track, the extension prompt message including at least one of prompt text and prompt image, the extension prompt message indicating at least one of the content and style of the incremental video clip to be generated, the incremental video clip being located after and adjacent to the video clip.

[0065] In some embodiments, the display unit is configured to perform any of the following:

[0066] In response to a trigger operation on any video clip in the video track, a time window of the video clip is displayed in the video track. The length of the time window represents the display duration of the video clip. When the length of the time window is equal to the maximum duration of the video clip and a lengthening operation on the time window is detected, the video extension panel is displayed.

[0067] For any video clip in the video track, in response to the editing operation of the video clip, an extension control is displayed in the video editing interface, and in response to the triggering operation of the extension control, the video extension panel is displayed.

[0068] In some embodiments, the incremental video material is at least one of the following:

[0069] The incremental video material is generated based on the prompt text entered in the video extension panel;

[0070] The incremental video material is generated based on the prompt image input in the video extension panel, and the prompt image is the image located in the last frame of the video material.

[0071] In some embodiments, the display unit is configured to, during the process of generating the incremental video material based on the extended prompt information, highlight a second target area in the video track, the second target area indicating the position to be occupied by the incremental video material in the video track, the second target area being located after and adjacent to the area occupied by the video material; and displaying the incremental video material in the second target area after the incremental video material is generated.

[0072] In some embodiments, the display unit is configured to, during the process of generating the incremental video material based on the extended prompt information, display the generated images in the incremental video material frame by frame after the video material in the video track until the incremental video material is generated.

[0073] In some embodiments, the apparatus further includes:

[0074] The determining unit is configured to perform a response to a transition material generation operation, determining a fourth and a fifth material indicated by the transition material generation operation;

[0075] The first generation unit is further configured to generate transition material based on the last frame image in the fourth material and the first frame image in the fifth material when the fifth material is located after the fourth material; or, when the fifth material is located before the fourth material, generate the transition material based on the first frame image in the fourth material and the last frame image in the fifth material.

[0076] The transition material is displayed between the fourth material and the fifth material.

[0077] In some embodiments, the determining unit is configured to perform a trigger operation in response to a transition generation control in the video editing interface, displaying a transition generation panel. The transition generation panel displays a first generation option and a second generation option. The first generation option indicates the generation of a transition material between the fourth material and the material preceding the fourth material, and the second generation option indicates the generation of a transition material between the fourth material and the material following the fourth material. The fourth material is the material indicated by the timeline in the video track. When the first generation option is selected, the material preceding the fourth material is determined as the fifth material. When the second generation option is selected, the material following the fourth material is determined as the fifth material.

[0078] In some embodiments, the determining unit is configured to, in response to a triggering operation of the transition generation control in the video editing interface, display the first generation option as selected in the transition generation panel when the distance between the first and second source materials is greater than the distance between the second and third source materials; and in response to a triggering operation of the transition generation control in the video editing interface, display the second generation option as selected in the transition generation panel when the distance between the first and second source materials is less than the distance between the second and third source materials; wherein the first source material distance is the distance between the position indicated by the timeline in the video track and the source material preceding the fourth source material, and the second source material distance is the distance between the position indicated by the timeline in the video track and the source material following the fourth source material.

[0079] According to another aspect of the embodiments of this disclosure, an electronic device is provided, the electronic device comprising:

[0080] One or more processors;

[0081] Memory used to store the executable program code of the processor;

[0082] The processor is configured to execute the program code to implement the video generation method described above.

[0083] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when program code in the computer-readable storage medium is executed by a processor of an electronic device, enables the electronic device to perform the video generation method described above.

[0084] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the video generation method described above.

[0085] This disclosure provides a video generation method. During the video generation process within a video editing scenario, a material generation control is displayed in the video editing interface. By triggering the material generation control, at least one of image and video materials can be generated using artificial intelligence technology, enriching the materials used for video generation. These materials are then directly displayed in the video track, so that subsequent video generation operations can be triggered to directly generate the video based on the materials in the video track. The operation is simple and does not require switching to other applications to generate image or video materials. This method deeply integrates image / video material generation functions into the video editing scenario, allowing seamless connection between material generation and video generation, thereby improving video generation efficiency.

[0086] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0087] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0088] Figure 1 This is a schematic diagram illustrating the implementation environment of a video generation method according to an exemplary embodiment.

[0089] Figure 2 This is a flowchart illustrating a video generation method according to an exemplary embodiment.

[0090] Figure 3 This is a flowchart illustrating another video generation method according to an exemplary embodiment.

[0091] Figure 4 This is a schematic diagram illustrating a material generation control according to an exemplary embodiment.

[0092] Figure 5 This is a schematic diagram illustrating a material generation panel according to an exemplary embodiment.

[0093] Figure 6 This is a schematic diagram illustrating the determination of prompt text according to an exemplary embodiment.

[0094] Figure 7 This is a schematic diagram illustrating another way of determining prompt text according to an exemplary embodiment.

[0095] Figure 8 This is a schematic diagram illustrating the determination of a prompt image according to an exemplary embodiment.

[0096] Figure 9 This is a schematic diagram illustrating another way of determining a prompt image according to an exemplary embodiment.

[0097] Figure 10 This is a schematic diagram illustrating the determination of a prompt image according to an exemplary embodiment.

[0098] Figure 11 This is a schematic diagram illustrating a setting of material types according to an exemplary embodiment.

[0099] Figure 12 This is a schematic diagram illustrating another type of material setting according to an exemplary embodiment.

[0100] Figure 13 This is a schematic diagram illustrating the generation of video footage from the first and last frames according to an exemplary embodiment.

[0101] Figure 14 This is a schematic diagram illustrating a minimized material generation panel according to an exemplary embodiment.

[0102] Figure 15 This is a schematic diagram illustrating the selection of materials according to an exemplary embodiment.

[0103] Figure 16 This is a schematic diagram illustrating an update of image material according to an exemplary embodiment.

[0104] Figure 17 This is a schematic diagram illustrating an image-to-video conversion according to an exemplary embodiment.

[0105] Figure 18 This is a schematic diagram illustrating a triggering extension operation according to an exemplary embodiment.

[0106] Figure 19 This is a schematic diagram illustrating a video extension panel according to an exemplary embodiment.

[0107] Figure 20 This is a schematic diagram illustrating another video extension panel according to an exemplary embodiment.

[0108] Figure 21 This is a schematic diagram illustrating an incremental video display according to an exemplary embodiment.

[0109] Figure 22 This is a schematic diagram illustrating another method for displaying incremental video material according to an exemplary embodiment.

[0110] Figure 23 This is a schematic diagram illustrating a method for generating transition material based on the first and last frames, according to an exemplary embodiment.

[0111] Figure 24 This is a block diagram of a video generation apparatus according to an exemplary embodiment.

[0112] Figure 25 This is a block diagram illustrating a terminal according to an exemplary embodiment. Detailed Implementation

[0113] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0114] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0115] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the prompts involved in this disclosure were obtained with full authorization.

[0116] Figure 1 This is a schematic diagram illustrating an implementation environment of a video generation method according to an exemplary embodiment. Taking an electronic device as an example, see [example description missing]. Figure 1The implementation environment specifically includes: terminal 101 and server 102. Terminal 101 and server 102 can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0117] Terminal 101 is at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 player, MP4 player, and laptop computer. Terminal 101 runs an application that supports video generation. This application can be an editing application or a multimedia application, and this embodiment of the disclosure is not limited thereto. Users can log in to the application through terminal 101 to access the services provided by the application. For example, users can generate videos through the application on terminal 101. During the video generation process, terminal 101 can use artificial intelligence technology to generate image or video materials, and then generate a video from the generated materials.

[0118] Terminal 101 generally refers to one of a plurality of terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. For example, there may be several terminals, or dozens or hundreds of terminals, or even more. This disclosure does not limit the number of terminals or the type of device.

[0119] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 provides background services for applications that support video generation. Server 102 can connect to terminal 101 via a wireless or wired network. Server 102 can provide terminal 101 with various materials such as image materials, video materials, audio materials, and texture materials for video generation, so that terminal 101 can subsequently generate videos based on these materials. In some embodiments, the number of servers may be more or fewer, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0120] Figure 2 This is a flowchart illustrating a video generation method according to an exemplary embodiment, see [link to flowchart]. Figure 2 The video generation method, when applied to a terminal, includes the following steps:

[0121] In step 201, the terminal displays a video editing interface, which includes material generation controls and a video track.

[0122] In this embodiment of the disclosure, during the video generation process, the terminal displays a video editing interface so that the video can be generated subsequently through the video editing interface. The terminal displays a material generation control and a video track in the video editing interface. The material generation control is used to provide at least one type of material, including image material and video material generated using artificial intelligence technology. This embodiment of the disclosure does not limit the display style or position of the material generation control. The video track is used to display the materials used in the video generation process, which include at least one type of image material and video material.

[0123] In step 202, in response to the triggering operation of the material generation control, the terminal uses artificial intelligence technology to generate first material, which includes at least one of image material and video material.

[0124] In this embodiment of the disclosure, when the material generation control is triggered, the terminal uses artificial intelligence technology to generate first material. The first material may be only image material, or only video material, or may include both image material and video material. This embodiment of the disclosure does not limit the type and quantity of materials in the first material. The process of generating materials using artificial intelligence technology can be implemented based on an AI compiler.

[0125] In step 203, the terminal displays the first footage in the video track.

[0126] In this embodiment of the disclosure, after generating the first material, the terminal displays the first material in the video track. The essence of "displaying the first material" is displaying multiple frames of the first material. The number of images in the first material is positively correlated with the display duration of the first material.

[0127] The video track supports media editing. That is, the media in the video track can be edited during the video generation process. Users can edit the display duration or display effects of the media through the video track. The edited media will also be displayed in the video track. This embodiment does not limit whether the first media is edited.

[0128] In step 204, in response to the video generation operation, the terminal generates a video based on the first material in the video track.

[0129] In this embodiment of the disclosure, when a video generation operation is triggered, the terminal generates a video based on the first material in the video track. The video includes the first material. The display duration of the first material in the video is equal to the display duration of the first material in the video track. The material in the video track is referred to as the first material.

[0130] This disclosure provides a video generation method. During the video generation process within a video editing scenario, a material generation control is displayed in the video editing interface. By triggering the material generation control, at least one of image and video materials can be generated using artificial intelligence technology, enriching the materials used for video generation. These materials are then directly displayed in the video track, so that subsequent video generation operations can be triggered to directly generate the video based on the materials in the video track. The operation is simple and does not require switching to other applications to generate image or video materials. This method deeply integrates image / video material generation functions into the video editing scenario, allowing seamless connection between material generation and video generation, thereby improving video generation efficiency.

[0131] In some embodiments, in response to a triggering operation on the material generation control, artificial intelligence technology is used to generate first material, including:

[0132] In response to a trigger operation on the material generation control, the material generation panel is displayed;

[0133] Obtain prompt information, which includes at least one of prompt text and prompt image, and the prompt information is used to indicate at least one of the content and style of the material to be generated;

[0134] Display a prompt message in the material generation panel;

[0135] In response to the confirmation of the generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information.

[0136] The solution provided in this embodiment can acquire at least one of prompt text and prompt image as prompt information during the material generation process, so that the first material can be generated based on the prompt information. This not only enriches the material generation method, but also ensures that at least one of the content and style of the generated first material conforms to the prompt information. Since the prompt information is information actively acquired by the user and can be displayed in the material generation panel, the prompt information can accurately reflect the user's needs for the material, thereby ensuring that the generated first material meets the user's needs and improving the accuracy of material generation.

[0137] In some embodiments, the material generation control includes at least one of an image generation control, a video generation control, and a batch generation control, and the material generation panel includes at least one of an image generation panel, a video generation panel, and a batch generation panel.

[0138] In response to the confirmation generation operation, artificial intelligence technology is used to generate first material that matches the prompt message, including any of the following:

[0139] In response to the confirmation action in the image generation panel, based on the prompts in the image generation panel, artificial intelligence technology is used to generate image materials that match the prompts.

[0140] In response to the confirmation action in the video generation panel, and based on the prompts in the video generation panel, artificial intelligence technology is used to generate video footage that matches the prompts.

[0141] In response to the confirmation of generation in the batch generation panel, based on multiple prompts in the batch generation panel, artificial intelligence technology is used to generate image or video materials that match each of the multiple prompts.

[0142] The solution provided in this disclosure provides users with at least one of image generation control, video generation control, and batch generation control, so that users can obtain prompt information in the corresponding generation panel through the corresponding generation control according to their needs, in order to generate the required image materials, video materials, or batch materials. This not only enriches the material generation methods, but also makes the operation simple, allowing users to obtain the image materials, video materials, or batch materials they need, thereby improving the efficiency of material acquisition.

[0143] In some embodiments, the video editing interface also displays tracks other than the video track;

[0144] Get the prompt information, including:

[0145] If the corresponding material has already been added to other tracks, determine the prompt message based on the material in those other tracks.

[0146] The solution provided in this disclosure, during the generation of image or video materials, can determine the prompt information based on the materials already added in other tracks of the video editing interface. This ensures that the image or video materials generated based on the prompt information match the materials already added in other tracks. Since the materials already added in other tracks are materials actively added by the user, they can accurately reflect the video effect desired by the user (including at least one of content and style). Therefore, it ensures that the generated image or video materials meet the user's desired video effect, that is, meet the user's intent. This not only reduces the probability of the user editing the image or video materials, which is beneficial to improving video generation efficiency, but also helps to improve the quality of the subsequently generated video.

[0147] In some embodiments, the material generation panel includes at least one of an image generation panel and a video generation panel;

[0148] If corresponding materials have already been added to other tracks, determine the prompt message based on the materials in those other tracks, including:

[0149] If corresponding material has already been added to other tracks, determine the second material in that position from the other tracks based on the position indicated by the timeline in those other tracks;

[0150] Based on the second source material, determine the prompt message.

[0151] The solution provided in this disclosure, during the generation of image or video materials, utilizes a timeline to indicate the position of the image or video material to be generated within the video track. Based on the second material indicated by the timeline in other tracks, prompting information is determined, ensuring that the subsequently generated image or video materials based on the prompting information match the second material. Since the materials indicated by the timeline in each track are simultaneously displayed in the subsequently generated video, it ensures that the image or video material matching the second material is displayed simultaneously in the video, conforming to user intent and improving the quality of the subsequently generated video.

[0152] In some embodiments, displaying first footage in a video track includes:

[0153] Based on the position of the second clip in other tracks, the first clip is displayed in the video track.

[0154] The solution provided in this disclosure can display the first material according to the position of the second material in other tracks, achieving the purpose of automatically aligning the matching first and second materials. Since the materials aligned between the tracks will be displayed simultaneously in the subsequently generated video, it ensures that the first material matching the second material is displayed in the video at the same time as the second material. This not only meets the user's intention and eliminates the need for the user to manually align the matching materials between the tracks, reducing user operations and improving video generation efficiency, but also helps to improve the quality of the subsequently generated video.

[0155] In some embodiments, the prompt information includes a second material, the length of which is equal to the length of the second material, and the length of the material is used to indicate the display duration of the material.

[0156] The solution provided in this embodiment ensures that the display duration of the first material matching the second material is the same as that of the second material, thereby ensuring that the first material and the second material are displayed and disappear simultaneously in the subsequently generated video. This guarantees that the materials displayed at various time points in the subsequently generated video are matched, which is beneficial to improving the quality of the video.

[0157] In some embodiments, the material generation panel is a batch generation panel;

[0158] If corresponding materials have already been added to other tracks, determine the prompt message based on the materials in those other tracks, including:

[0159] If corresponding materials have already been added to other tracks, determine the corresponding prompt information for each of the multiple second materials based on the multiple second materials in other tracks.

[0160] The solution provided in this embodiment allows for the determination of corresponding prompt information based on multiple second materials in other tracks when multiple second materials have already been added. This enables the generation of corresponding first materials based on each prompt information in a single operation, eliminating the need for users to generate corresponding first materials for each second material. This simplifies the operation and improves material generation efficiency.

[0161] In some embodiments, in response to a confirmation generation operation, artificial intelligence technology is used to generate first material that matches the prompt information, including:

[0162] In response to the confirmation of the generation operation, based on the prompt information corresponding to each of the multiple second materials in other tracks, artificial intelligence technology is used to generate the first material corresponding to each prompt information.

[0163] The first footage is displayed on the video track, including:

[0164] Following the order of multiple second clips in other tracks, the first clip corresponding to each prompt message is displayed in the video track.

[0165] The solution provided in this embodiment can not only generate corresponding first materials based on the prompts for multiple second materials in other tracks, thus generating multiple first materials at once and improving material generation efficiency; but also display the generated multiple first materials in the order of the corresponding multiple second materials, making the matching first and second materials relatively aligned, thereby ensuring that the matching materials are displayed in the video simultaneously. This not only meets the user's intention but also eliminates the need for the user to manually align the matching materials between tracks, reducing user operations and improving video generation efficiency and video quality.

[0166] In some embodiments, the video editing interface also displays tracks other than the video track;

[0167] The method also includes:

[0168] For the prompt information corresponding to each second material in other tracks in the material generation panel, display the corresponding editing control for the prompt information;

[0169] In response to a triggering operation on the editing control, an editing panel is displayed to display prompt messages. The editing panel is used to edit the prompt messages.

[0170] The solution provided in this embodiment allows users to edit the prompts displayed in the material generation panel using corresponding editing controls. This ensures that the prompts meet the user's intent, improves the accuracy of subsequent video generation, and results in a video that meets the user's intent.

[0171] In some embodiments, the video editing interface displays multiple other tracks in addition to the video track;

[0172] Get the prompt information, including:

[0173] In response to the selection operation of any other track in the video editing interface, the prompt information corresponding to each second material in the other track is determined respectively.

[0174] The solution provided in this disclosure allows users to select any other track in the video editing interface during the generation of image or video materials. The prompt information is determined by the materials in the selected other track, ensuring that the prompt information matches the user's intent. This improves the accuracy of subsequent material generation, resulting in materials that match the user's intent and reducing the probability of the user editing the materials, thereby improving video generation efficiency.

[0175] In some embodiments, the method further includes:

[0176] For any prompt message in the material generation panel, in response to the type selection operation of the prompt message, the material type corresponding to the prompt message is displayed in the material generation panel;

[0177] In response to the confirmation and generation operation, artificial intelligence technology is used to generate first-level material that matches the prompt information, including:

[0178] In response to the confirmation of the generation operation, artificial intelligence technology is used to generate a first material that matches the prompt information and conforms to the corresponding material type.

[0179] The solution provided in this embodiment allows users to select the material type of the material to be generated from each prompt in the material generation panel before generating the material. This ensures that the generated material type meets the user's needs, improves the accuracy of subsequent video generation, and produces a video that meets the user's intent.

[0180] In some embodiments, for any prompt message in the material generation panel, in response to a type selection operation for the prompt message, the material type corresponding to the prompt message is displayed in the material generation panel, including:

[0181] For any prompt message in the material generation panel, display the material selection control for the prompt message in the material generation panel;

[0182] In response to a trigger action on the media selection control, display image type options and video type options;

[0183] When either the image type option or the video type option is selected, the media type indicated by the selected type option is displayed in the media generation panel.

[0184] The solution provided in this embodiment offers a corresponding material selection control for each prompt in the material generation panel, allowing users to independently set the material type for each prompt. This not only simplifies operation but also improves the flexibility of material type settings. Furthermore, it intuitively displays the selected material type to the user, enhancing the utilization rate and information transmission efficiency of the material generation panel, and enabling users to promptly adjust material types that do not meet their intentions.

[0185] In some embodiments, for any prompt message in the material generation panel, in response to a type selection operation for the prompt message, the material type corresponding to the prompt message is displayed in the material generation panel, including:

[0186] In response to the selection of any number of prompts in the media generation panel, display image type options and video type options;

[0187] When either the image type option or the video type option is selected, the media type indicated by the selected type option is displayed in the media generation panel.

[0188] The solution provided in this embodiment can set multiple prompt message material types at once, which is simple to operate and can improve the efficiency of setting multiple prompt message material types, thereby improving the efficiency of material generation.

[0189] In some embodiments, a template selection control is displayed in the material generation panel;

[0190] Get the prompt information, including:

[0191] In response to a trigger action on the template selection control, display multiple text templates;

[0192] In response to any operation applied to a text template, the text template is determined as the prompt message.

[0193] The solution provided in this disclosure provides multiple text templates for users to choose from during the text-based material generation process. This not only enriches the prompt text but also eliminates the need for users to manually input prompt text, simplifying the operation and improving the efficiency of material generation.

[0194] In some embodiments, the material generation panel displays an image acquisition control;

[0195] Get the prompt information, including:

[0196] In response to a trigger action on the image acquisition control, display multiple images;

[0197] In response to the selection of at least one of a plurality of images, a prompt message is determined based on at least one image.

[0198] The solution provided in this disclosure can acquire at least one image as prompt information to generate materials. Compared with prompt text, image prompts provide more information and can more accurately reflect the user's requirements for materials, making the materials generated based on images more in line with the user's intentions, i.e., with higher accuracy.

[0199] In some embodiments, in response to a selection operation of at least one image among a plurality of images, a prompt message is determined based on at least one image, including:

[0200] For any one of the multiple images, in response to the selection of the image, the image is displayed in the prompt editing interface;

[0201] In response to the selection of an image in the prompt editing interface, the prompt area in the image is determined, and the image and the prompt information used to indicate the prompt area are defined as the prompt information.

[0202] The solution provided in this disclosure allows for the selection of a prompt area in the image before generating material based on the image. This not only more accurately indicates the user's requirements for the material, making the image-generated material more in line with the user's intent (i.e., higher accuracy), but also improves the flexibility of image-based material generation compared to methods that generate material based on the entire image. This allows for the flexible selection of effective information in the image to generate material.

[0203] In some embodiments, the method further includes:

[0204] In response to text input in the media generation panel, display input prompt text in the media generation panel;

[0205] In response to a triggering action on the image acquisition control, display multiple images, including any of the following:

[0206] In response to a trigger operation on the image acquisition control, multiple images are displayed based on the prompt text in the material generation panel, wherein at least one of the content and style of the multiple images conforms to the prompt text;

[0207] In response to a triggering operation on the image acquisition control, multiple images are displayed based on the prompt text in the material generation panel, and at least one image among the multiple images whose content and style match the prompt text is highlighted.

[0208] The solution provided in this disclosure can provide users with images that match the prompt text input by the user, so that subsequent material can be generated quickly based on the images without the user having to manually search for the desired images, thereby improving the efficiency of material generation.

[0209] In some embodiments, the material generation panel includes a video generation panel, the prompts in the video generation panel include multiple images, and the video generation panel displays a first and last frame mode control.

[0210] In response to the confirmation and generation operation, artificial intelligence technology is used to generate first-level material that matches the prompt information, including:

[0211] When the first and last frame mode control is triggered, in response to the confirmation of generation operation, for any two adjacent images in the multiple images in the video generation panel, an interpolation algorithm is used to process the two images to obtain the transition image between the two images.

[0212] The first source material is generated based on multiple images in the video generation panel and the transition images between those images.

[0213] The solution provided in this embodiment of the present disclosure, in the process of generating video material based on images, can perform interpolation processing on multiple images used for video generation through the first and last frame mode control, to obtain transition images between multiple images, making the transition between multiple images in the subsequently generated video more natural and improving the quality of the video.

[0214] In some embodiments, the method further includes:

[0215] The first material is displayed in the material generation panel;

[0216] The first footage is displayed on the video track, including:

[0217] In response to the selection of the first clip, if the timeline is in an empty area of ​​the video track, the first clip is displayed after any clips already added to the video track; the empty area is the region in the video track where no clips exist. Alternatively,

[0218] In response to the selection of the first clip, if the timeline is in a non-idle area of ​​the video track, the first clip is displayed between the two clips closest to the timeline in the video track.

[0219] The solution provided in this disclosure embodiment can display the generated first material in the material generation panel for user selection. When the user selects the first material, the first material can be displayed in the video track according to the position of the timeline. For example, the first material can be displayed after the materials already added in the video track, or the first material can be displayed between the two materials closest to the timeline, thus enriching the ways of displaying materials in the video track.

[0220] In some embodiments, the method further includes:

[0221] During the generation of the first material, a minimized control is displayed in the material generation panel;

[0222] In response to a trigger action on the minimize control, minimize the asset generation panel;

[0223] Highlight the first target area in the video track. The first target area is used to indicate the position that the first clip will occupy in the video track.

[0224] The first footage is displayed on the video track, including:

[0225] Display the first material in the first target area.

[0226] The solution provided in this embodiment allows for minimizing the media generation panel via a minimize control during the media generation process. This eliminates the need to wait for media generation to complete, enabling other operations to be performed in the video editing interface. For example, during the generation of the first media, additional prompts can be obtained to prepare for the generation of the next media, or existing media in the video editing interface can be edited, thus improving video generation efficiency. Furthermore, after minimizing the media generation panel, the first target area can be highlighted in the video track, intuitively showing the user the position occupied by the first media being generated, enhancing the user's sense of control over media generation and improving the user experience.

[0227] In some embodiments, the method further includes:

[0228] In response to an update operation on the first clip in the video track, a third clip is regenerated;

[0229] Replace the first clip in the video track with the third clip.

[0230] The solution provided in this disclosure allows for the regeneration and replacement of existing footage in a video track using artificial intelligence technology. This facilitates adjustments to unsatisfactory footage, aligning with user intent.

[0231] In some embodiments, the method further includes:

[0232] For any image clip in the video track, in response to the conversion operation of the image clip, a video clip is generated based on the content and length of the image clip. The content of the video clip is related to the content of the image clip, and the length of the video clip conforms to a preset ratio with the length of the image clip. The length of the clip is used to represent the display duration of the clip.

[0233] Replace the image footage in the video track with video footage.

[0234] The solution provided in this disclosure can convert existing image materials in a video track into video materials with relevant content and corresponding display duration, increasing the dynamic changes of the material content, making the material content more visually appealing, and improving the quality of subsequently generated videos; moreover, video materials can be directly generated based on image materials, without requiring the user to obtain prompt information again to generate video materials, which is simple to operate and can improve the efficiency of video material generation.

[0235] In some embodiments, the method further includes:

[0236] In response to an extension operation on any video clip in the video track, the video extension panel is displayed;

[0237] Based on the extension prompt information in the video extension panel, incremental video footage matching the extension prompt information is displayed in the video track. The extension prompt information includes at least one of prompt text and prompt image. The extension prompt information is used to indicate at least one of the content and style of the incremental video footage to be generated. The incremental video footage is located after and adjacent to the video footage.

[0238] The solution provided in this embodiment allows for the extension of any video clip in a video track without needing to regenerate the complete video clip based on the prompt information used in the video clip. Instead, the extended portion (i.e., incremental video clip) is generated by extending the instruction information, and then connected to the original video clip. Compared to regenerating the complete video clip, the incremental video clip generates less data and can be generated more quickly, thereby improving the efficiency of extending the video.

[0239] In some embodiments, in response to an extension operation on any video clip in a video track, a video extension panel is displayed, including any of the following:

[0240] In response to a trigger operation on any video clip in the video track, a time window of the video clip is displayed in the video track. The length of the time window represents the display duration of the video clip. When the length of the time window is equal to the maximum duration of the video clip and a time window stretching operation is detected, a video extension panel is displayed.

[0241] For any video clip in the video track, in response to editing operations on the video clip, an extension control is displayed in the video editing interface, and in response to triggering the extension control, a video extension panel is displayed.

[0242] The solution provided in this disclosure can extend the video clip's time window. If the time window continues to extend even when it is at its maximum length, a video extension panel can be displayed to extend the video clip. Alternatively, the video extension panel can be displayed by triggering the extension control in the video editing interface to extend the video clip. This not only enriches the ways to extend video clips but also simplifies the operation and improves the efficiency of extending videos.

[0243] In some embodiments, the incremental video material is at least one of the following:

[0244] Incremental video footage is generated based on the prompt text entered in the video extension panel;

[0245] The incremental video clip is generated based on the prompt image entered in the video extension panel. The prompt image is the image located in the last frame of the video clip.

[0246] The solution provided in this disclosure can generate incremental video material by inputting prompt text, or by generating incremental video material based on the last frame of the video material to be extended. This not only enriches the generation methods of incremental video material, but also ensures that the incremental video material matches the video material to be extended by generating incremental video material based on the last frame of the video material to be extended, which is beneficial to improving the quality of the video material.

[0247] In some embodiments, based on the extension prompt information in the video extension panel, incremental video footage matching the extension prompt information is displayed in the video track, including:

[0248] During the process of generating incremental video material based on extended prompt information, a second target area is highlighted in the video track. The second target area is used to indicate the position to be occupied by the incremental video material in the video track. The second target area is located after the area occupied by the video material and is adjacent to the area occupied by the video material.

[0249] After generating incremental video footage, the incremental video footage is displayed in the second target area.

[0250] The solution provided in this disclosure can highlight the second target area in the video track during the generation of incremental video material, intuitively showing the user the position occupied by the incremental video material being generated. This not only improves the utilization rate and information transmission efficiency of the video editing interface, but also enhances the user's sense of control over the material generation, thus improving the user experience.

[0251] In some embodiments, based on the extension prompt information in the video extension panel, incremental video footage matching the extension prompt information is displayed in the video track, including:

[0252] During the process of generating incremental video footage based on extended prompts, images generated in the incremental video footage are displayed frame by frame after the video footage in the video track until the incremental video footage is generated.

[0253] The solution provided in this disclosure can display the generated images in the incremental video material frame by frame after the video material in the video track during the generation of incremental video material. This intuitively shows the user the generation progress of the incremental video material, which not only improves the utilization rate and information transmission efficiency of the video editing interface, but also enhances the user's sense of control over the material generation, thus improving the user experience.

[0254] In some embodiments, the method further includes:

[0255] In response to the transition material generation operation, determine the fourth and fifth materials indicated by the transition material generation operation;

[0256] If the fifth clip is located after the fourth clip, a transition clip is generated based on the last frame image of the fourth clip and the first frame image of the fifth clip; or, if the fifth clip is located before the fourth clip, a transition clip is generated based on the first frame image of the fourth clip and the last frame image of the fifth clip.

[0257] A transitional element is displayed between the fourth and fifth elements.

[0258] The solution provided in this embodiment can generate a transition material between any two adjacent materials in a video track and display the transition material between the two materials, making the transition between multiple materials in the subsequently generated video more natural and improving the quality of the video.

[0259] In some embodiments, in response to a transition material generation operation, determining the fourth and fifth materials indicated by the transition material generation operation includes:

[0260] In response to the triggering operation of the transition generation control in the video editing interface, the transition generation panel is displayed. The transition generation panel displays a first generation option and a second generation option. The first generation option is used to indicate the generation of the transition material between the fourth material and the material before the fourth material, and the second generation option is used to indicate the generation of the transition material between the fourth material and the material after the fourth material. The fourth material is the material indicated by the time progress line in the video track.

[0261] If the first generation option is selected, the material preceding the fourth material will be designated as the fifth material;

[0262] If the second generation option is selected, the material following the fourth material will be designated as the fifth material.

[0263] The solution provided in this disclosure, by displaying a first generation option and a second generation option in the transition generation panel, allows the user to choose whether to generate transition material between the previous material and the next material during the generation process. This makes the generation of transition material more flexible and controllable, resulting in transition material that meets the user's intentions. This reduces the probability of the user editing the material, thereby improving video generation efficiency.

[0264] In some embodiments, in response to a triggering operation on a transition generation control in the video editing interface, a transition generation panel is displayed, including:

[0265] When the distance between the first source material and the second source material is greater than the distance between the second source material and the transition generation control in the video editing interface is triggered, the first generation option is displayed as selected in the transition generation panel.

[0266] When the distance between the first and second clips is less than the distance between the second clips, in response to the triggering operation of the transition generation control in the video editing interface, the second generation option is displayed as selected in the transition generation panel;

[0267] The first clip distance is the distance between the position indicated by the timeline on the video track and the clip preceding the fourth clip, and the second clip distance is the distance between the position indicated by the timeline on the video track and the clip following the fourth clip.

[0268] The solution provided in this disclosure automatically selects the generation option based on the position indicated by the timeline in the video track and the positional relationship (i.e., distance) between the footage in the video track and the timeline. This eliminates the need for manual selection by the user, simplifies the operation, and improves the efficiency of footage generation.

[0269] The above Figure 2The diagram shown is merely the basic process of this disclosure. The following section will further elaborate on the solution provided in this disclosure based on a specific implementation method. Figure 3 This is a flowchart illustrating another video generation method according to an exemplary embodiment. Taking an electronic device provided as a terminal as an example, see [link to example]. Figure 3 The method includes:

[0270] In step 301, the terminal displays a video editing interface, which includes material generation controls and a video track.

[0271] In this embodiment of the disclosure, during the video generation process, the terminal displays a material generation control and a video track in the video editing interface. The user can use the material generation control to instruct the terminal to generate at least one type of image or video material using artificial intelligence technology for subsequent video production. The video track displays all the image or video materials used in the video generation process to the user.

[0272] This disclosure does not limit the timing of the display of the material generation control. Optionally, the material generation control can be permanently displayed in a fixed position in the video editing interface. Alternatively, the material generation control can also be displayed in the video editing interface when a certain condition is met. Optionally, the video editing interface displays other tracks besides the video track. Other tracks may include audio tracks, subtitle tracks, sticker tracks, etc., and this disclosure does not limit this. The material generation control may only be displayed in the video editing interface when the number of materials added to other tracks reaches a preset quantity. For example, when the number of subtitles added to the subtitle track reaches a preset quantity, the terminal displays the material generation control in the video editing interface.

[0273] Since video generation requires aggregating materials from all tracks, corresponding image or video materials should be added to the video track if materials exist in other tracks. This prevents a situation where the generated video contains only other materials (e.g., subtitles, audio) at a certain point in time, resulting in a black screen. When there are many materials in other tracks, a material generation control can be displayed to prompt the user that artificial intelligence technology can be used to quickly generate corresponding images for materials in other tracks, thus improving video generation efficiency. If there are no materials in other tracks, the material generation control can be hidden to save runtime resources.

[0274] This public embodiment does not limit the display location of the media generation control. For example, the media generation control can be directly displayed in the operation panel of the video editing interface; or, the media generation control can be displayed on the video track; or, the media generation control can be displayed in a prompt window. This prompt window is used to inform the user that the video may have a black screen and to instruct the user to generate media using the media generation control.

[0275] For example, Figure 4 This is a schematic diagram illustrating a material generation control according to an exemplary embodiment. See also Figure 4 The video editing interface includes a preview area 401 and an operation area 402. The operation area 402 displays multiple material tracks, including a video track 403, audio track 404, subtitle track 405, and sticker track 406, as well as an operation panel 407. The video track 403 displays image or video materials used in the video generation process. The preview area 401 previews the material indicated by the timeline 408 on the video track 403. See also... Figure 4 In (a) of this diagram, the material generation control 409 is fixedly displayed in the operation panel 407. See also... Figure 4 In (b) of this example, in a subtitle editing scenario, the source material generation control 409 can be displayed on video track 403. See also... Figure 4 In case of a black screen appearing in preview area 401 (c), the terminal displays a prompt window 410 in the video editing interface to inform the user of the black screen and instruct the user to generate footage using the footage generation control. The footage generation control 409 is displayed in the prompt window 410.

[0276] In step 302, in response to the triggering operation of the material generation control, the terminal displays the material generation panel.

[0277] In this embodiment of the disclosure, when the material generation control is triggered, the terminal displays a material generation panel in the video editing interface. The material generation control may include at least one of an image generation control, a video generation control, and a batch generation control. The material generation panel may include at least one of an image generation panel, a video generation panel, and a batch generation panel. The terminal can generate image materials through the image generation panel, generate video materials through the video generation panel, and generate multiple materials at once through the batch generation panel. The multiple materials include at least one of image materials and video materials.

[0278] For example, Figure 5 This is a schematic diagram illustrating a material generation panel according to an exemplary embodiment. See also: Figure 5The terminal displays a material generation control 502 in the operation panel 501 of the video editing interface. When the material generation control 502 is triggered, the terminal displays at least one secondary generation control in the operation panel 501, including an image generation control 503, a video generation control 504, and a batch generation control 505. When the image generation control 503 is triggered, the terminal displays an image generation panel 506 in the video editing interface. Users can input keywords (prompt text), prompt images, material style, aspect ratio, and other prompt information in the image generation panel 506 so that the terminal can generate image materials based on the prompt information in the image generation panel 506. When the video generation control 504 is triggered, the terminal displays a video generation panel 507 in the video editing interface. Users can input graphic descriptions (prompt text), prompt images, material instructions, camera movement, duration, and other prompt information in the video generation panel 507 so that the terminal can generate video materials based on the prompt information in the image generation panel 507. When the batch generation control 505 is triggered, the terminal displays a batch generation panel 508 in the video editing interface. Users can enter prompts and the type of material (image or video) to be generated in the batch generation panel 508, so that the terminal can generate at least one of the image material and video material based on the information in the batch generation panel 508.

[0279] In step 303, the terminal obtains a prompt message and displays the prompt message in the material generation panel. The prompt message includes at least one of prompt text and prompt image, and the prompt message is used to indicate at least one of the content and style of the material to be generated.

[0280] In this embodiment of the disclosure, the prompt information used to generate materials includes at least one of prompt text and prompt image. The prompt information can be information manually entered by the user or selected from multiple candidate information; this embodiment of the disclosure does not limit this. After obtaining the prompt information, the terminal displays the prompt information in the material generation panel, intuitively showing the user the prompt information used for subsequent material generation.

[0281] In some embodiments, the prompt information includes prompt text. A template selection control is displayed in the material generation panel. The prompt text may be selected by the user from multiple text templates. Accordingly, the process of the terminal obtaining the prompt information includes: in response to a trigger operation on the template selection control, the terminal displays multiple text templates. Then, in response to an application operation on any text template, the terminal determines the text template as the prompt information. The solution provided by the embodiments of this disclosure can provide multiple text templates for the user to choose from during the process of generating materials based on text. This not only enriches the prompt text but also eliminates the need for the user to manually input the prompt text, simplifying the operation and improving the efficiency of material generation.

[0282] For example, Figure 6 A schematic diagram illustrating the determination of prompt text according to an exemplary embodiment is shown. See also Figure 6 During the process of generating image materials based on text, a template selection control 602 is displayed in the image generation panel 601. When the template selection control 602 is triggered, the terminal displays multiple text templates. These text templates can be displayed according to their respective styles. Due to screen size limitations, Figure 6 Only the first few text templates are shown as examples. For any text template, the terminal can display an image example corresponding to that text template to demonstrate to the user the style of the image generated based on that text template. When the application control 603 of text template 2 is triggered, the terminal displays text template 2 in the prompt box 604 in the image generation panel 601. Alternatively, the user can directly enter prompt text in the prompt box 604.

[0283] For example, Figure 7 This is an illustration of another method for determining prompt text, according to an exemplary embodiment. See also Figure 7 During the process of generating video footage from text, the video generation panel 701 displays two options: "Image-based video" and "Text-based video." When the "Text-based video" option is selected, the terminal displays a prompt box 702 in the video generation panel 701. In response to text input in the prompt box 702, the terminal displays the entered prompt text in the prompt box 702. This prompt text can also come from multiple text templates, similar to... Figure 6 The method shown will not be elaborated further here. The video generation panel 701 also displays quality options such as "High Performance" and "High Performance" to allow users to specify the quality of the video footage. The video generation panel 701 may also display duration selection controls 703 and aspect ratio selection controls 704 to allow users to set the duration and aspect ratio of the video footage.

[0284] In some embodiments, the prompt information includes a prompt image. An image acquisition control is displayed in the material generation panel. The prompt image may be selected by the user from multiple images. Accordingly, the process of the terminal acquiring the prompt information includes: in response to a trigger operation on the image acquisition control, the terminal displays multiple images. Then, in response to a selection operation on at least one image among the multiple images, the terminal determines the prompt information based on at least one image. The solution provided by the embodiments of this disclosure can acquire at least one image as prompt information to generate material. Compared with prompt text, image prompts provide more information and can more accurately reflect the user's requirements for the material, making the material generated based on the image more in line with the user's intent, i.e., higher accuracy.

[0285] In this embodiment, the terminal can use the selected entire image as the prompt information, or it can use a local area of ​​the image as the prompt information; this embodiment does not limit the specific choice. Optionally, during the process of determining the prompt information, the user can select an image as a local area for the prompt. Accordingly, the process of the terminal determining the prompt information based on at least one image includes: for any image among multiple images, in response to a selection operation on the image, the terminal displays the image in the prompt editing interface. In response to a prompt selection operation on the image in the prompt editing interface, the terminal determines the prompt area in the image, and determines the image and the prompt information for the area used to indicate the prompt area as the prompt information. The solution provided by this embodiment allows selection of a prompt area in the image before generating material based on the image. This not only more accurately indicates the user's requirements for the material, making the material generated based on the image more in line with the user's intention (i.e., higher accuracy), but also, compared to the solution of generating material based on the entire image, generating material through a specific area of ​​the image improves the flexibility of generating material based on the image, thereby allowing for flexible selection of effective information in the image to generate material.

[0286] For example, Figure 8 This is a schematic diagram illustrating the determination of a prompt image according to an exemplary embodiment. See also: Figure 8 During the image generation process, an image acquisition control 802 is displayed in the image generation panel 801. When the image acquisition control 802 is triggered, the terminal displays multiple images. When image 1 is selected, the terminal displays image 1 in the prompt editing interface. The prompt editing interface also displays multiple area options, such as "person's appearance" and "subject". When any option is selected, the terminal highlights the area indicated by that option in the prompt editing interface, i.e., the prompt area. The prompt editing interface also displays a smudge control 803 and a delete control 804. When the smudge control 803 is triggered, in response to a touch operation on image 1, the terminal determines the area swiped by the touch operation as the prompt area. When the delete control 804 is triggered, in response to a touch operation on image 1, the terminal determines the area swiped by the touch operation as the non-prompt area, i.e., the prompt is canceled. Then, in response to the triggering operation of the completion control 805 in the prompt editing interface, the terminal determines the prompt area in the image, identifies the image and the prompt information used to indicate the prompt area as the prompt information, and displays it in the prompt box 806. As can be seen from the prompt box 806, the prompt area of ​​Image 1 is the main body area, and the main content of the main body area is "a beautiful woman with big eyes and curly hair".

[0287] The terminal can also modify the selected prompt image. For example, see [link to example]. Figure 8When image 1 in prompt box 806 is triggered, the terminal displays an editing panel 807 for image 1 in the image generation panel 801. The editing panel 807 displays an editing control 808, a replacement control 809, and a deletion control 810. When the editing control 808 is triggered, the terminal displays the prompt editing interface again, allowing the user to re-edit the prompt area in image 1. When the replacement control 809 is triggered, the terminal displays multiple images again, allowing the user to reselect the prompt image. When the deletion control 810 is triggered, the terminal deletes image 1 from prompt box 806.

[0288] For example, Figure 9 This is a schematic diagram illustrating another method of determining a prompt image according to an exemplary embodiment. See also Figure 9 During the process of generating video footage from images, an image acquisition control 902 is displayed in the video generation panel 901. When the image acquisition control 902 is triggered, the terminal displays multiple images. When image 6 is selected, the terminal displays image 6 in a prompt box 903 in the video generation panel 901. The user can also enter prompt text in the prompt box 903 so that video footage can be generated based on the prompt image and prompt text later. In the video generation panel, the terminal can also set video quality, camera movement, and duration. For example, during the camera movement settings, the terminal can also display multiple camera movement options and preview the camera movement effect indicated by the selected camera movement option.

[0289] During the display of multiple images, the terminal can provide images to the user based on prompt text entered by the user. Accordingly, in response to a text input operation in the material generation panel, the terminal displays the entered prompt text in the material generation panel. Then, in response to a trigger operation on the image acquisition control, the terminal displays multiple images based on the prompt text in the material generation panel. At least one of the content and style of the multiple images conforms to the prompt text. Alternatively, in response to a trigger operation on the image acquisition control, multiple images are displayed based on the prompt text in the material generation panel, and at least one image among the multiple images whose content and style conform to the prompt text is highlighted. The solution provided by the embodiments of this disclosure can provide the user with images that conform to the prompt text entered by the user, so that subsequent material can be generated quickly based on the images without the user having to manually search for the desired image, thereby improving the efficiency of material generation.

[0290] In some embodiments, the video editing interface also displays tracks other than the video track. The terminal can determine the prompt information based on the materials already added to the other tracks. Accordingly, the process of the terminal obtaining the prompt information includes: if corresponding materials have been added to other tracks, the terminal determines the prompt information based on the materials in the other tracks.

[0291] The solution provided in this disclosure, during the generation of image or video materials, can determine the prompt information based on the materials already added in other tracks of the video editing interface. This ensures that the image or video materials generated based on the prompt information match the materials already added in other tracks. Since the materials already added in other tracks are materials actively added by the user, they can accurately reflect the video effect desired by the user (including at least one of content and style). Therefore, it ensures that the generated image or video materials meet the user's desired video effect, that is, meet the user's intent. This not only reduces the probability of the user editing the image or video materials, which is beneficial to improving video generation efficiency, but also helps to improve the quality of the subsequently generated video.

[0292] For example, the terminal can generate prompt text or prompt images based on the subtitles in the subtitle track and display them in the material generation panel. The terminal can also display the subtitles used to generate the prompt information in the material generation panel, but this embodiment of the present disclosure does not limit this.

[0293] The media generation panel includes at least one of an image generation panel and a video generation panel. Correspondingly, the process by which the terminal determines the prompt information based on media in other tracks includes: if corresponding media has already been added to other tracks, the terminal determines the second media located at that position from the other tracks based on the position indicated by the timeline; and determines the prompt information based on the second media. Media in tracks other than the video track are referred to as the second media.

[0294] The solution provided in this disclosure, during the generation of image or video materials, utilizes a timeline to indicate the position of the image or video material to be generated within the video track. Based on the second material indicated by the timeline in other tracks, prompting information is determined, ensuring that the subsequently generated image or video materials based on the prompting information match the second material. Since the materials indicated by the timeline in each track are simultaneously displayed in the subsequently generated video, it ensures that the image or video material matching the second material is displayed simultaneously in the video, conforming to user intent and improving the quality of the subsequently generated video.

[0295] Alternatively, the material generation panel may include a batch generation panel. Correspondingly, the process by which the terminal determines the prompt information based on materials in other tracks includes: when corresponding materials have already been added to other tracks, the terminal determines the prompt information corresponding to each of the multiple second materials in those tracks. The solution provided by this disclosure allows for the determination of corresponding prompt information based on multiple second materials in other tracks when multiple second materials have already been added. This enables the generation of corresponding first materials based on each prompt information at once, eliminating the need for the user to generate corresponding first materials for each second material separately. This simplifies the operation and improves material generation efficiency.

[0296] For example, Figure 10 This is a schematic diagram illustrating the determination of a prompt image according to an exemplary embodiment. See also: Figure 10 The subtitle track already contains multiple subtitles, while the video track has few or no footage. In this situation, users can use a batch generation control to generate multiple clips based on the subtitles. During the batch generation process, the terminal displays a list of prompts for each subtitle in the batch generation panel. The prompts for each subtitle can be generated using artificial intelligence technology.

[0297] In some embodiments, the terminal can also edit the prompt information in the material generation panel. For the prompt information corresponding to each second material in other tracks of the material generation panel, the terminal displays the corresponding editing control. In response to the triggering operation of the editing control, the terminal displays the editing panel for the prompt information. The editing panel is used to edit the prompt information. The solution provided by the embodiments of this disclosure allows the prompt information already displayed in the material generation panel to be edited in the corresponding editing panel through the corresponding editing control, ensuring that the prompt information conforms to the user's intention, which is beneficial to improving the accuracy of subsequent video generation and obtaining a video that conforms to the user's intention.

[0298] For example, see continue. Figure 10Each prompt message corresponds to an editing control. When the editing control is triggered, the terminal displays an editing panel for editing the corresponding prompt message. This embodiment does not limit the display style of the editing panel. Optionally, the editing panel for editing prompt messages can be any panel capable of acquiring prompt messages in this solution, such as an image generation panel, a video generation panel, or a video extension panel. That is, when editing prompt messages, the terminal can jump from the current interface to the panel of the corresponding scene to reacquire the prompt message. The scene can be a scene for generating image materials, a scene for generating video materials, or a scene for extending video, etc. Alternatively, the editing panel can also be a panel with the same display style as any panel capable of acquiring prompt messages in this solution, but existing independently. Or, the editing panel can have a different display style from any other panel.

[0299] In some embodiments, the video editing interface displays multiple other tracks besides the video track. The user can select any other track from these tracks to determine a prompt message based on the second material in the selected track. Accordingly, the process of the terminal obtaining the prompt message is as follows: in response to a selection operation on any other track in the video editing interface, the terminal determines the prompt message corresponding to each second material in each of the other tracks. The selection operation can be a long press operation; for example, in response to a long press operation on any other track, the terminal determines the corresponding prompt message based on the second material in the long-pressed track. Alternatively, the selection operation includes a long press operation and an application operation on the track; for example, in response to a long press operation on any other track, the terminal displays multiple application controls for other tracks in the video editing interface, and in response to a trigger operation on the application control of any other track, the terminal determines the corresponding prompt message based on the second material in the triggered track. Alternatively, the selection operation can also be a drag operation; for example, for any other track, in response to the other track being dragged to the video track, the terminal determines the corresponding prompt message based on the second material in that other track. This disclosure does not limit the selection operation.

[0300] The solution provided in this disclosure allows users to select any other track in the video editing interface during the generation of image or video materials. The prompt information is determined by the materials in the selected other track, ensuring that the prompt information matches the user's intent. This improves the accuracy of subsequent material generation, resulting in materials that match the user's intent and reducing the probability of the user editing the materials, thereby improving video generation efficiency.

[0301] In some embodiments, in response to any prompt message in the material generation panel, and in response to a type selection operation for the prompt message, the terminal displays the material type corresponding to the prompt message in the material generation panel. In response to a confirmation generation operation, the terminal uses artificial intelligence technology to generate a first material that matches the prompt message and conforms to the corresponding material type. The solution provided by this disclosure allows selection of the material type for each prompt message in the material generation panel before material generation. This ensures that the generated material type meets user needs, improves the accuracy of subsequent video generation, and yields a video that matches the user's intent.

[0302] The terminal can independently set the material type for each prompt message, or it can set the material type for multiple prompt messages at once. This embodiment does not limit this aspect.

[0303] In some embodiments, the terminal independently sets the media type for each prompt. Correspondingly, for any prompt in the media generation panel, the terminal displays a media selection control for that prompt in the media generation panel. In response to a trigger operation on the media selection control, the terminal displays image type options and video type options. When either the image type option or the video type option is selected, the terminal displays the media type indicated by the selected type option in the media generation panel. The solution provided by this disclosure provides a corresponding media selection control for each prompt in the media generation panel, allowing users to independently set the media type for each prompt. This not only simplifies operation but also improves the flexibility of media type settings; furthermore, it intuitively displays the selected media type to the user, improving the utilization rate and information transmission efficiency of the media generation panel, enabling users to promptly adjust media types that do not meet their intentions.

[0304] For example, Figure 11 This is a schematic diagram illustrating a setting of material types according to an exemplary embodiment. See also... Figure 11 Each prompt message corresponds to a type selection control, used to determine the type of material corresponding to that prompt. Material types include image and video types, so that the terminal can subsequently generate image or video materials based on the prompt message. The selected material type can be displayed in the batch generation panel. Figure 11 As can be seen, the material type corresponding to prompt message 1, prompt message 2 and prompt message 5 is image type; the material type corresponding to prompt message 3, prompt message 4 and prompt message 6 is video type.

[0305] In other embodiments, the terminal can simultaneously set the media type for multiple prompts. Accordingly, in response to the selection of any number of prompts in the media generation panel, the terminal displays image type options and video type options. When either the image type option or the video type option is selected, the terminal displays the media type indicated by the selected type option in the media generation panel. The solution provided by this disclosure provides a corresponding media selection control for each prompt in the media generation panel, allowing users to independently set the media type for each prompt. This not only simplifies operation but also improves the flexibility of media type setting; furthermore, it intuitively displays the selected media type to the user, improving the utilization rate and information transmission efficiency of the media generation panel, enabling users to promptly adjust media types that do not meet their intentions.

[0306] For example, Figure 12 This is a schematic diagram illustrating another setting of material type according to an exemplary embodiment. See also Figure 12 When prompts 3, 4, and 6 are selected, the terminal displays image type option 1201 and video type option 1202. When video type option 1202 is triggered, the terminal displays that the material type corresponding to prompts 3, 4, and 6 is video.

[0307] In step 304, in response to the confirmation generation operation, the terminal uses artificial intelligence technology to generate first material that matches the prompt information and displays the first material in the video track.

[0308] In this embodiment of the disclosure, when it is confirmed that the generation operation has been triggered, the terminal uses artificial intelligence technology to generate a first material that matches the prompt information, and then displays the first material in the video track.

[0309] In some embodiments, the terminal can use artificial intelligence technology to generate corresponding materials in any of the aforementioned material generation panels. Correspondingly, in response to the confirmation generation operation in the image generation panel, the terminal uses artificial intelligence technology to generate image materials matching the prompt information in the image generation panel. Alternatively, in response to the confirmation generation operation in the video generation panel, the terminal uses artificial intelligence technology to generate video materials matching the prompt information in the video generation panel. Alternatively, in response to the confirmation generation operation in the batch generation panel, the terminal uses artificial intelligence technology to generate image materials or video materials matching each of the multiple prompts in the batch generation panel. The solution provided by this disclosure embodiment can provide users with at least one of image generation controls, video generation controls, and batch generation controls, so that users can obtain prompt information in the corresponding generation panel according to their needs through the corresponding generation controls to generate the required image materials, video materials, or batch materials. This not only enriches the material generation methods but also simplifies the operation, allowing users to obtain the desired image materials, video materials, or batch materials immediately, thus improving material acquisition efficiency.

[0310] In some embodiments, the material generation panel includes a video generation panel. During the image-based video generation process, the terminal can employ artificial intelligence technology to augment the content of a single image to obtain video material. Alternatively, the terminal can also employ artificial intelligence technology to supplement multiple images with transitional content to obtain video material.

[0311] Optionally, the prompt information in the video generation panel includes multiple images. The video generation panel displays a first and last frame mode control. Accordingly, the process of the terminal generating the first material matching the prompt information includes: when the first and last frame mode control is triggered, in response to the confirmation generation operation, for any two adjacent images in any order among the multiple images in the video generation panel, the terminal uses an interpolation algorithm to process the two images to obtain a transition image between the two images. The terminal generates the first material based on the multiple images in the video generation panel and the transition images of the multiple images. The solution provided by the embodiments of this disclosure, in the process of generating video material based on images, can perform interpolation processing on the multiple images used for video generation through the first and last frame mode control to obtain a transition image between the multiple images, making the transition between multiple images in the subsequently generated video more natural and improving the video quality.

[0312] For example, Figure 13 This is a schematic diagram illustrating the generation of video footage from the first and last frames according to an exemplary embodiment. See also... Figure 13The video generation panel 1301 displays a first and last frame mode control 1302. When the first and last frame mode control 1302 is triggered, the terminal displays a first frame image acquisition control 1303 and a last frame image acquisition control 1304 on the video generation panel 1302. In response to triggering any of the image acquisition controls, the terminal displays multiple images for the user to select, and displays the image selected by the user at the corresponding position. Figure 13 As can be seen, image 3 is the first frame image, and image 6 is the last frame image. Then, in response to the trigger operation of the generation control 1305, the terminal uses an interpolation algorithm to... Figure 3 Image 3 and Image 6 are processed to obtain a transition image between them. The terminal then places the transition image between Image 3 and Image 6 to generate video footage. Figure 3 It is the first frame of the video footage. Figure 6 It is the last frame image of the video footage.

[0313] In some embodiments, considering that the material generation process typically takes some time, this solution can minimize the material generation panel to avoid hindering the user's continued operation, enabling asynchronous material generation and display. Accordingly, during the generation of the first material, the terminal displays a minimize control in the material generation panel. In response to triggering the minimize control, the terminal minimizes the material generation panel. The terminal highlights a first target area in the video track, indicating the position the first material will occupy in the video track. Then, the terminal displays the first material in the first target area.

[0314] The solution provided in this embodiment allows for minimizing the media generation panel via a minimize control during the media generation process. This eliminates the need to wait for media generation to complete, enabling other operations to be performed in the video editing interface. For example, during the generation of the first media, additional prompts can be obtained to prepare for the generation of the next media, or existing media in the video editing interface can be edited, thus improving video generation efficiency. Furthermore, after minimizing the media generation panel, the first target area can be highlighted in the video track, intuitively showing the user the position occupied by the first media being generated, enhancing the user's sense of control over media generation and improving the user experience.

[0315] For example, Figure 14 This is a schematic diagram illustrating a minimized material generation panel according to an exemplary embodiment. See also Figure 14In response to a trigger operation on the generation control 1402 in the video generation panel 1401, the terminal generates video footage based on the prompt image (Image 6), video quality options, camera movement, duration, and other prompt information in the video generation panel 1401. During the video footage generation process, the terminal displays a minimize control 1403 in the video generation panel 1401. When the minimize control 1403 is triggered, the terminal minimizes the video footage generation panel 1401. The terminal highlights the target area (gray area) in the video track 1404 to indicate the position the video footage will occupy in the video track. Then, after generation, the terminal displays multiple frames of the video footage in the target area. During the video footage generation process, the terminal can also display the generation progress of the video footage, such as 18% or 80%.

[0316] In some embodiments, after the footage is generated, the terminal can first display it in the footage generation panel for user confirmation, and then display it in the footage track. Accordingly, the terminal displays the first footage in the footage generation panel. Then, in response to the selection of the first footage, if the timeline is in an empty area of ​​the video track, the terminal displays the first footage after any footage already added to the video track; the empty area is the area in the video track where no footage exists. Alternatively, in response to the selection of the first footage, if the timeline is in a non-empty area of ​​the video track, the terminal displays the first footage between the two footage closest to the timeline in the video track. That is, the terminal can add newly generated footage after any footage already added to the video track, or it can add newly generated footage between two already added footage.

[0317] The solution provided in this disclosure embodiment can display the generated first material in the material generation panel for user selection. When the user selects the first material, the first material can be displayed in the video track according to the position of the timeline. For example, the first material can be displayed after the materials already added in the video track, or the first material can be displayed between the two materials closest to the timeline, thus enriching the ways of displaying materials in the video track.

[0318] During the media generation process, the terminal can generate multiple media clips at once and display them in the media generation panel for the user to select. When any media clip is selected, the terminal adds that clip to the video track.

[0319] For example, Figure 15 This is a schematic diagram illustrating the selection of materials according to an exemplary embodiment. See also... Figure 15During the image material generation process, in response to the trigger operation of the generation control 1502 in the image generation panel 1501, the terminal generates multiple image materials based on the text prompts (text template 2), style, aspect ratio, and other prompt information in the image generation panel 1501. The terminal displays multiple image materials in the image generation panel 1501, including image material 1, image material 2, image material 3, and image material 4. In response to the selection operation of image material 1, the terminal displays multiple frames of image material 1 in the video track 1503. The multiple frames are obtained by copying image material 1. The number of frames of image material 1 is positively correlated with the display duration of image material 1 in the video to be generated. The user can automatically adjust the number of frames of image material 1.

[0320] During the display of footage in the video track, in addition to displaying newly generated footage according to the position indicated by the timeline, as mentioned earlier, the terminal can also display newly generated footage according to the position of footage already added in other tracks.

[0321] In some embodiments, the first material is generated based on the prompt information corresponding to the second material. Accordingly, the process of the terminal displaying the first material in the video track includes: the terminal displaying the first material in the video track according to the position of the second material in other tracks. The solution provided by this disclosure embodiment can display the first material according to the position of the second material in other tracks, achieving the purpose of automatically aligning matching first and second materials. Since materials aligned between tracks will be displayed simultaneously in the subsequently generated video, it ensures that the first material matching the second material is displayed simultaneously with the second material in the video. This not only conforms to the user's intention and eliminates the need for the user to manually align matching materials between tracks, reducing user operations and improving video generation efficiency, but also helps improve the quality of the subsequently generated video.

[0322] The prompt information used to generate the first material includes the second material. Correspondingly, the length of the first material is equal to the length of the second material. The length of the material represents its display duration. That is, the terminal can generate a first material with the same display duration based on the display duration of the second material in the prompt information. During the display of the first material, the first and second materials can achieve positional and duration alignment. The solution provided in this embodiment ensures that the display duration of the first material matching the second material is the same as that of the second material, guaranteeing that the first and second materials are displayed and disappear simultaneously in the subsequently generated video. This ensures that the materials displayed at various time points in the subsequently generated video are matched, which is beneficial for improving video quality.

[0323] In other embodiments, during the batch generation of materials, multiple second materials exist in other tracks. In response to the confirmation of generation, the terminal, based on the prompts corresponding to the multiple second materials in other tracks, uses artificial intelligence technology to generate first materials corresponding to each prompt. Then, the terminal displays the first materials corresponding to each prompt in the video track according to the order of the multiple second materials in other tracks. The solution provided by this disclosure not only generates corresponding first materials based on the prompts corresponding to the multiple second materials in other tracks—that is, it can generate multiple first materials at once, improving material generation efficiency—but also displays the generated multiple first materials in the order of the corresponding multiple second materials, ensuring that matching first and second materials are aligned. This guarantees that matching materials will be displayed simultaneously in the video, conforming to user intent, eliminating the need for manual alignment of matching materials between tracks, reducing user operations, and improving video generation efficiency and video quality.

[0324] In this embodiment of the disclosure, after the generated material is displayed in the video track, the terminal can also edit the material in the video track. For example, the terminal can regenerate existing material in the video track; or, the terminal can convert image material in the video track into video material; or, the terminal can extend video material; or, the terminal can perform transition processing on existing material in the video track, etc., and this embodiment of the disclosure does not limit these possibilities.

[0325] In some embodiments, the terminal can regenerate existing footage in the video track. Accordingly, in response to an update operation on the first footage in the video track, the terminal regenerates a third footage. Then, the terminal replaces the first footage in the video track with the third footage. The solution provided by this disclosure allows for the regeneration and replacement of existing footage in the video track using artificial intelligence technology, facilitating user adjustments to unsatisfactory footage and conforming to user intent.

[0326] For example, Figure 16 This is a schematic diagram illustrating an update of image material according to an exemplary embodiment. See also: Figure 16For image material 1 in video track 1601, in response to a trigger operation on image generation control 1601, the terminal displays image generation panel 1603. Image generation panel 1603 displays multiple previously generated image materials, a regeneration control 1604, and an adjustment request control 1605. When regeneration control 1604 is triggered, the terminal regenerates multiple image materials based on the prompt information used to generate image material 1 and displays them in image generation panel 1603. When adjustment request control 1605 is triggered, the terminal can display the prompt information used to generate image material 1 (such as prompt text, style, aspect ratio, etc.) so that the user can edit the prompt information. Then, in response to a trigger operation on generation control 1606, the terminal regenerates multiple image materials based on the edited prompt information and displays them in image generation panel 1603. Then, when image material 5 is triggered, the terminal replaces image material 1 in the video track with image material 5. The principle of updating video materials is the same as that of updating image materials, and will not be repeated here.

[0327] In some embodiments, the terminal can convert image footage in a video track into video footage. For any image footage in the video track, in response to the conversion operation, the terminal generates video footage based on the content and length of the image footage. Then, the terminal replaces the image footage in the video track with video footage. The content of the video footage is related to the content of the image footage. The length of the video footage and the length of the image footage conform to a preset ratio. The length of the footage is used to represent the display duration of the footage. The preset ratio can be 1:1, 1:2, or 2:3, etc., and this embodiment does not limit this. That is, the display duration of the video footage can be equal to or different from the display duration of the image footage, and this embodiment does not limit this.

[0328] The solution provided in this disclosure can convert existing image materials in a video track into video materials with relevant content and corresponding display duration, increasing the dynamic changes of the material content, making the material content more visually appealing, and improving the quality of subsequently generated videos; moreover, video materials can be directly generated based on image materials, without requiring the user to obtain prompt information again to generate video materials, which is simple to operate and can improve the efficiency of video material generation.

[0329] For example, Figure 17 This is a schematic diagram illustrating an image-to-video conversion process according to an exemplary embodiment. See also... Figure 17For image material 1 in video track 1701, in response to a trigger operation on conversion control 1702, the terminal displays video generation panel 1703. The video generation panel 1703 displays image material 1 as a prompt image. In response to a trigger operation on generation control 1704, the terminal generates video material based on image material 1. During video material generation, the terminal can minimize the video generation panel and highlight the position of image material 1 in the video track (gray area). Then, after generating video material 3, the terminal directly replaces image material 1 in the video track with video material 3. Figure 17 As can be seen, the width of video material 3 is equal to the width of image material 1, meaning that their display durations are equal.

[0330] In some embodiments, the terminal can extend video footage in a video track. Accordingly, in response to an extension operation on any video footage in the video track, the terminal displays a video extension panel. Then, based on extension prompt information in the video extension panel, the terminal displays incremental video footage in the video track that matches the extension prompt information. The extension prompt information includes at least one of prompt text and prompt image. The extension prompt information is used to indicate at least one of the content and style of the incremental video footage to be generated. The incremental video footage is located after and contiguous with the original video footage.

[0331] The solution provided in this embodiment allows for the extension of any video clip in a video track without needing to regenerate the complete video clip based on the prompt information used in the video clip. Instead, the extended portion (i.e., incremental video clip) is generated simply by extending the instruction information, and then connected to the original video clip. Compared to regenerating the complete video clip, the incremental video clip generates less data and can be generated more quickly, thereby improving the efficiency of extending the video clip.

[0332] The extension operation can be triggered by lengthening the display duration of any video clip. Accordingly, the process of the terminal displaying the video extension panel includes: in response to a trigger operation on any video clip in the video track, the terminal displays a time window of that video clip in the video track. The length of the time window represents the display duration of the video clip. When the length of the time window is equal to the maximum duration of the video clip, and a lengthening operation on the time window is detected, the terminal displays the video extension panel. The video clip can be generated based on artificial intelligence technology, or it can be video clip captured by camera, etc., and this embodiment does not limit this. For video clips generated by artificial intelligence technology, the maximum duration of the video clip is the duration of the video clip at the time of generation and before editing. For video clips captured by camera, the maximum duration of the video clip is the original duration of the video clip. That is, if any video clip is lengthened while it is at its maximum duration, the extension operation is triggered.

[0333] Alternatively, for any video clip in the video track, in response to editing operations on the video clip, the terminal displays an extension control in the video editing interface. In response to triggering the extension control, a video extension panel is displayed. That is, the extension operation is triggered by triggering the control.

[0334] The solution provided in this disclosure can extend the video clip's time window. If the time window continues to extend even when it is at its maximum length, a video extension panel can be displayed to extend the video clip. Alternatively, the video extension panel can be displayed by triggering the extension control in the video editing interface to extend the video clip. This not only enriches the ways to extend video clips but also simplifies the operation and improves the efficiency of extending videos.

[0335] For example, Figure 18 This is a schematic diagram illustrating a triggered extension operation according to an exemplary embodiment. See also: Figure 18 Video track 1801 displays multiple frames of video footage 1. See also... Figure 18 In (a), in response to the selection of video clip 1 in video track 1801, the terminal displays a time window (dashed line) around video clip 1 in video track 1801. In response to dragging the right side of the time window to the right, the time window is extended, and the display duration of video clip 1 is extended accordingly, but the content of video clip 1 is not increased. When the display duration of video clip 1 reaches its maximum length, if the right side of the time window is dragged further to the right, the display time of video clip 1 will not be extended, but instead an extension operation on the content of video clip 1 will be triggered, thereby displaying the video extension panel. Alternatively, see [link to relevant documentation]. Figure 18In (b), the video editing interface displays an extension control 1802. When the extension control 1802 is triggered, an extension operation is performed on video material 1, thereby displaying the video extension panel.

[0336] The incremental video footage is generated based on the prompt text entered in the video extension panel. For example, Figure 19 This is a schematic diagram illustrating a video extension panel according to an exemplary embodiment. See also: Figure 19 A text prompt box 1902 is displayed in the video extension panel 1901. In response to text input in the text prompt box 1902, the terminal displays the entered prompt text in the text prompt box 1902. Then, in response to a trigger operation on the extension control 1903, the terminal generates incremental video footage based on the prompt text. During the generation of incremental video footage, the terminal can minimize the video extension panel and highlight the position to be occupied by the incremental video footage in the video track, similar to... Figure 14 The display method in the document will not be elaborated here.

[0337] Alternatively, incremental video clips are generated based on the cue image entered in the video extension panel. The cue image is the image located at the end of the last frame in video clip 1 (i.e., the end frame image). For example, Figure 20 This is a schematic diagram illustrating another video extension panel according to an exemplary embodiment. See also Figure 20 The video extension panel is similar to the video generation panel. A prompt box 2002 is displayed in the video extension panel 2001. The prompt box 2002 displays the last frame image of video clip 1. The user can also enter prompt text in the prompt box 2002, or leave it blank. In response to the triggering operation of the generation control 2003, the terminal generates incremental video clips based on the last frame image of video clip 1. During the generation of incremental video clips, the terminal can minimize the video extension panel and highlight the position to be occupied by the incremental video clips in the video track, similar to... Figure 14 The display method in the document will not be elaborated here.

[0338] This disclosure does not limit the method of generating incremental video footage. The solution provided in this disclosure can generate incremental video footage by inputting prompt text, or it can generate incremental video footage based on the last frame of the video footage to be extended. This not only enriches the methods of generating incremental video footage, but also ensures that the incremental video footage matches the video footage to be extended by generating incremental video footage based on the last frame of the video footage to be extended, which is beneficial to improving the quality of the video footage.

[0339] In the process of displaying incremental video material based on extended prompt information, the terminal can first generate the incremental video material, and then display the entire incremental video material after it is generated; alternatively, the terminal can also display the already generated image while generating the incremental video material.

[0340] In some embodiments, during the generation of incremental video material based on extended prompt information, the terminal highlights a second target area in the video track. The second target area indicates the position the incremental video material will occupy in the video track. The second target area is located after and adjacent to the area occupied by the video material. Then, after generating the incremental video material, the terminal displays the incremental video material in the second target area. The solution provided by this disclosure, during the generation of incremental video material, can highlight the second target area in the video track, intuitively showing the user the position occupied by the incremental video material being generated. This not only improves the utilization rate and information transmission efficiency of the video editing interface but also enhances the user's sense of control over the material generation, thus improving the user experience.

[0341] For example, Figure 21 This is a schematic diagram illustrating the display of incremental video footage according to an exemplary embodiment. See also... Figure 21 The terminal highlights the second target area 2102 (gray area) after video material 1 in video track 2101. Then, after the incremental video material is generated, the terminal directly displays the incremental video material in the second target area 2102.

[0342] In other embodiments, during the generation of incremental video material based on extended prompts, the terminal displays the generated images in the incremental video material frame by frame after the video material in the video track until the generation of the incremental video material is complete. The solution provided by this disclosure, during the generation of incremental video material, can display the generated images in the incremental video material frame by frame after the video material in the video track, intuitively showing the user the generation progress of the incremental video material. This not only improves the utilization rate and information transmission efficiency of the video editing interface but also enhances the user's sense of control over the material generation, thus improving the user experience.

[0343] For example, Figure 22 This is an illustration of another method for displaying incremental video footage, according to an exemplary embodiment. See also... Figure 22 During the generation of incremental video footage, for each image generated in the incremental video footage, the terminal displays that image sequentially in video track 2201. Figure 22 As can be seen, the first frame of the incremental video footage has been generated and displayed.

[0344] In some embodiments, the terminal can perform transition processing on existing footage in a video track to seamlessly connect the content between adjacent footage. Accordingly, in response to a transition footage generation operation, the terminal determines the fourth and fifth footage indicated by the transition footage generation operation. If the fifth footage follows the fourth footage, the terminal generates transition footage based on the last frame image of the fourth footage and the first frame image of the fifth footage; or, if the fifth footage precedes the fourth footage, the terminal generates transition footage based on the first frame image of the fourth footage and the last frame image of the fifth footage. Then, the transition footage is displayed between the fourth and fifth footage. The solution provided by this disclosure embodiment can generate transition footage between any two adjacent footage in a video track and display the transition footage between them, making the transition between multiple footage in the subsequently generated video more natural and improving video quality.

[0345] During the process of determining the fourth and fifth clips for the transition clip generation operation, the terminal can display a transition generation panel in response to the triggering operation of the transition generation control in the video editing interface. The transition generation panel displays a first generation option and a second generation option. The first generation option indicates the generation of transition clips between the fourth clip and the clip preceding it. The second generation option indicates the generation of transition clips between the fourth clip and the clip following it. The fourth clip is the clip indicated by the timeline in the video track. When the first generation option is selected, the terminal determines the clip preceding the fourth clip as the fifth clip. When the second generation option is selected, the terminal determines the clip following the fourth clip as the fifth clip.

[0346] The solution provided in this disclosure, by displaying a first generation option and a second generation option in the transition generation panel, allows the user to choose whether to generate transition material between the previous material and the next material during the generation process. This makes the generation of transition material more flexible and controllable, resulting in transition material that meets the user's intentions. This reduces the probability of the user editing the material, thereby improving video generation efficiency.

[0347] While displaying the transition generation panel, the terminal can show that a certain generation option in the transition generation panel is automatically selected. Accordingly, when the distance between the first and second clips is greater than the distance between the second and third clips, in response to triggering the transition generation control in the video editing interface, the terminal displays that the first generation option is selected in the transition generation panel. When the distance between the first and second clips is less than the distance between the second and third clips, in response to triggering the transition generation control in the video editing interface, the terminal displays that the second generation option is selected in the transition generation panel. The first clip distance is the distance between the position indicated by the timeline on the video track and the clip preceding the fourth clip, and the second clip distance is the distance between the position indicated by the timeline on the video track and the clip following the fourth clip.

[0348] The solution provided in this disclosure automatically selects the generation option based on the position indicated by the timeline in the video track and the positional relationship (i.e., distance) between the footage in the video track and the timeline. This eliminates the need for manual selection by the user, simplifies the operation, and improves the efficiency of footage generation.

[0349] For example, Figure 23 This is a schematic diagram illustrating the generation of transition material according to an exemplary embodiment. See also: Figure 23 The video editing interface displays a transition generation control 2301. When the transition generation control 2301 is triggered, the terminal displays a transition generation panel for generating transition footage between clips. The transition generation panel displays a first generation option 2302 and a second generation option 2303. The terminal can determine the fourth clip as clip 2 based on the position indicated by the timeline 2104 in the video track. When the first generation option 2302 is selected, the terminal determines the clip preceding clip 2 as the fifth clip. Then, the terminal can generate transition footage between clip 1 and clip 2.

[0350] Before generating transition materials, users can input transition prompt text, creative relevance, generation mode (i.e., video quality requirements), and video duration in the transition generation panel as prompts for generating transition materials. Creative relevance indicates the degree of correlation between the generated transition materials and the transition prompt file. Then, based on the prompts in the transition generation panel, the terminal performs interpolation processing on material 1 and material 2 to obtain the transition material between material 1 and material 2. During the generation of transition materials, the terminal can minimize the transition compositing panel and highlight the position to be occupied by the transition material in the video track, similar to... Figure 14 The display method in the document will not be elaborated here.

[0351] In step 305, in response to the video generation operation, the terminal generates a video based on the first material in the video track.

[0352] In this embodiment of the disclosure, when a video generation operation is triggered, the terminal generates a video based on the first material in the video track. The video includes the first material. The display duration of the first material in the video is equal to the display duration of the first material in the video track.

[0353] This disclosure provides a video generation method. During the video generation process within a video editing scenario, a material generation control is displayed in the video editing interface. By triggering the material generation control, at least one of image and video materials can be generated using artificial intelligence technology, enriching the materials used for video generation. These materials are then directly displayed in the video track, so that subsequent video generation operations can be triggered to directly generate the video based on the materials in the video track. The operation is simple and does not require switching to other applications to generate image or video materials. This method deeply integrates image / video material generation functions into the video editing scenario, allowing seamless connection between material generation and video generation, thereby improving video generation efficiency.

[0354] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0355] Figure 24 This is a block diagram illustrating a video generation apparatus according to an exemplary embodiment. See also... Figure 24 The device includes:

[0356] Display unit 2401 is configured to display a video editing interface, which displays material generation controls and a video track.

[0357] The first generation unit 2402 is configured to perform a trigger operation in response to a material generation control, and generate a first material using artificial intelligence technology. The first material includes at least one of image material and video material.

[0358] Display unit 2401 is also configured to display the first footage in the video track;

[0359] The second generation unit 2403 is configured to perform a video generation operation in response to the video generation operation, generating a video based on the first material in the video track.

[0360] In some embodiments, the first generating unit 2402 includes:

[0361] The display sub-unit is configured to display the material generation panel in response to a trigger operation on the material generation control;

[0362] The acquisition subunit is configured to acquire prompt information, the prompt information including at least one of prompt text and prompt image, the prompt information being used to indicate at least one of the content and style of the material to be generated;

[0363] The display subunit is also configured to display a prompt message in the material generation panel;

[0364] The generation subunit is configured to perform a response to the confirmation generation operation, using artificial intelligence technology to generate first material that matches the prompt message.

[0365] In some embodiments, the material generation control includes at least one of an image generation control, a video generation control, and a batch generation control, and the material generation panel includes at least one of an image generation panel, a video generation panel, and a batch generation panel.

[0366] The generated subunit is configured to perform any of the following:

[0367] In response to the confirmation action in the image generation panel, based on the prompts in the image generation panel, artificial intelligence technology is used to generate image materials that match the prompts.

[0368] In response to the confirmation action in the video generation panel, and based on the prompts in the video generation panel, artificial intelligence technology is used to generate video footage that matches the prompts.

[0369] In response to the confirmation of generation in the batch generation panel, based on multiple prompts in the batch generation panel, artificial intelligence technology is used to generate image or video materials that match each of the multiple prompts.

[0370] In some embodiments, the video editing interface also displays tracks other than the video track;

[0371] The sub-unit is configured to determine the prompt message based on the materials in other tracks, provided that corresponding materials have already been added to other tracks.

[0372] In some embodiments, the material generation panel includes at least one of an image generation panel and a video generation panel;

[0373] The sub-unit is configured to, when corresponding material has been added to other tracks, determine the second material at the position indicated by the timeline in other tracks; and determine the prompt information based on the second material.

[0374] In some embodiments, the display unit 2401 is configured to display the first material in a video track according to the position of the second material in other tracks.

[0375] In some embodiments, the prompt information includes a second material, the length of which is equal to the length of the second material, and the length of the material is used to indicate the display duration of the material.

[0376] In some embodiments, the material generation panel is a batch generation panel;

[0377] The sub-unit is configured to determine the corresponding prompt information for each of the multiple second materials based on the multiple second materials in other tracks, provided that corresponding materials have been added to other tracks.

[0378] In some embodiments, the generation subunit is configured to perform, in response to a confirmation generation operation, generate first material corresponding to each prompt information based on prompt information corresponding to each of the multiple second materials in other tracks, using artificial intelligence technology.

[0379] Display unit 2401 is configured to display the first material corresponding to each prompt message in the video track in the order of multiple second materials in other tracks.

[0380] In some embodiments, the video editing interface also displays tracks other than the video track;

[0381] The display unit 2401 is also configured to execute prompt information for each second material in other tracks of the material generation panel, display the editing control corresponding to the prompt information; and in response to a trigger operation on the editing control, display the editing panel for editing the prompt information.

[0382] In some embodiments, the video editing interface displays multiple other tracks in addition to the video track;

[0383] The sub-unit is configured to perform a response to the selection operation of any other track in the video editing interface, and determine the prompt information corresponding to each second material based on each second material in the other track.

[0384] In some embodiments, the display unit 2401 is further configured to perform a response to any prompt message in the material generation panel, and in response to a type selection operation of the prompt message, display the material type corresponding to the prompt message in the material generation panel;

[0385] The generation subunit is configured to perform a response to the confirmation generation operation, using artificial intelligence technology to generate a first material that matches the prompt information and conforms to the corresponding material type.

[0386] In some embodiments, the display unit 2401 is configured to perform the following actions in response to any prompt in the material generation panel: display a material selection control for the prompt in the material generation panel; display an image type option and a video type option in response to a trigger operation on the material selection control; and display the material type indicated by the selected type option in the material generation panel when either the image type option or the video type option is selected.

[0387] In some embodiments, the display unit 2401 is configured to perform an operation in response to the selection of any plurality of prompts in the media generation panel, displaying image type options and video type options; when either the image type option or the video type option is selected, the media type indicated by the selected type option is displayed in the media generation panel.

[0388] In some embodiments, a template selection control is displayed in the material generation panel;

[0389] The sub-unit is configured to display multiple text templates in response to a trigger operation on the template selection control; and to determine the text template as a prompt message in response to an application operation on any text template.

[0390] In some embodiments, the material generation panel displays an image acquisition control;

[0391] The acquisition sub-unit is configured to perform actions in response to a triggering operation of an image acquisition control, displaying multiple images; and in response to a selection operation of at least one of the multiple images, determining a prompt message based on at least one image.

[0392] In some embodiments, the acquisition subunit is configured to perform the following actions for any one of a plurality of images: in response to a selection operation on the image, display the image in a prompt editing interface; in response to a prompt selection operation on the image in the prompt editing interface, determine a prompt area in the image, and determine the image and the area prompt information used to indicate the prompt area as prompt information.

[0393] In some embodiments, the display unit 2401 is further configured to perform a text input operation in response to the material generation panel, displaying input prompt text in the material generation panel;

[0394] The subunit is retrieved and configured to perform any of the following:

[0395] In response to a trigger operation on the image acquisition control, multiple images are displayed based on the prompt text in the material generation panel, wherein at least one of the content and style of the multiple images conforms to the prompt text;

[0396] In response to a triggering operation on the image acquisition control, multiple images are displayed based on the prompt text in the material generation panel, and at least one image among the multiple images whose content and style match the prompt text is highlighted.

[0397] In some embodiments, the material generation panel includes a video generation panel, the prompts in the video generation panel include multiple images, and the video generation panel displays a first and last frame mode control.

[0398] The generation subunit is configured to, in response to the confirmation generation operation, when the first and last frame mode control is triggered, process two adjacent images in any order among the multiple images in the video generation panel using an interpolation algorithm to obtain a transition image between the two images; and generate the first material based on the multiple images in the video generation panel and the transition image of the multiple images.

[0399] In some embodiments, the display unit 2401 is further configured to display the first material in the material generation panel;

[0400] In response to the selection of the first clip, if the timeline is in an empty area of ​​the video track, the first clip is displayed after any clips already added to the video track; the empty area is the region in the video track where no clips exist. Alternatively,

[0401] In response to the selection of the first clip, if the timeline is in a non-idle area of ​​the video track, the first clip is displayed between the two clips closest to the timeline in the video track.

[0402] In some embodiments, the display unit 2401 is further configured to perform the following actions during the generation of the first material: displaying a minimize control in the material generation panel; minimizing the material generation panel in response to a triggering operation on the minimize control; highlighting a first target area in the video track, the first target area indicating the position to be occupied by the first material in the video track; and displaying the first material in the first target area.

[0403] In some embodiments, the first generation unit 2402 is further configured to perform a third material regeneration in response to an update operation on the first material in the video track;

[0404] The display unit is also configured to replace the first clip in the video track with the third clip.

[0405] In some embodiments, the first generation unit 2402 is further configured to perform, for any image material in the video track, in response to a conversion operation on the image material, generate video material based on the content and length of the image material, wherein the content of the video material is related to the content of the image material, the length of the video material conforms to a preset proportional relationship with the length of the image material, and the length of the material is used to represent the display duration of the material;

[0406] Display unit 2401 is also configured to perform the replacement of image material in the video track with video material.

[0407] In some embodiments, the display unit 2401 is further configured to perform a video extension panel in response to an extension operation on any video clip in the video track;

[0408] The display unit 2401 is further configured to perform an extension prompt message based on the video extension panel, displaying incremental video material matching the extension prompt message in the video track. The extension prompt message includes at least one of prompt text and prompt image, and is used to indicate at least one of the content and style of the incremental video material to be generated. The incremental video material is located after and adjacent to the video material.

[0409] In some embodiments, the display unit 2401 is configured to perform any of the following:

[0410] In response to a trigger operation on any video clip in the video track, a time window for that video clip is displayed in the video track. The length of the time window represents the display duration of the video clip. If the length of the time window is equal to the maximum duration of the video clip, and a time window stretching operation is detected, a video extension panel is displayed.

[0411] For any video clip in the video track, in response to editing operations on the video clip, an extension control is displayed in the video editing interface, and in response to triggering the extension control, a video extension panel is displayed.

[0412] In some embodiments, the incremental video material is at least one of the following:

[0413] Incremental video footage is generated based on the prompt text entered in the video extension panel;

[0414] The incremental video clip is generated based on the prompt image entered in the video extension panel. The prompt image is the image located in the last frame of the video clip.

[0415] In some embodiments, the display unit 2401 is configured to, during the process of generating incremental video material based on extended prompt information, highlight a second target area in the video track, the second target area indicating the position to be occupied by the incremental video material in the video track, the second target area being located after and adjacent to the area occupied by the video material; after generating the incremental video material, display the incremental video material in the second target area.

[0416] In some embodiments, the display unit 2401 is configured to perform the following steps during the process of generating incremental video material based on extended prompt information: displaying the generated images in the incremental video material frame by frame after the video material in the video track until the incremental video material is generated.

[0417] In some embodiments, the apparatus further includes:

[0418] The determining unit is configured to perform a response to the transition material generation operation, determining the fourth and fifth materials indicated by the transition material generation operation;

[0419] The first generation unit 2402 is further configured to generate transition material based on the last frame image in the fourth material and the first frame image in the fifth material when the fifth material is after the fourth material; or, when the fifth material is before the fourth material, generate transition material based on the first frame image in the fourth material and the last frame image in the fifth material.

[0420] The display unit is also configured to display transition material between the fourth and fifth materials.

[0421] In some embodiments, the determining unit is configured to perform a trigger operation in response to a transition generation control in the video editing interface, displaying a transition generation panel. The transition generation panel displays a first generation option and a second generation option. The first generation option indicates the generation of a transition material between the fourth material and the material preceding the fourth material, and the second generation option indicates the generation of a transition material between the fourth material and the material following the fourth material. The fourth material is the material indicated by the timeline in the video track. When the first generation option is selected, the material preceding the fourth material is determined as the fifth material. When the second generation option is selected, the material following the fourth material is determined as the fifth material.

[0422] In some embodiments, the determining unit is configured to, in response to a triggering operation of the transition generation control in the video editing interface, display a first generation option as selected in the transition generation panel when the distance between the first and second clips is greater than the distance between the second clips; and in response to a triggering operation of the transition generation control in the video editing interface, display a second generation option as selected in the transition generation panel when the distance between the first and second clips is less than the distance between the second and third clips; wherein the first clip distance is the distance between the position indicated by the timeline in the video track and the clip preceding the fourth clip, and the second clip distance is the distance between the position indicated by the timeline in the video track and the clip following the fourth clip.

[0423] This disclosure provides a video generation method. During the video generation process within a video editing scenario, a material generation control is displayed in the video editing interface. By triggering the material generation control, at least one of image and video materials can be generated using artificial intelligence technology, enriching the materials used for video generation. These materials are then directly displayed in the video track, so that subsequent video generation operations can be triggered to directly generate the video based on the materials in the video track. The operation is simple and does not require switching to other applications to generate image or video materials. This method deeply integrates image / video material generation functions into the video editing scenario, allowing seamless connection between material generation and video generation, thereby improving video generation efficiency.

[0424] It should be noted that the video generation apparatus provided in the above embodiments is only illustrated by the division of the above functional units when generating video. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the video generation apparatus and the video generation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0425] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0426] When an electronic device is provided as a terminal, Figure 25 This is a block diagram illustrating a terminal 2500 according to an exemplary embodiment. The terminal... Figure 25This disclosure provides a structural block diagram of a terminal 2500 according to an exemplary embodiment. The terminal 2500 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 2500 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0427] Typically, terminal 2500 includes a processor 2501 and a memory 2502.

[0428] Processor 2501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 2501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0429] The memory 2502 may include one or more computer-readable storage media, which may be non-transitory. The memory 2502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2502 are used to store at least one computer program, which is executed by the processor 2501 to implement the video generation method provided in the method embodiments of this application.

[0430] In some embodiments, the terminal 2500 may also optionally include a peripheral device interface 2503 and at least one peripheral device. The processor 2501, memory 2502, and peripheral device interface 2503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 2503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: radio frequency circuitry 2504, display screen 2505, camera assembly 2506, audio circuitry 2507, and power supply 2508.

[0431] Peripheral device interface 2503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 2501 and memory 2502. In some embodiments, processor 2501, memory 2502 and peripheral device interface 2503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 2501, memory 2502 and peripheral device interface 2503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0432] The radio frequency (RF) circuit 2504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 2504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 2504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 2504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 2504 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 2504 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0433] Display screen 2505 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 2505 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 2501 for processing. In this case, display screen 2505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 2505, disposed on the front panel of terminal 2500; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 2500 or in a folded design; in still other embodiments, display screen 2505 may be a flexible display screen, disposed on a curved or folded surface of terminal 2500. Furthermore, display screen 2505 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 2505 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0434] The camera assembly 2506 is used to acquire images or videos. In some embodiments, the camera assembly 2506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 2506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0435] The audio circuit 2507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 2501 for processing, or to the radio frequency circuit 2504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 2500. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 2501 or the radio frequency circuit 2504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 2507 may also include a headphone jack.

[0436] Power supply 2508 is used to power the various components in terminal 2500. Power supply 2508 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 2508 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0437] Those skilled in the art will understand that Figure 25 The structure shown does not constitute a limitation on terminal 2500 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0438] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 2502 including instructions, which can be executed by a processor 2501 of a terminal 2500 to complete the video generation method described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0439] A computer program product includes a computer program / instructions that, when executed by a processor, implement the video generation method described above.

[0440] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0441] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that, The method comprises: The video editing interface is displayed, which includes material generation controls and a video track. In response to a trigger operation on the material generation control, artificial intelligence technology is used to generate a first material, which includes at least one of image material and video material; The first footage is displayed in the video track; In response to an extension operation on any video clip in the video track, a video extension panel is displayed; Based on the extension prompt information in the video extension panel, incremental video material matching the extension prompt information is displayed in the video track. The extension prompt information includes at least one of prompt text and prompt image. The extension prompt information is used to indicate at least one of the content and style of the incremental video material to be generated. The incremental video material is located after the video material and is connected to the video material. In response to the video generation operation, a video is generated based on all the footage in the video track.

2. The video generation method according to claim 1, characterized in that, In response to a trigger operation on the material generation control, the generation of first material using artificial intelligence technology includes: In response to a trigger operation on the material generation control, the material generation panel is displayed; Obtain prompt information, the prompt information including at least one of prompt text and prompt image, the prompt information being used to indicate at least one of the content and style of the material to be generated; The prompt message is displayed in the material generation panel; In response to the confirmation and generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information.

3. The video generation method according to claim 2, characterized in that, The material generation control includes at least one of an image generation control, a video generation control, and a batch generation control; the material generation panel includes at least one of an image generation panel, a video generation panel, and a batch generation panel. In response to the confirmation generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information, including any one of the following: In response to the confirmation generation operation in the image generation panel, based on the prompt information in the image generation panel, artificial intelligence technology is used to generate image materials that match the prompt information; In response to the confirmation and generation operation in the video generation panel, based on the prompt information in the video generation panel, artificial intelligence technology is used to generate video material that matches the prompt information; In response to the confirmation and generation operation in the batch generation panel, based on multiple prompts in the batch generation panel, artificial intelligence technology is used to generate image or video materials that match each of the multiple prompts.

4. The video generation method according to claim 2, characterized in that, The video editing interface also displays other tracks besides the video track; The process of obtaining the prompt information includes: If corresponding materials have already been added to the other tracks, the prompt message is determined based on the materials in the other tracks.

5. The video generation method according to claim 4, characterized in that, The material generation panel includes at least one of an image generation panel and a video generation panel; When corresponding materials have already been added to the other tracks, the prompt information is determined based on the materials in the other tracks, including: If corresponding material has been added to the other tracks, based on the position indicated by the timeline in the other tracks, determine the second material at that position from the other tracks; Based on the second material, the prompt message is determined.

6. The video generation method according to claim 5, characterized in that, Displaying the first material in the video track includes: The first material is displayed on the video track according to the position of the second material on the other tracks.

7. The video generation method according to claim 5, characterized in that, The prompt information includes the second material, the length of the first material is equal to the length of the second material, and the length of the material is used to indicate the display duration of the material.

8. The video generation method according to claim 4, characterized in that, The material generation panel includes a batch generation panel; When corresponding materials have already been added to the other tracks, the prompt information is determined based on the materials in the other tracks, including: If corresponding materials have been added to the other tracks, the prompt information corresponding to each of the multiple second materials in the other tracks is determined respectively.

9. The video generation method according to claim 8, characterized in that, In response to the confirmation generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information, including: In response to the confirmation and generation operation, based on the prompt information corresponding to each of the multiple second materials in the other tracks, artificial intelligence technology is used to generate the first material corresponding to each prompt information. Displaying the first material in the video track includes: Following the order of multiple second materials in the other tracks, the first material corresponding to each prompt message is displayed in the video track.

10. The video generation method according to claim 2, characterized in that, The video editing interface also displays other tracks besides the video track, and the material generation panel includes a batch generation panel; The method further includes: For each second material in the other tracks of the material generation panel, display the corresponding editing control for the prompt information; In response to a trigger operation on the editing control, an editing panel for editing the prompt information is displayed, the editing panel being used to edit the prompt information.

11. The video generation method according to claim 10, characterized in that, In addition to the video track, the video editing interface also displays several other tracks; The process of obtaining the prompt information includes: In response to the selection operation of any other track in the video editing interface, based on each second material in the other track, the prompt information corresponding to each second material is determined respectively.

12. The video generation method according to claim 2, characterized in that, The material generation panel includes a batch generation panel, and the method further includes: For any prompt message in the material generation panel, in response to the type selection operation of the prompt message, the material type corresponding to the prompt message is displayed in the material generation panel; In response to the confirmation generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information, including: In response to the confirmation of the generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information and conforms to the corresponding material type.

13. The video generation method according to claim 12, characterized in that, In response to any prompt message in the material generation panel, and in response to a type selection operation for the prompt message, the material type corresponding to the prompt message is displayed in the material generation panel, including: For any prompt message in the material generation panel, display the material selection control for that prompt message in the material generation panel; In response to a trigger operation on the material selection control, image type options and video type options are displayed; When either the image type option or the video type option is selected, the media type indicated by the selected type option is displayed in the media generation panel.

14. The video generation method according to claim 12, characterized in that, In response to any prompt message in the material generation panel, and in response to a type selection operation for the prompt message, the material type corresponding to the prompt message is displayed in the material generation panel, including: In response to the selection of any number of prompts in the material generation panel, image type options and video type options are displayed; When either the image type option or the video type option is selected, the media type indicated by the selected type option is displayed in the media generation panel.

15. The video generation method according to claim 2, characterized in that, The material generation panel displays a template selection control; The process of obtaining the prompt information includes: In response to a trigger operation on the template selection control, multiple text templates are displayed; In response to the application operation of any text template, the text template is determined as the prompt message.

16. The video generation method according to claim 2, characterized in that, The material generation panel displays an image acquisition control; The process of obtaining the prompt information includes: In response to a trigger operation on the image acquisition control, multiple images are displayed; In response to a selection operation of at least one of the plurality of images, the prompt information is determined based on the at least one image.

17. The video generation method according to claim 16, characterized in that, In response to a selection operation of at least one of the plurality of images, determining the prompt information based on the at least one image includes: For any one of the plurality of images, in response to the selection operation of the image, the image is displayed in the prompt editing interface; In response to the prompt selection operation of the image in the prompt editing interface, the prompt area in the image is determined, and the image and the regional prompt information used to indicate the prompt area are determined as the prompt information.

18. The video generation method according to claim 16, characterized in that, The method further includes: In response to a text input operation in the material generation panel, a prompt text for the input is displayed in the material generation panel; The action of displaying multiple images in response to a triggering operation on the image acquisition control includes any one of the following: In response to a trigger operation on the image acquisition control, the plurality of images are displayed based on the prompt text in the material generation panel, wherein at least one of the content and style of the plurality of images conforms to the prompt text; In response to a trigger operation on the image acquisition control, the plurality of images are displayed based on the prompt text in the material generation panel, and at least one of the plurality of images whose content and style conform to the prompt text is highlighted.

19. The video generation method according to claim 2, characterized in that, The material generation panel includes a video generation panel, the prompt information in the video generation panel includes multiple images, and the video generation panel displays a first and last frame mode control; In response to the confirmation generation operation, artificial intelligence technology is used to generate the first material that matches the prompt information, including: When the first and last frame mode control is triggered, in response to the confirmation generation operation, for any two adjacent images in the plurality of images in the video generation panel, an interpolation algorithm is used to process the two images to obtain a transition image between the two images; The first material is generated based on the plurality of images in the video generation panel and the transition images of the plurality of images.

20. The video generation method according to claim 2, characterized in that, The method further includes: The first material is displayed in the material generation panel; Displaying the first material in the video track includes: In response to the selection of the first clip, if the timeline is in an empty area of ​​the video track, the first clip is displayed after any clip already added to the video track, where the empty area is a region where no clip exists in the video track; or... In response to the selection of the first clip, if the timeline is in a non-idle area of ​​the video track, the first clip is displayed between the two clips closest to the timeline in the video track.

21. The video generation method according to claim 2, characterized in that, The method further includes: During the generation of the first material, a minimize control is displayed in the material generation panel; In response to a trigger operation on the minimize control, the material generation panel is minimized; A first target area is highlighted in the video track, the first target area being used to indicate the position to be occupied by the first material in the video track; Displaying the first material in the video track includes: The first material is displayed in the first target area.

22. The video generation method according to claim 1, characterized in that, The method further includes: In response to the update operation on the first material in the video track, the third material is regenerated; Replace the first material in the video track with the third material.

23. The video generation method according to claim 1, characterized in that, The method further includes: For any image material in the video track, in response to the conversion operation of the image material, a video material is generated based on the content and length of the image material. The content of the video material is related to the content of the image material, and the length of the video material conforms to a preset ratio with the length of the image material. The length of the material is used to represent the display duration of the material. Replace the image material in the video track with the video material.

24. The video generation method according to claim 1, characterized in that, The method of displaying a video extension panel in response to an extension operation on any video clip in the video track includes any of the following: In response to a trigger operation on any video clip in the video track, a time window of the video clip is displayed in the video track. The length of the time window represents the display duration of the video clip. When the length of the time window is equal to the maximum duration of the video clip and a lengthening operation on the time window is detected, the video extension panel is displayed. For any video clip in the video track, in response to the editing operation of the video clip, an extension control is displayed in the video editing interface, and in response to the triggering operation of the extension control, the video extension panel is displayed.

25. The video generation method according to claim 1, characterized in that, The incremental video material is at least one of the following: The incremental video material is generated based on the prompt text entered in the video extension panel; The incremental video material is generated based on the prompt image input in the video extension panel, and the prompt image is the image located in the last frame of the video material.

26. The video generation method according to claim 1, characterized in that, The step of displaying incremental video footage matching the extension prompt information in the video track based on the extension prompt information in the video extension panel includes: During the process of generating the incremental video material based on the extended prompt information, a second target area is highlighted in the video track. The second target area is used to indicate the position to be occupied by the incremental video material in the video track. The second target area is located after the area occupied by the video material and is adjacent to the area occupied by the video material. After the incremental video material is generated, the incremental video material is displayed in the second target area.

27. The video generation method according to claim 1, characterized in that, The step of displaying incremental video footage matching the extension prompt information in the video track based on the extension prompt information in the video extension panel includes: During the process of generating the incremental video material based on the extended prompt information, after the video material in the video track, the images generated in the incremental video material are displayed frame by frame until the generation of the incremental video material is completed.

28. The video generation method according to claim 1, characterized in that, The method further includes: In response to the transition material generation operation, determine the fourth and fifth materials indicated by the transition material generation operation; When the fifth material is located after the fourth material, a transition material is generated based on the last frame image of the fourth material and the first frame image of the fifth material; or, when the fifth material is located before the fourth material, the transition material is generated based on the first frame image of the fourth material and the last frame image of the fifth material. The transition material is displayed between the fourth material and the fifth material.

29. The video generation method according to claim 28, characterized in that, The step of determining the fourth and fifth materials indicated by the transition material generation operation in response to the transition material generation operation includes: In response to a trigger operation on the transition generation control in the video editing interface, a transition generation panel is displayed. The transition generation panel displays a first generation option and a second generation option. The first generation option is used to indicate the generation of transition material between the fourth material and the material preceding the fourth material. The second generation option is used to indicate the generation of transition material between the fourth material and the material following the fourth material. The fourth material is the material indicated by the time progress line in the video track. When the first generation option is selected, the material preceding the fourth material is determined as the fifth material; If the second generation option is selected, the material following the fourth material will be designated as the fifth material.

30. The video generation method according to claim 29, characterized in that, The step of displaying a transition generation panel in response to a trigger operation on the transition generation control in the video editing interface includes: If the distance between the first source material and the second source material is greater than the distance between the second source material, in response to the triggering operation of the transition generation control in the video editing interface, the first generation option is displayed as selected in the transition generation panel; When the distance between the first material and the second material is less than the distance between the second material, in response to the triggering operation of the transition generation control in the video editing interface, the second generation option is displayed as selected in the transition generation panel; Wherein, the first material distance is the distance between the position indicated by the time progress line in the video track and the material preceding the fourth material, and the second material distance is the distance between the position indicated by the time progress line in the video track and the material following the fourth material.

31. A video generation apparatus, characterized in that, The device comprises: The display unit is configured to display a video editing interface, which displays material generation controls and a video track; The first generation unit is configured to perform a trigger operation in response to the material generation control, and generate a first material using artificial intelligence technology, wherein the first material includes at least one of image material and video material; The display unit is also configured to display the first material in the video track; The display unit is further configured to perform a video extension panel in response to an extension operation on any video clip in the video track; based on the extension prompt information in the video extension panel, to display incremental video clips in the video track that match the extension prompt information, wherein the extension prompt information includes at least one of prompt text and prompt image, and the extension prompt information is used to indicate at least one of the content and style of the incremental video clip to be generated, wherein the incremental video clip is located after the video clip and is adjacent to the video clip; The second generation unit is configured to perform a video generation operation in response to the video generation operation, generating a video based on all the footage in the video track.

32. An electronic device, characterized in that, The electronic device includes: one or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the video generation method as described in any one of claims 1 to 30.

33. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video generation method as described in any one of claims 1 to 30.

34. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method according to any one of claims 1 to 30.

Images