Video modification method and device, equipment, storage medium and product

Abstract

The invention discloses a video modification method and device, equipment, a storage medium and a product, and the method comprises the steps: responding to a modification triggering instruction triggered by an original video generated based on a large model, and displaying all links of the original video generated by the large model and video materials of all the links; under the condition that a material modification instruction is received, extracting a to-be-modified material and modification prompt information from the material modification instruction; modifying the to-be-modified material in the target link through the large model according to the modification prompt information to obtain a modified material; and performing video generation again based on the modification material through the large model. According to the video generation method and device, the video generation process can be deconstructed, each link can be displayed in a visual mode, and the video material of any link is allowed to be directly modified in real time, so that the flexibility of video generation is improved, the period of video verification is shortened, and the trial and error cost is reduced.

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to video modification methods, apparatus, devices, storage media, and products. Background Technology

[0002] With the rapid development of artificial intelligence, especially generative large model technology, AI-driven video content creation is gradually moving from science fiction to reality. However, related AI video generation tools generally adopt a "black box" workflow, that is, strictly following a preset generation order, generating intermediate results step by step, and finally compositing them into a video. If the generated video does not meet the user's expectations, it needs to go through the entire long and computationally intensive generation process again, resulting in low video generation efficiency. Summary of the Invention

[0003] The main purpose of this application is to provide a video modification method, apparatus, device, storage medium, and product, aiming to solve the technical problem of low flexibility in existing AI video generation.

[0004] To achieve the above objectives, this application proposes a video modification method, which includes: In response to a modification trigger command triggered by the original video generated based on the large model, the various stages of the generation of the original video by the large model and the video materials of each stage are displayed. Upon receiving a material modification instruction, the material to be modified and the modification prompt information are extracted from the material modification instruction; Based on the modification prompts, the material to be modified in the target stage is modified using the large model to obtain the modified material; The video is regenerated based on the modified material using the large model.

[0005] Optionally, the step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: When the material modification instruction is to modify the storyboard video, the modification type of the storyboard video is determined according to the modification prompt information; When the storyboard video modification type is video frame modification, the video frame to be modified is determined according to the modification prompt information and the material to be modified; Based on the modification prompts, the video frame to be modified is modified using the large model to obtain the modified material.

[0006] Optionally, the step of modifying the video frame to be modified using the large model based on the modification prompt information to obtain the modified material includes: Based on the modification prompt information, the video frame to be modified is modified using the large model to obtain the modified video frame; Based on the material to be modified, the adjacent video frames of the modified video frame are determined using the large model; Based on the modified video frames, the adjacent video frames are modified using the large model to obtain the modified material.

[0007] Optionally, the step of modifying adjacent video frames using the large model based on the modified video frame to obtain modified material includes: Based on the modified video frames, the adjacent video frames are modified using the large model to obtain a modified video frame sequence. The visual style of the material to be modified is determined using the large model; Based on the aforementioned visual style, the video frame sequence is modified, and the material to be modified is used to generate a storyboard video through the aforementioned large model to obtain the modified material.

[0008] Optionally, the step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: When the material modification instruction is to modify the storyboard script, the large model determines the modified entity and modification parameter information based on the modification prompt information; Based on the modified entity and the modified parameter information, the material to be modified in the storyboard generation stage is modified using the large model to obtain the modified material.

[0009] Optionally, the step of regenerating the video based on the modified material using the large model includes: When the modified material is a storyboard material, the location information of the material to be modified in the original video is determined by the large model; Based on the location information and the modified material, the video is regenerated using the large model.

[0010] Optionally, prior to responding to the modification trigger instruction triggered by the original video generated based on the large model, the method further includes: In response to the triggered overall video effect preview command, determine the video generation progress; If the video generation progress meets the preset preview conditions, determine the scene information for each scene. A preview video is generated based on the storyboard information.

[0011] Optionally, the storyboard information includes the storyboard stage corresponding to each storyboard, and the storyboard stage includes a structured storyboard script stage and a visual storyboard preview stage; The step of generating a preview video based on the storyboard information includes: Identify the first storyboard in the visual storyboard preview stage and the second storyboard in the structured storyboard script stage; Determine the visual storyboard of the first storyboard and the structured storyboard script of the second storyboard; A preview video is generated based on the visual storyboard and the structured storyboard script.

[0012] Optionally, generating a preview video based on the visualized storyboard and the structured storyboard script includes: Generate a second storyboard video frame based on the structured storyboard script; The first storyboard video frame is determined based on the visualized storyboard. A preview video is generated based on the second and first storyboard video frames.

[0013] Optionally, the video generation process of the original video includes at least one of the following: storyboard generation, prompt generation, style setting, still image generation, and storyboard video generation.

[0014] Optionally, the step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: Determine the modification category and modification intent corresponding to the modification prompt information; Generate material modification prompts based on the modification category and the modification intent; Based on the modified prompts, the modified material is obtained by modifying the material to be modified in the target stage using the large model.

[0015] Optionally, the step of displaying the various stages of generating the original video from the large model and the video footage of each stage includes: The process of generating the original video using the large model and the video footage for each stage are determined. The video materials are displayed hierarchically based on the aforementioned steps.

[0016] Furthermore, to achieve the above objectives, this application also proposes a video modification device, the video modification device comprising: The response module is used to respond to the modification trigger command triggered by the original video generated based on the large model, and to display the various stages of the generation of the original video by the large model and the video materials of each stage; The extraction module is used to extract the material to be modified and the modification prompt information from the material modification instruction when a material modification instruction is received; The material modification module is used to modify the material to be modified in the target stage through the large model according to the modification prompt information, so as to obtain the modified material; The video regeneration module is used to regenerate the video based on the modified material using the large model.

[0017] Optionally, the material modification module is further configured to determine the storyboard video modification type based on the modification prompt information when the material modification instruction is to modify the storyboard video; When the storyboard video modification type is video frame modification, the video frame to be modified is determined according to the modification prompt information and the material to be modified; Based on the modification prompts, the video frame to be modified is modified using the large model to obtain the modified material.

[0018] Optionally, the material modification module is further configured to modify the video frame to be modified using the large model according to the modification prompt information, so as to obtain the modified video frame; Based on the material to be modified, the adjacent video frames of the modified video frame are determined using the large model; Based on the modified video frames, the adjacent video frames are modified using the large model to obtain the modified material.

[0019] Optionally, the material modification module is further configured to modify the adjacent video frames according to the modified video frames using the large model to obtain a modified video frame sequence; The visual style of the material to be modified is determined using the large model; Based on the aforementioned visual style, the video frame sequence is modified, and the material to be modified is used to generate a storyboard video through the aforementioned large model to obtain the modified material.

[0020] Optionally, the material modification module is further configured to, when the material modification instruction is to modify the storyboard script, determine the modified entity and modification parameter information based on the modification prompt information through the large model; Based on the modified entity and the modified parameter information, the material to be modified in the storyboard generation stage is modified using the large model to obtain the modified material.

[0021] Optionally, the video regeneration module is further configured to determine the position information of the material to be modified in the original video using the large model when the modified material is a storyboard material; Based on the location information and the modified material, the video is regenerated using the large model.

[0022] Optionally, the response module is further configured to determine the video generation progress in response to a triggered overall video effect preview command; If the video generation progress meets the preset preview conditions, determine the scene information for each scene. A preview video is generated based on the storyboard information.

[0023] Optionally, the storyboard information includes the storyboard stage corresponding to each storyboard, and the storyboard stage includes a structured storyboard script stage and a visual storyboard preview stage; The response module is also used to determine the first storyboard in the visual storyboard preview stage and the second storyboard in the structured storyboard script stage; Determine the visual storyboard of the first storyboard and the structured storyboard script of the second storyboard; A preview video is generated based on the visual storyboard and the structured storyboard script.

[0024] Optionally, the response module is further configured to generate a second storyboard video frame based on the structured storyboard script; The first storyboard video frame is determined based on the visualized storyboard. A preview video is generated based on the second and first storyboard video frames.

[0025] Optionally, the video generation process of the original video includes at least one of the following: storyboard generation, prompt generation, style setting, still image generation, and storyboard video generation.

[0026] Optionally, the material modification module is further configured to determine the modification category and modification intent corresponding to the modification prompt information; Generate material modification prompts based on the modification category and the modification intent; Based on the modified prompts, the modified material is obtained by modifying the material to be modified in the target stage using the large model.

[0027] Optionally, the response module is further configured to determine each stage of the large model generating the original video and the video material for each stage; The video materials are displayed hierarchically based on the aforementioned steps.

[0028] In addition, to achieve the above objectives, this application also proposes a video modification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video modification method as described above.

[0029] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the video modification method described above.

[0030] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the video modification method described above.

[0031] This application responds to a modification trigger command triggered by the original video generated based on a large model, and displays each stage of the large model's generation of the original video and the video materials of each stage; upon receiving a material modification command, it extracts the material to be modified and modification prompt information from the material modification command; based on the modification prompt information, it modifies the material to be modified in the target stage using the large model to obtain the modified material; and then regenerates the video using the large model based on the modified material. This application can deconstruct the video generation process, display each stage in a visual manner, and allow direct, real-time modification of video materials at any stage, improving the flexibility of video generation, shortening the video verification cycle, and reducing trial and error costs. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the video modification method of this application in Embodiment 1. Figure 2 This is a schematic diagram of the page display effect provided in Embodiment 1 of the video modification method of this application; Figure 3 This is a static image preview diagram provided for Embodiment 1 of the video modification method of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the video modification method of this application; Figure 5 This is a flowchart illustrating Embodiment 3 of the video modification method of this application; Figure 6This is a schematic diagram of the module structure of the video modification device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the video modification method in this application embodiment.

[0035] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0036] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0037] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0038] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or video editing device capable of performing the above functions. The following description uses a video editing device as an example to illustrate this embodiment and the subsequent embodiments.

[0039] Based on this, this application provides a video modification method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the video modification method of this application.

[0040] In this embodiment, the video modification method includes the following steps: Step S10: In response to the modification triggering command triggered by the original video generated based on the large model, the various stages of the large model generating the original video and the video materials of each stage are displayed. It should be noted that the original video can be a video generated by a large model based on user-input video generation instructions. The large model can be an artificial intelligence model with powerful understanding and generation capabilities, such as a generative AI model. The modification trigger instruction can be a command issued by the user based on the original video to modify it, which can involve modifying the overall style of the original video or video footage at various stages. Each stage in generating the original video can be a series of continuous, logically dependent steps or phases that the large model undergoes when generating the original video. The video footage at each stage can be the intermediate product corresponding to each stage. For example, in the script generation stage, the video footage can be the generated script; in the storyboard generation stage, the video footage can be the generated storyboard video.

[0041] Furthermore, the video generation process of the original video includes at least one of the following: storyboard generation, prompt generation, style setting, still image generation, and storyboard video generation.

[0042] It should be noted that the video footage corresponding to the storyboard generation stage can be the generated storyboards; the video footage corresponding to the cue word generation stage can be the generated AI cue words; the video footage corresponding to the style setting stage can be style cue words or style reference images, used to indicate the style of the final video generated by the large model; the video footage corresponding to the still image generation stage can be the image files corresponding to each storyboard; in the single storyboard video generation, the video footage can be the image files used to generate the final video; and the video footage corresponding to the storyboard video generation stage can be video clips from each storyboard.

[0043] To facilitate users' more efficient viewing and modification of each stage, the process of generating the original video from the large model and the video footage for each stage are displayed, including: The process of generating the original video using the large model and the video footage for each stage are determined. The video materials are displayed hierarchically based on the aforementioned steps.

[0044] It should be noted that the hierarchical display of the video materials based on the aforementioned stages can be a visual navigation structure built around the sequence of video generation stages. When a user clicks on a stage, the corresponding video materials under that stage are displayed, along with corresponding input boxes or edit buttons for easy modification. For example, the navigation structure displayed in the sidebar of the interface can be a vertical navigation structure. After the user clicks on the corresponding stage in the navigation, the video materials for that stage will be displayed on the main interface. Furthermore, the way users input material modification commands differs for different video materials. For example, for script materials, users can directly intervene and modify based on the displayed script. For instance, if the script is "Forest in the Daytime," the user can directly delete "Daytime" in the text box of the displayed script and change it to "Nighttime." For storyboard videos, users can drag the video timeline and input material modification commands in the input boxes below the corresponding video frames, or directly modify the positions of entities appearing in the video frames and between entities by smearing, dragging, or other methods on the video frames.

[0045] In specific implementation, it can be referred to Figure 2 , Figure 2 This is a schematic diagram of the page display effect provided in Embodiment 1 of the video modification method of this application; Figure 2The section shown here is the storyboard generation stage. The homepage displays the scripts for each storyboard generated from the large model, presented through editable text boxes. Users can directly locate specific text within these boxes and perform modifications, deletions, and other operations. For other stages, please refer to [link / reference]. Figure 3 , Figure 3 This is a static image preview diagram provided for Embodiment 1 of the video modification method of this application; Figure 3 The homepage displays static images generated by a script. Figure 3 The system displays one or more static images at the same time. Below each image are buttons for deletion, replacement, and regeneration. Users can click these buttons to directly trigger editing instructions or enter editing prompts in the text boxes.

[0046] Step S20: Upon receiving a material modification instruction, extract the material to be modified and modification prompt information from the material modification instruction; It should be noted that the material to be modified can be video material that needs to be modified, and can be video material from any stage. The modification prompt information can be the user's modification request, such as changing the background to daytime, changing the style, compressing the video length to 10 seconds, deleting the third shot, etc.

[0047] Step S30: Modify the material to be modified in the target stage using the large model according to the modification prompt information to obtain the modified material; It should be noted that the modified material can be a new version of the material generated by the large model after processing the material to be modified based on the modification prompt information. For example, if the material to be modified is a prompt word, specifically "forest at night," and the modification prompt information is "change to daytime," then the modified material can be "forest during the day." The process of modifying the material to be modified in the target stage according to the modification prompt information using the large model to obtain the modified material can be achieved by modifying the material to be modified based on the modification prompt information using the large model. The target stage can be the video generation stage corresponding to the material to be modified.

[0048] Furthermore, for different video generation stages, the process of modifying the material to be modified and obtaining the modified material may be different. Step S30 may include: when the material modification instruction is to modify the storyboard, determining the modified entity and modification parameter information based on the modification prompt information through the large model; Based on the modified entity and the modified parameter information, the material to be modified in the storyboard generation stage is modified using the large model to obtain the modified material.

[0049] It should be noted that the modified entity can be the smallest data unit that needs to be modified, such as characters, environment, objects, etc. For a structured (e.g., JSON format) storyboard, it contains multiple shots, and each shot contains fields such as shot ID (shot_id), duration, description, and prompt. The modified entity can be these specific fields or objects. For example, if the user command is "modify the duration of the third shot", then the modified entity is scenes[2].duration_seconds. The modified parameter information can be the specific modification requirements provided by the user based on the modified entity. For example, if the number of characters in the storyboard is controlled to be within 3, then the modified entity can be the number of characters, and the modified parameter information can be within 3. The modification of the material to be modified in the storyboard generation stage based on the modified entity and the modified parameter information through the large model to obtain the modified material can be achieved by locating the modified entity in the material to be modified through the large model, modifying the modified entity based on the modified parameter information, and modifying other script information associated with the modified entity to obtain the modified material.

[0050] In practice, the user-input material modification command is: change the duration of the third shot (shot ID: shot_003) from 4 seconds to 5 seconds, and change 'playing guitar' in the description to 'playing piano'. The video modification device calls a large model to complete the following tasks: Intent recognition: understand the user's desired action to modify the storyboard script. Entity recognition: based on the material modification command, parse the target object to be modified from "the third shot" as the data object shot_003. From the words duration and description, identify the entity to be modified (modification fields) as duration_seconds and description. Parameter extraction: extract the new value of duration_seconds (5) from "change to 5 seconds". Extract the modification parameter information of the description field from "change 'playing guitar' to 'playing piano'" as: replace "playing guitar" with "playing piano". Material modification: modify the entity based on the modification parameter information to obtain the modified material.

[0051] Furthermore, step S30 may include: determining the modification category and modification intent corresponding to the modification prompt information; Generate material modification prompts based on the modification category and the modification intent; Based on the modified prompts, the modified material is obtained by modifying the material to be modified in the target stage using the large model.

[0052] It should be noted that the modification prompts can be user-inputted information, such as: "This video is too dark, and the protagonist's clothes are unattractive. Could you change the background to a cyberpunk style?". The modification categories can include global attributes, content entities, and spatial structures. Global attributes can be video style, video brightness, etc.; content entities can be modifications to a specific object in the frame, such as changing the background or adding elements; spatial structures can be parameters of the camera itself, such as camera movement, stability, and duration. The modification intent can be the user's specific modification request, such as increasing the overall brightness by 50%, replacing the protagonist's clothes, or changing the visual style. The material modification prompts can be instructions specifically generated based on the modification categories and modification intents to drive large-scale models.

[0053] Step S40: Regenerate the video based on the modified material using the large model.

[0054] It should be noted that the process of regenerating the video based on the modified material using the large model can be described as modifying the original video based on the modified material. Specifically, the large model needs to determine the target stage corresponding to the modified material, and starting from the target stage, rerun the subsequent video generation process based on the modified material to finally synthesize a new and updated video.

[0055] Furthermore, in order to improve the efficiency of video regeneration, step S40 may include: in the case that the modified material is a storyboard material, determining the position information of the material to be modified in the original video through the large model; Based on the location information and the modified material, the video is regenerated using the large model.

[0056] It should be noted that the storyboard footage can be storyboard videos generated by a large model based on a script. The location information can be the position of the storyboard video within the original video, for example, the storyboard video between the 10th and 20th seconds of the original video. Regenerating the video using the large model based on the location information and the modified footage can involve replacing the footage to be modified in the original video with the modified footage, or replacing the footage between the 10th and 20th seconds of the original video with the modified footage to obtain a regenerated video.

[0057] It should be understood that the processing steps for regenerating the video in the large model are different depending on the modification stage. For example, if only the storyboard video is modified, the corresponding positions of the storyboard in the original video are replaced with the modified material of the storyboard video, and the overall harmony is adjusted. However, for the modification in the script generation stage, since the video generation script is modified, it is also necessary to regenerate the storyboard static images and the storyboard video based on the modified script.

[0058] This embodiment responds to a modification trigger command triggered by the original video generated based on a large model, displaying each stage of the large model's generation of the original video and the video materials for each stage. Upon receiving a material modification command, it extracts the material to be modified and modification prompt information from the command. Based on the modification prompt information, it modifies the material to be modified in the target stage using the large model to obtain the modified material. Finally, it regenerates the video using the large model based on the modified material. This embodiment deconstructs the video generation process, visually displaying each stage, and allows direct, real-time modification of video materials at any stage, improving the flexibility of video generation, shortening the video verification cycle, and reducing trial-and-error costs.

[0059] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating Embodiment 2 of the video modification method of this application. Step S30 further includes the following steps: Step S301: If the material modification instruction is to modify the storyboard video, determine the storyboard video modification type according to the modification prompt information; It should be noted that the modification of the storyboard video can refer to modifications made to the storyboard video generated during the storyboard video generation process. Modifications to the storyboard video include altering video frames within the storyboard and making overall modifications to the storyboard video's style, sound, environment, etc. The type of storyboard video modification can be determined based on semantic understanding of the modification prompts.

[0060] Step S302: When the storyboard video modification type is video frame modification, determine the video frame to be modified according to the modification prompt information and the material to be modified; It should be noted that the video frame to be modified can be one or more images in the storyboard video that need to be modified. Determining the video frame to be modified based on the modification prompt information and the material to be modified can be done by determining the positional information or feature information of the video frame to be modified based on the modification prompt information, and then locating the corresponding video frame to be modified within the material to be modified based on the positional information or feature information. For example, if the modification prompt information is to delete the large tree in the material to be modified, then the video frames to be modified are all video frames in the material to be modified that contain the large tree; if the modification prompt information is to delete the first 2 seconds of the material to be modified, then the video frames to be modified are all video frames corresponding to the first 2 seconds of the video in the material to be modified.

[0061] Step S303: Modify the video frame to be modified using the large model according to the modification prompt information to obtain the modified material.

[0062] It should be noted that modifying the video frame to be modified using the large model based on the modification prompt information to obtain the modified material can involve adjusting the video frame to be modified according to the user's requirements in the modification prompt information to obtain an adjusted video frame, and then determining the modified material based on the adjusted video frame and other video frames in the material to be modified. For example, if the modification prompt information is to delete the large tree in the material to be modified, then the video frames to be modified are all video frames in the material to be modified that contain the large tree. Deleting the large tree from the video frames to be modified yields an adjusted video frame, and then replacing the original video frame in the material to be modified with the adjusted video frame yields the modified material.

[0063] Furthermore, if only the video frame to be modified is modified, there may be problems with the smoothness of video connection in the modified material, such as abrupt transitions between video frames. Therefore, step S303 may include: modifying the video frame to be modified through the large model according to the modification prompt information to obtain the modified video frame; Based on the material to be modified, the adjacent video frames of the modified video frame are determined using the large model; Based on the modified video frames, the adjacent video frames are modified using the large model to obtain the modified material.

[0064] It should be noted that the modified video frame can be a single static frame that has already been successfully modified. Determining the adjacent video frames of the modified video frame based on the material to be modified using the large model can be achieved by extracting several frames surrounding the modified video frame from the material to be modified using the large model. Specifically, this can involve extracting all video frames within 5 seconds before and after the modified video frame, i.e., adjacent video frames. If the time before and after the modified video frame is less than 5 seconds, then all adjacent video frames are extracted. Modifying the adjacent video frames using the large model based on the modified video frame to obtain the modified material can be done by modifying the adjacent video frames based on the scene information of the modified video frame, making the transition between the modified video frame and its adjacent video frames smoother. Specifically, this can be done by first determining a video segment based on the modified video frame and adjacent video frames, then analyzing the smoothness of the transition between video frames in the video segment using the large model, and checking for obvious misalignments or stutters in lighting, objects, or motion speed. If any are found, the video segment is optimized to obtain a smooth, seamless video segment. Finally, the video segment is inserted into the corresponding position of the material to be modified to obtain the modified material.

[0065] Furthermore, to ensure stylistic consistency across the modified materials, the process of modifying adjacent video frames using the large model based on the modified video frames to obtain the modified materials includes: Based on the modified video frames, the adjacent video frames are modified using the large model to obtain a modified video frame sequence. The visual style of the material to be modified is determined using the large model; Based on the aforementioned visual style, the video frame sequence is modified, and the material to be modified is used to generate a storyboard video through the aforementioned large model to obtain the modified material.

[0066] It should be noted that, based on the modified video frames, the adjacent video frames are modified using the large model to obtain a modified video frame sequence. This can be achieved by using the modified video frames as references to modify adjacent video frames, resulting in a set of coordinated, modified static images, i.e., a modified video frame sequence. The visual style can include information such as style reference images and style keywords for the material to be modified. The process of generating storyboard video based on the visual style, the modified video frame sequence, and the material to be modified using the large model to obtain modified material can be achieved by replacing corresponding video frames in the material to be modified with the modified video frame sequence using the large model, obtaining modified video clips, and then adjusting the video clips overall based on the visual style to ensure a unified style and smooth transitions between video frames, thus obtaining modified material.

[0067] In this embodiment, when the material modification instruction is to modify the storyboard video, the modification type of the storyboard video is determined according to the modification prompt information; when the modification type of the storyboard video is to modify a video frame, the video frame to be modified is determined according to the modification prompt information and the material to be modified; the video frame to be modified is modified using the large model according to the modification prompt information to obtain the modified material.

[0068] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating the video modification method of this application in Embodiment 3. Before step S10, the following steps are also included: Step S001: In response to the triggered video overall effect preview command, determine the video generation progress; It should be noted that the overall video effect preview command can be a user-triggered command to generate the overall video effect. An overall video effect preview button can be displayed on the interface, which the user can click to trigger the preview. The video generation progress can include the current stage of the video generation task, such as the storyboard generation stage or the script generation stage.

[0069] Step S002: If the video generation progress meets the preset preview conditions, determine the scene information of each scene; It should be noted that the video generation progress meeting the preset preview conditions can occur when the video generation progress is after the script generation stage. If the video generation progress is in the script generation stage, it is necessary to wait for the video script generation stage to complete before generating the preview video. This is because the generation of the preview video depends entirely on the script generated in the video script generation stage. After the script generation stage, the storyboard generation stage begins. However, the storyboard generation stage includes a structured storyboard script stage and a visual storyboard preview stage. When in the structured storyboard script stage, there are no static images or storyboard videos for that storyboard. Therefore, the corresponding storyboard information is the structured storyboard script, which is not a visual; it only contains all the instructions for generating the visual. When the storyboard generation stage is in the visual storyboard preview stage, it indicates that a storyboard video or static images used to generate the storyboard video have already been generated.

[0070] Step S003: Generate a preview video based on the storyboard information.

[0071] It should be noted that the preview video generated from the storyboard information can be generated by creating preview storyboard images based on the structured storyboard script when the storyboard information is a structured storyboard script. These images are then spliced ​​together with static images or storyboard videos contained in other storyboard information. The splicing is performed according to the time sequence of each storyboard in the storyboard script to obtain the preview video.

[0072] Furthermore, the storyboard information includes the storyboard stage corresponding to each storyboard, and the storyboard stage includes a structured storyboard script stage and a visual storyboard preview stage; The step of generating a preview video based on the storyboard information includes: Identify the first storyboard in the visual storyboard preview stage and the second storyboard in the structured storyboard script stage; Determine the visual storyboard of the first storyboard and the structured storyboard script of the second storyboard; A preview video is generated based on the visual storyboard and the structured storyboard script.

[0073] It should be noted that the first storyboard can be a storyboard in the visualization storyboard preview stage, at which the large model has already generated the corresponding static images or storyboard videos. The second storyboard can be a storyboard in the structured storyboard script stage, at which the large model has not yet generated the corresponding static images. The visualization storyboard of the first storyboard can be the corresponding static image or storyboard video. The structured storyboard script of the second storyboard can be the script corresponding to the second storyboard. Generating a preview video based on the visualization storyboard and the structured storyboard script can be done by generating target static images based on the structured storyboard script, and then stitching the target static images and the visualization storyboard of the first storyboard in the chronological order of each storyboard to obtain the preview video.

[0074] Furthermore, the step of generating a preview video based on the visualized storyboard and the structured storyboard script includes: Generate a second storyboard video frame based on the structured storyboard script; The first storyboard video frame is determined based on the visualized storyboard. A preview video is generated based on the second and first storyboard video frames.

[0075] It should be noted that generating the second storyboard video frame based on the structured storyboard script can be based on generating one or more static images of preview-level quality. Preview-level quality can mean low image resolution. Determining the first storyboard video frame based on the visualized storyboard can be based on extracting several static images from the visualized storyboard as the first storyboard video frame. Generating a preview video based on the second and first storyboard video frames can be achieved by stitching the second and first storyboard video frames together in chronological order according to the corresponding scenes using a large model. The preview video can have a resolution of 720p or even lower, and a frame rate of 24fps or 30fps. Its purpose is not for final delivery, but for real-time feedback. Users can also modify trigger commands based on the preview video input.

[0076] In practice, users can edit directly on the preview video. For example, by dragging along the video timeline of the preview video to select the problematic shot (such as from the 3rd to the 5th second), they can directly enter a new, more precise prompt (i.e., modify the trigger command) on that shot, such as "[Shot 3: Close-up, astronaut elegantly plays electric guitar on Mars, with smooth and passionate finger movements]". At this point, the material to be modified is the shot selected by the user, and the modification prompt is the new, more precise prompt entered by the user on that shot.

[0077] This embodiment responds to a triggered video overall effect preview command, determines the video generation progress, and, if the video generation progress meets preset preview conditions, determines the storyboard information for each scene; then, generates a preview video based on the storyboard information. This embodiment enables overall effect previewing during the script generation stage, allowing users to quickly verify their ideas, greatly improving user experience and creative efficiency, while reducing computational power consumption and resource waste.

[0078] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the video modification method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0079] This application also provides a video editing device; please refer to [reference needed]. Figure 6 The video modification device includes: The response module 10 is used to respond to the modification trigger command triggered by the original video generated based on the large model, and to display the various stages of the generation of the original video by the large model and the video materials of each stage. Extraction module 20 is used to extract the material to be modified and modification prompt information from the material modification instruction when a material modification instruction is received; The material modification module 30 is used to modify the material to be modified in the target stage through the large model according to the modification prompt information, so as to obtain the modified material; The video regeneration module 40 is used to regenerate the video based on the modified material using the large model.

[0080] This embodiment responds to a modification trigger command triggered by the original video generated based on a large model, displaying each stage of the large model's generation of the original video and the video materials for each stage. Upon receiving a material modification command, it extracts the material to be modified and modification prompt information from the command. Based on the modification prompt information, it modifies the material to be modified in the target stage using the large model to obtain the modified material. Finally, it regenerates the video using the large model based on the modified material. This embodiment deconstructs the video generation process, visually displaying each stage, and allows direct, real-time modification of video materials at any stage, improving the flexibility of video generation, shortening the video verification cycle, and reducing trial-and-error costs.

[0081] The video modification apparatus provided in this application, employing the video modification method described in the above embodiments, can solve the technical problem of low flexibility in existing AI video generation. Compared with the prior art, the beneficial effects of the video modification apparatus provided in this application are the same as those of the video modification method provided in the above embodiments, and other technical features in the video modification apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0082] This application provides a video modification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the video modification method in Embodiment 1 above.

[0083] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a video modification device suitable for implementing embodiments of this application. The video modification device in this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7The video modification device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0084] like Figure 7 As shown, the video editing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the video editing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the video editing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show video editing devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0085] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0086] The video modification device provided in this application, employing the video modification method described in the above embodiments, can solve the technical problem of low flexibility in existing AI video generation. Compared with the prior art, the beneficial effects of the video modification device provided in this application are the same as those of the video modification method provided in the above embodiments, and other technical features of this video modification device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0087] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0089] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the video modification method described in the above embodiments.

[0090] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0091] The aforementioned computer-readable storage medium may be included in the video editing device; or it may exist independently and not assembled into the video editing device.

[0092] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0095] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video modification method, thereby solving the technical problem of low flexibility in existing AI video generation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video modification method provided in the above embodiments, and will not be repeated here.

[0096] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video modification method described above.

[0097] The computer program product provided in this application can solve the technical problem of low flexibility in existing AI video generation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the video modification method provided in the above embodiments, and will not be repeated here.

[0098] All user-related data (e.g., user input data) involved in this application were obtained with the user's permission or consent. In other words, when this application is applied to specific products or technologies, user permission is required to acquire and process the relevant data, and the processing of the data must comply with the relevant laws, regulations, and regulatory standards of the relevant countries and regions. For example, when it is necessary to obtain a user's current geographical location, a location acquisition prompt can be displayed on the user's terminal. After receiving confirmation from the user regarding the location acquisition prompt, the terminal can obtain the user's current geographical location.

[0099] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

[0100] This invention discloses A1. A video modification method, the video modification method comprising the following steps: In response to a modification trigger command triggered by the original video generated based on the large model, the various stages of the generation of the original video by the large model and the video materials of each stage are displayed. Upon receiving a material modification instruction, the material to be modified and the modification prompt information are extracted from the material modification instruction; Based on the modification prompts, the material to be modified in the target stage is modified using the large model to obtain the modified material; The video is regenerated based on the modified material using the large model.

[0101] A2. The video modification method as described in A1, wherein the step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: When the material modification instruction is to modify the storyboard video, the modification type of the storyboard video is determined according to the modification prompt information; When the storyboard video modification type is video frame modification, the video frame to be modified is determined according to the modification prompt information and the material to be modified; Based on the modification prompts, the video frame to be modified is modified using the large model to obtain the modified material.

[0102] A3. The video modification method as described in A2, wherein modifying the video frame to be modified using the large model based on the modification prompt information to obtain the modified material includes: Based on the modification prompt information, the video frame to be modified is modified using the large model to obtain the modified video frame; Based on the material to be modified, the adjacent video frames of the modified video frame are determined using the large model; Based on the modified video frames, the adjacent video frames are modified using the large model to obtain the modified material.

[0103] A4. The video modification method as described in A3, wherein the step of modifying adjacent video frames using the large model based on the modified video frame to obtain modified material includes: Based on the modified video frames, the adjacent video frames are modified using the large model to obtain a modified video frame sequence. The visual style of the material to be modified is determined using the large model; Based on the aforementioned visual style, the video frame sequence is modified, and the material to be modified is used to generate a storyboard video through the aforementioned large model to obtain the modified material.

[0104] A5. The video modification method as described in A1, wherein the modification of the material to be modified in the target stage is performed using the large model based on the modification prompt information to obtain the modified material, includes: When the material modification instruction is to modify the storyboard script, the large model determines the modified entity and modification parameter information based on the modification prompt information; Based on the modified entity and the modified parameter information, the material to be modified in the storyboard generation stage is modified using the large model to obtain the modified material.

[0105] A6. The video modification method as described in any one of A1-A5, wherein the step of regenerating the video based on the modified material using the large model includes: When the modified material is a storyboard material, the location information of the material to be modified in the original video is determined by the large model; Based on the location information and the modified material, the video is regenerated using the large model.

[0106] A7. The video modification method as described in any one of A1-A5, further comprising, prior to the modification trigger instruction triggered in response to the original video generated based on the large model: In response to the triggered overall video effect preview command, determine the video generation progress; If the video generation progress meets the preset preview conditions, determine the scene information for each scene. A preview video is generated based on the storyboard information.

[0107] A8. The video modification method as described in A7, wherein the storyboard information includes the storyboard stage corresponding to each storyboard, and the storyboard stage includes a structured storyboard script stage and a visual storyboard preview stage; The step of generating a preview video based on the storyboard information includes: Identify the first storyboard in the visual storyboard preview stage and the second storyboard in the structured storyboard script stage; Determine the visual storyboard of the first storyboard and the structured storyboard script of the second storyboard; A preview video is generated based on the visual storyboard and the structured storyboard script.

[0108] A9. The video modification method as described in A8, wherein generating a preview video based on the visualized storyboard and the structured storyboard script includes: Generate a second storyboard video frame based on the structured storyboard script; The first storyboard video frame is determined based on the visualized storyboard. A preview video is generated based on the second and first storyboard video frames.

[0109] A10. The video modification method as described in any one of A1-A5, wherein the video generation stage of the original video includes at least one of the following: storyboard generation stage, prompt word generation stage, style setting stage, still image generation stage, and storyboard video generation stage.

[0110] A11. The video modification method as described in any one of A1-A5, wherein the step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: Determine the modification category and modification intent corresponding to the modification prompt information; Generate material modification prompts based on the modification category and the modification intent; Based on the modified prompts, the modified material is obtained by modifying the material to be modified in the target stage using the large model.

[0111] A12. The video modification method as described in any one of A1-A5, wherein the step of generating the original video from the large model and displaying the video materials for each step includes: The process of generating the original video using the large model and the video footage for each stage are determined. The video materials are displayed hierarchically based on the aforementioned steps.

[0112] This invention discloses B13. A video modification device, the video modification device comprising: The response module is used to respond to the modification trigger command triggered by the original video generated based on the large model, and to display the various stages of the generation of the original video by the large model and the video materials of each stage; The extraction module is used to extract the material to be modified and the modification prompt information from the material modification instruction when a material modification instruction is received; The material modification module is used to modify the material to be modified in the target stage through the large model according to the modification prompt information, so as to obtain the modified material; The video regeneration module is used to regenerate the video based on the modified material using the large model.

[0113] B14. The video editing device as described in B13, wherein the material editing module is further configured to determine the storyboard video editing type based on the editing prompt information when the material editing instruction is to modify the storyboard video; When the storyboard video modification type is video frame modification, the video frame to be modified is determined according to the modification prompt information and the material to be modified; Based on the modification prompts, the video frame to be modified is modified using the large model to obtain the modified material.

[0114] B15. The video editing device as described in B14, wherein the material editing module is further configured to modify the video frame to be modified through the large model according to the modification prompt information, thereby obtaining the modified video frame; Based on the material to be modified, the adjacent video frames of the modified video frame are determined using the large model; Based on the modified video frames, the adjacent video frames are modified using the large model to obtain the modified material.

[0115] B16. The video modification apparatus as described in B15, wherein the material modification module is further configured to modify the adjacent video frames according to the modified video frames through the large model to obtain a modified video frame sequence; The visual style of the material to be modified is determined using the large model; Based on the aforementioned visual style, the video frame sequence is modified, and the material to be modified is used to generate a storyboard video through the aforementioned large model to obtain the modified material.

[0116] B17. The video editing device as described in B13, wherein the material editing module is further configured to, when the material editing instruction is to modify the storyboard, determine the modified entity and modification parameter information based on the modification prompt information through the large model; Based on the modified entity and the modified parameter information, the material to be modified in the storyboard generation stage is modified using the large model to obtain the modified material.

[0117] This invention discloses C18. A video modification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video modification method as described in any one of A1 to A12.

[0118] This invention discloses D19. A storage medium, which is a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of the video modification method as described in any one of A1 to A12.

[0119] This invention discloses E20. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the video modification method as described in any one of A1 to A12.

Claims

1. A video modification method, characterized in that, The video modification method includes the following steps: In response to a modification trigger command triggered by the original video generated based on the large model, the various stages of the generation of the original video by the large model and the video materials of each stage are displayed. Upon receiving a material modification instruction, the material to be modified and the modification prompt information are extracted from the material modification instruction; Based on the modification prompts, the material to be modified in the target stage is modified using the large model to obtain the modified material; The video is regenerated based on the modified material using the large model.

2. The video modification method as described in claim 1, characterized in that, The step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: When the material modification instruction is to modify the storyboard video, the modification type of the storyboard video is determined according to the modification prompt information; When the storyboard video modification type is video frame modification, the video frame to be modified is determined according to the modification prompt information and the material to be modified; Based on the modification prompts, the video frame to be modified is modified using the large model to obtain the modified material.

3. The video modification method as described in claim 2, characterized in that, The step of modifying the video frame to be modified using the large model based on the modification prompt information to obtain modified material includes: Based on the modification prompt information, the video frame to be modified is modified using the large model to obtain the modified video frame; Based on the material to be modified, the adjacent video frames of the modified video frame are determined using the large model; Based on the modified video frames, the adjacent video frames are modified using the large model to obtain the modified material.

4. The video modification method as described in claim 3, characterized in that, The process of modifying adjacent video frames using the large model based on the modified video frames to obtain modified material includes: Based on the modified video frames, the adjacent video frames are modified using the large model to obtain a modified video frame sequence. The visual style of the material to be modified is determined using the large model; Based on the aforementioned visual style, the video frame sequence is modified, and the material to be modified is used to generate a storyboard video through the aforementioned large model to obtain the modified material.

5. The video modification method as described in claim 1, characterized in that, The step of modifying the material to be modified in the target stage according to the modification prompt information through the large model to obtain the modified material includes: When the material modification instruction is to modify the storyboard script, the large model determines the modified entity and modification parameter information based on the modification prompt information; Based on the modified entity and the modified parameter information, the material to be modified in the storyboard generation stage is modified using the large model to obtain the modified material.

6. The video modification method according to any one of claims 1-5, characterized in that, The process of regenerating the video based on the modified material using the large model includes: When the modified material is a storyboard material, the location information of the material to be modified in the original video is determined by the large model; Based on the location information and the modified material, the video is regenerated using the large model.

7. A video editing device, characterized in that, The video editing device includes: The response module is used to respond to the modification trigger command triggered by the original video generated based on the large model, and to display the various stages of the generation of the original video by the large model and the video materials of each stage; The extraction module is used to extract the material to be modified and the modification prompt information from the material modification instruction when a material modification instruction is received; The material modification module is used to modify the material to be modified in the target stage through the large model according to the modification prompt information, so as to obtain the modified material; The video regeneration module is used to regenerate the video based on the modified material using the large model.

8. A video editing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video modification method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the video modification method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the video modification method as described in any one of claims 1 to 6.

Images