Video processing method and device, equipment, storage medium and product

Abstract

The invention discloses a video processing method and device, equipment, a storage medium and a product, relates to the technical field of artificial intelligence, and discloses that a canvas interface is displayed in response to an editing instruction for a target video clip in a target video, and the canvas interface comprises a process material associated with the target video clip; generating a new process material based on the existing process material in the canvas interface through the intelligent agent, and adding the new process material into the canvas interface; and in response to an application instruction for at least one process material in the canvas interface, replacing the target video clip in the target video with the process material indicated by the application instruction. According to the method, the video generated by the agent can better match user requirements, and the practicability of the video generated by the agent is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video processing method, apparatus, device, storage medium and product. Background Technology

[0002] With the development of artificial intelligence technology, intelligent agents have become an important tool for video creation, widely used in short videos, advertising, education, and other scenarios. Relying on deep learning and massive amounts of data, they can quickly convert text and images into dynamic videos, significantly reducing the barrier to entry and time costs for video production.

[0003] However, due to the agent's lack of understanding of users' personalized needs and limited control over scene details and content logic, the generated videos often do not meet users' expectations, which seriously reduces the practicality of the generated videos and makes it difficult to meet high-quality and customized needs.

[0004] Therefore, how to improve the practicality of agent-generated videos and make them more in line with user needs has become an urgent technical problem to be solved.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a video processing method, apparatus, device, storage medium, and product that enables intelligent agents to generate videos that better match user needs and effectively improves the practicality of intelligent agent-generated videos.

[0007] To achieve the above objectives, this application proposes a video processing method, the method comprising: In response to an editing instruction on a target video segment in a target video, a canvas interface is displayed, the canvas interface containing process materials associated with the target video segment; Using an intelligent agent, new process materials are generated based on existing process materials in the canvas interface, and the new process materials are added to the canvas interface. In response to an application instruction on at least one process material in the canvas interface, the target video segment in the target video is replaced with the process material indicated by the application instruction.

[0008] Optionally, the existing process materials in the canvas interface include reference video clips and / or reference images on which the reference video clips are generated, and the reference video clips include the target video clip and / or candidate video clips of the target video clip; The new process material includes new video clips generated based on the reference video clips and / or the reference images.

[0009] Optionally, the new video clip differs from the reference video clip in at least one of the following ways: Sound effects, dialogue corresponding to the character's lip movements, compositional elements of the video frame, number of elements in the video frame, layout of elements in the video frame, dynamic effects of elements in the video frame, camera movement effects of the video frame, and style of the video frame.

[0010] Optionally, the step of generating new process materials based on existing process materials in the canvas interface through an intelligent agent includes at least one of the following: In response to the video regeneration instruction, the intelligent agent re-converts the video based on the reference image to obtain the new video segment; In response to the lip-sync video generation command, the intelligent agent converts the lip movements of the characters in the reference video clip from those adapted to the original dialogue to those adapted to the new dialogue, thereby obtaining the new video clip; In response to the sound effect generation command, the agent converts the original sound effect in the reference video clip into a new sound effect to obtain the new video clip; In response to the multi-image video generation instruction, the intelligent agent performs video conversion based on the reference image and the new image to obtain the new video segment; In response to a video frame to video conversion instruction, the agent performs video conversion based on at least one frame of the reference video segment to obtain the new video segment. In response to a video element update instruction, the agent adjusts the target element in the reference video segment to obtain the new video segment. The adjustment of the target element includes: adjusting at least one of the number, layout, and dynamic effects of the target element, or replacing the target element with a new element. In response to a video style conversion instruction, the agent adjusts the video style of the reference video segment to obtain the new video segment.

[0011] Optionally, the canvas interface includes multiple types of controls, and the method further includes at least one of the following: Based on the trigger operation of the video regeneration control, the video regeneration instruction is generated; Based on the trigger operation of the lip-sync video generation control, new lines are obtained, and the lip-sync video generation instruction is generated based on the new lines; Based on the trigger operation of the sound effect generation control, the sound effect generation instruction is generated; Based on the trigger operation of the multi-image video generation control, the uploaded new image is obtained, and the multi-image video generation instruction is generated based on the reference image and the new image. Based on the trigger operation of the video frame to video control, at least one frame image selected from the reference video segment is obtained, and based on the at least one frame image, the video frame to video instruction is generated. Based on the trigger operation of the video element update control, the element modification conditions are obtained. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, the video element update instruction is generated. Based on the selection operation of the target element in the reference video segment, the video element update instruction is generated; The video style conversion instruction is generated based on the trigger operation of the video style conversion control.

[0012] Optionally, the new process material further includes a new reference image generated by the agent, and a new video clip generated based on the new reference image; wherein, the method for obtaining the new reference image includes at least one of the following: In response to the image regeneration instruction, the agent generates the new reference image based on the reference text, where the reference text is the basis for generating the existing reference image in the canvas interface. In response to the text-to-image command, the intelligent agent generates the new reference image based on the new reference text. In response to the image-to-image command, the intelligent agent generates the new reference image based on the uploaded image; In response to the image modification command, the intelligent agent modifies the existing reference image in the canvas interface based on the input image modification prompt to obtain the new reference image; In response to the region redraw command, the intelligent agent redraws the target region of the existing reference image in the canvas interface to obtain the new reference image. In response to an image element update command, the agent adjusts the target element in an existing reference image on the canvas interface to obtain the new reference image. The adjustment of the target element includes: adjusting at least one of the quantity and layout of the target element, or replacing the target element with a new element.

[0013] Optionally, the canvas interface includes multiple types of controls, and the method further includes at least one of the following: Based on the trigger operation of the image regeneration control, the image regeneration instruction is generated; Based on the trigger operation of the text-to-image control, obtain the new reference text input, and generate the text-to-image instruction based on the new reference text; Based on the trigger operation of the image-generating control, the uploaded image is obtained, and the image-generating instruction is generated based on the uploaded image; Based on the trigger operation of the image modification control, the input image modification prompt is obtained, and based on the image modification prompt, the image modification instruction is generated; Based on the trigger operation of the region redraw control, the target region selected in the existing reference image is determined, and the region redraw instruction is generated based on the target region. Based on the trigger operation of the image element update control, the element modification conditions are obtained. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, the image element update instruction is generated. Based on the selection operation of target elements in the existing reference image, the image element update instruction is generated.

[0014] Optionally, the step of generating new process materials based on existing process materials in the canvas interface through an intelligent agent includes: Obtain modification instruction information input in the canvas interface for the existing process material, wherein the modification instruction information indicates the content to be modified; The intelligent agent adjusts the content to be modified in the existing process material based on the modification instruction information to obtain the new process material.

[0015] Optionally, the modification instruction information further includes the modification target of the content to be modified. The step of adjusting the content to be modified in the existing process material based on the modification instruction information, through the intelligent agent, to obtain the new process material, includes: The intelligent agent adjusts the content to be modified in the existing process material according to the modification instruction information to meet the modification target, thereby obtaining new process material.

[0016] Optionally, the step of generating new process materials based on existing process materials in the canvas interface through an intelligent agent, and adding the new process materials to the canvas interface, includes: The process involves iteratively executing the steps of generating new process materials based on existing process materials in the canvas interface through the intelligent agent, and adding the new process materials to the canvas interface.

[0017] Optionally, the step of generating new process materials based on existing process materials in the canvas interface through an intelligent agent includes: Based on the material quantity specification control in the canvas interface, determine the target quantity of the new process materials to be generated, and through the intelligent agent, generate the target quantity of new process materials based on the existing process materials in the canvas interface; and / or, Based on the agent-specified control in the canvas interface, a target agent is determined to generate the new process material. The new process material is then generated by the target agent based on the existing process material in the canvas interface.

[0018] Optionally, the canvas interface includes multiple material display areas, and different material display areas are used to display different types of process materials, including video and image types; The material display area corresponding to the process material of the video type includes video clips and the lines corresponding to the lip movements of the characters in the video clips, or icons of the sound effects of the video clips, or intelligent agent prompts used to generate the video clips; The material display area corresponding to the process material of the image type includes the image and the intelligent agent prompt words used to generate the image.

[0019] Furthermore, to achieve the above objectives, this application also proposes a video processing apparatus, the apparatus comprising: The canvas display module is used to display a canvas interface in response to editing instructions for a target video segment in the target video. The canvas interface contains process materials associated with the target video segment. The material generation module is used to generate new process materials based on the existing process materials in the canvas interface through an intelligent agent, and add the new process materials to the canvas interface; The media replacement module is used to replace the target video segment in the target video with the media indicated by the application instruction in response to an application instruction for at least one process media in the canvas interface.

[0020] Optionally, the existing process materials in the canvas interface include reference video clips and / or reference images on which the reference video clips are generated, and the reference video clips include the target video clip and / or candidate video clips of the target video clip; The new process material includes new video clips generated based on the reference video clips and / or the reference images.

[0021] Optionally, the new video clip differs from the reference video clip in at least one of the following ways: Sound effects, dialogue corresponding to the character's lip movements, compositional elements of the video frame, number of elements in the video frame, layout of elements in the video frame, dynamic effects of elements in the video frame, camera movement effects of the video frame, and style of the video frame.

[0022] Optionally, the material generation module is configured to perform at least one of the following: In response to the video regeneration instruction, the intelligent agent re-converts the video based on the reference image to obtain the new video segment; In response to the lip-sync video generation command, the intelligent agent converts the lip movements of the characters in the reference video clip from those adapted to the original dialogue to those adapted to the new dialogue, thereby obtaining the new video clip; In response to the sound effect generation command, the agent converts the original sound effect in the reference video clip into a new sound effect to obtain the new video clip; In response to the multi-image video generation instruction, the intelligent agent performs video conversion based on the reference image and the new image to obtain the new video segment; In response to a video frame to video conversion instruction, the agent performs video conversion based on at least one frame of the reference video segment to obtain the new video segment. In response to a video element update instruction, the agent adjusts the target element in the reference video segment to obtain the new video segment. The adjustment of the target element includes: adjusting at least one of the number, layout, and dynamic effects of the target element, or replacing the target element with a new element. In response to a video style conversion instruction, the agent adjusts the video style of the reference video segment to obtain the new video segment.

[0023] Optionally, the canvas interface includes multiple types of controls, and the material generation module is also used to perform at least one of the following: Based on the trigger operation of the video regeneration control, the video regeneration instruction is generated; Based on the trigger operation of the lip-sync video generation control, new lines are obtained, and the lip-sync video generation instruction is generated based on the new lines; Based on the trigger operation of the sound effect generation control, the sound effect generation instruction is generated; Based on the trigger operation of the multi-image video generation control, the uploaded new image is obtained, and the multi-image video generation instruction is generated based on the reference image and the new image. Based on the trigger operation of the video frame to video control, at least one frame image selected from the reference video segment is obtained, and based on the at least one frame image, the video frame to video instruction is generated. Based on the trigger operation of the video element update control, the element modification conditions are obtained. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, the video element update instruction is generated. Based on the selection operation of the target element in the reference video segment, the video element update instruction is generated; The video style conversion instruction is generated based on the trigger operation of the video style conversion control.

[0024] Optionally, the new process material further includes a new reference image generated by the intelligent agent, and a new video clip generated based on the new reference image; wherein, the material generation module is further configured to perform at least one of the following: In response to the image regeneration instruction, the agent generates the new reference image based on the reference text, where the reference text is the basis for generating the existing reference image in the canvas interface. In response to the text-to-image command, the intelligent agent generates the new reference image based on the new reference text. In response to the image-to-image command, the intelligent agent generates the new reference image based on the uploaded image; In response to the image modification command, the intelligent agent modifies the existing reference image in the canvas interface based on the input image modification prompt to obtain the new reference image; In response to the region redraw command, the intelligent agent redraws the target region of the existing reference image in the canvas interface to obtain the new reference image. In response to an image element update command, the agent adjusts the target element in an existing reference image on the canvas interface to obtain the new reference image. The adjustment of the target element includes: adjusting at least one of the quantity and layout of the target element, or replacing the target element with a new element.

[0025] Optionally, the canvas interface includes multiple types of controls, and the material generation module is also used to perform at least one of the following: Based on the trigger operation of the image regeneration control, the image regeneration instruction is generated; Based on the trigger operation of the text-to-image control, obtain the new reference text input, and generate the text-to-image instruction based on the new reference text; Based on the trigger operation of the image-generating control, the uploaded image is obtained, and the image-generating instruction is generated based on the uploaded image; Based on the trigger operation of the image modification control, the input image modification prompt is obtained, and based on the image modification prompt, the image modification instruction is generated; Based on the trigger operation of the region redraw control, the target region selected in the existing reference image is determined, and the region redraw instruction is generated based on the target region. Based on the trigger operation of the image element update control, the element modification conditions are obtained. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, the image element update instruction is generated. Based on the selection operation of target elements in the existing reference image, the image element update instruction is generated.

[0026] Optionally, the material generation module includes: The instruction acquisition unit is used to acquire modification instruction information input in the canvas interface for the existing process material, wherein the modification instruction information indicates the content to be modified. The material adjustment unit is used by the intelligent agent to adjust the content to be modified in the existing process material based on the modification instruction information, so as to obtain the new process material.

[0027] Optionally, the modification instruction information may also include the modification target of the content to be modified. The material adjustment unit is used by the intelligent agent to adjust the content to be modified in the existing process material to meet the modification target based on the modification instruction information, so as to obtain new process material.

[0028] Optionally, the material generation module is used to iteratively execute the steps of generating new process materials based on existing process materials in the canvas interface through the intelligent agent, and adding the new process materials to the canvas interface.

[0029] Optionally, the material generation module is configured to determine the target quantity of the new process materials to be generated based on the material quantity specification control in the canvas interface, and generate the target quantity of new process materials based on the existing process materials in the canvas interface through the intelligent agent; and / or, determine the target intelligent agent for generating the new process materials based on the intelligent agent specification control in the canvas interface, and generate the new process materials based on the existing process materials in the canvas interface through the target intelligent agent.

[0030] Optionally, the canvas interface includes multiple material display areas, and different material display areas are used to display different types of process materials, including video and image types; The material display area corresponding to the process material of the video type includes video clips and the lines corresponding to the lip movements of the characters in the video clips, or icons of the sound effects of the video clips, or intelligent agent prompts used to generate the video clips; The material display area corresponding to the process material of the image type includes the image and the intelligent agent prompt words used to generate the image.

[0031] In addition, to achieve the above objectives, this application also proposes a video processing apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video processing method described above.

[0032] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the video processing method described above.

[0033] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the video processing method described above.

[0034] One or more technical solutions proposed in this application have at least the following technical effects: The video processing solution provided in this application, in response to editing instructions for a target video segment within a target video, displays a canvas interface containing process materials associated with the target segment, providing users with an editing foundation and enabling them to deeply intervene in the intermediate stages of video segment generation. By having an intelligent agent generate new process materials based on existing process materials in the canvas interface and add them to the canvas, the diversity of process materials is enhanced, providing users with rich options. Users can select the optimal process materials from the canvas interface to replace the target video segment in the target video with the selected process materials, achieving refined and personalized adjustments to the video content. This solution, through the visualization and editability of the process materials for video segments, enhances the user's control over the video generation process, compensates for the shortcomings of intelligent agents in semantic understanding and scene logic, effectively improves the quality and customization level of the final video, thereby enhancing the practicality of the target video. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A schematic diagram of an implementation environment for a video processing method; Figure 2 A flowchart illustrating the first embodiment of the video processing method; Figure 3 This is a detailed step diagram of step S20 in the second embodiment of the video processing method; Figure 4 This is a schematic diagram of a canvas interface; Figure 5 This is a schematic diagram illustrating the display effect of a task waiting area in a canvas interface. Figure 6 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 7 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 8 This is a schematic diagram illustrating the display effect of a task waiting area in a canvas interface. Figure 9 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 10 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 11 This is a schematic diagram illustrating the display effect of an image upload area in a canvas interface. Figure 12 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 13 This is a schematic diagram illustrating the display effect of a video frame selection area in a canvas interface. Figure 14 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 15 This is a detailed step diagram of step S20 in the third embodiment of the video processing method; Figure 16 This is a schematic diagram of a canvas interface; Figure 17 This is a schematic diagram of a text input area in a canvas interface. Figure 18 This is a schematic diagram illustrating the display effect of a video clip in a canvas interface. Figure 19 This is a schematic diagram illustrating the display effect of an image upload area in a canvas interface. Figure 20 This is a schematic diagram illustrating the display effect of an image upload area in a canvas interface. Figure 21 This is a schematic diagram illustrating the display effect of an image upload area in a canvas interface. Figure 22 This is a schematic diagram illustrating the display effect of an image editing area in a canvas interface. Figure 23 This is a schematic diagram illustrating the display effect of a reference image in the canvas interface; Figure 24 This is a schematic diagram illustrating the display effect of an image editing area in a canvas interface. Figure 25 This is a schematic diagram illustrating the display effect of a reference image in the canvas interface; Figure 26 This is a schematic diagram illustrating the display effect of an image editing area in a canvas interface. Figure 27 This is a detailed step diagram of step S20 in the fourth embodiment of the video processing method; Figure 28 This is a schematic diagram of the module structure of the video processing device according to an embodiment of this application; Figure 29 This is a schematic diagram of the device structure of the hardware operating environment involved in the video processing method in the embodiments of this application.

[0038] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0040] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0041] Figure 1 This is a schematic diagram illustrating an implementation environment provided by an embodiment of this disclosure. See also... Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network. For example, the terminal 101 is installed with a target application provided by the server 102, and the terminal 101 can perform functions such as data transmission and message interaction through the target application.

[0042] For example, terminal 101 is a computer, mobile phone, tablet computer, or other terminal. For example, the target application is a target application in the operating system of terminal 101, or a target application provided by a third party. For example, the target application is a search application, a short video application, a camera application, etc. For example, server 102 is the backend server corresponding to the target application. Accordingly, server 102 is a search application server, a short video application server, a camera application server, etc.

[0043] In this application, terminal 101 displays a canvas interface in response to an editing instruction for a target video segment in a target video. The canvas interface contains process footage associated with the target video segment. When a user issues a material regeneration instruction based on the canvas interface, terminal 101 sends a material regeneration request to server 102. The material regeneration request may carry existing process footage in the canvas interface. Server 102 receives the material regeneration request, generates new process footage based on the existing process footage in the canvas interface through an intelligent agent, and returns the new process footage to terminal 101. Terminal 101 adds the new process footage to the canvas interface; in response to an application instruction for at least one process footage in the canvas interface, it replaces the target video segment in the target video with the process footage indicated by the application instruction.

[0044] Alternatively, the above video processing can be performed solely by terminal 101. Alternatively, terminal 101 can perform the processing through the installed target application. This application embodiment does not impose any limitations on this.

[0045] Figure 2 This is a schematic flowchart of the first embodiment of the video processing method of this application. (Refer to...) Figure 2 Taking the terminal as the executing entity as an example, this video processing method includes the following steps S10~S40: Step S10: In response to the editing instruction for the target video segment in the target video, a canvas interface is displayed, which contains the process materials associated with the target video segment.

[0046] The target video refers to the complete video file that the user is currently editing. This video contains multiple video clips. For example, a video clip can be a storyboard video clip or a music-synced video clip. A storyboard video clip corresponds to a single scene, referring to an independent video unit presenting single content from a specific angle and shot. A music-synced video clip corresponds to a single musical beat, referring to an independent video unit where the visual content, such as action, transitions, and special effects, is precisely synchronized with a specific beat, drumbeat, or accent in the music. The target video clip is a specific video clip selected from the target video that the user wishes to edit.

[0047] For example, the target video is a video generated by an intelligent agent. For instance, by inputting a text description and / or a reference image into the video generation interface, the terminal uses the intelligent agent to automatically generate a complete video based on the style and elements of the input text description and / or reference image; this generated video is the target video. The text description can include scene, characters, plot information, etc. The intelligent agent is a model with video generation capabilities. Examples include Runway's first or second generation machine learning models, Pika Labs' Pika 1.0, and Sora.

[0048] The canvas interface is an interactive interface used to display and process materials related to the video editing process. Process materials are various materials generated during video creation or editing, including video clips and images, used to assist in completing the editing work.

[0049] Step S20: Using the intelligent agent, generate new process materials based on the existing process materials in the canvas interface, and add the new process materials to the canvas interface.

[0050] The existing process footage in the canvas interface consists of footage generated during video generation or historical editing. Optionally, the existing process footage in the canvas interface includes reference video clips and / or reference images on which the reference video clips are based. Reference video clips include target video clips and / or candidate video clips of the target video clip. New process footage includes new video clips generated based on reference video clips and / or reference images.

[0051] The reference video clips are video clips used as a basis for generating new video clips, including the target video clip and its candidate video clips. Candidate video clips are alternative video clips that can be used to replace the target video clip. Reference images are image materials used to generate reference video clips and are one of the bases for video clip generation. New process materials are new materials generated by the agent based on existing process materials in the canvas interface, including new video clips, new reference images, etc.

[0052] Optionally, the new video clip may differ from the reference video clip in at least one of the following: sound effects, dialogue corresponding to the character's lip movements, compositional elements of the video frame, number of elements in the video frame, layout of elements in the video frame, dynamic effects of elements in the video frame, camera movement effects of the video frame, and style of the video frame.

[0053] The constituent elements of a video frame refer to the basic visual elements that make up a single frame or a series of images. These are the "parts" that compose the video image, determining its core content and visual presentation. Examples include main subjects such as people, animals, and plants; scene elements such as buildings, furniture, and props; background environments such as the sky, mountains, and rivers; and supplementary elements such as text, icons, and special effects particles. The layout of elements in a video frame refers to the arrangement, size ratio, and spatial relationship of various visual elements within the frame of a single frame or a series of images. The dynamic effects of elements in a video frame refer to the movement and changes of visual elements themselves, independent of camera movement, but rather the independent actions or changes in state of the elements. Camera movement effects refer to the dynamic effects achieved by simulating camera movement, such as panning, tilting, and rotating, to change the viewpoint, framing, or follow the movement of the subject. The style of a video frame refers to the unified visual tone and artistic atmosphere created through visual language such as color, light and shadow, composition, texture, and lines, as well as the way the content is presented, such as character design and scene design.

[0054] This solution supports multi-dimensional adjustments to videos, from sound effects and dialogue to visual elements, camera movement, and style. It can quickly generate new video clips that meet the needs of different scenarios while retaining the basic framework of reference video clips. This not only improves the efficiency of content reuse but also flexibly optimizes video expressiveness, making creations more suitable for specific usage goals and achieving low-cost, high-efficiency video innovation.

[0055] Optionally, the agent generates new process materials based on existing process materials in the canvas interface and adds the new process materials to the canvas interface, including: iteratively executing the steps of generating new process materials based on existing process materials in the canvas interface and adding the new process materials to the canvas interface.

[0056] This scheme involves an agent generating new process materials in an iterative manner. Specifically, it first generates the first batch of new materials based on the initial process materials in the canvas interface and adds them to the canvas. Then, the agent can continue to generate more new materials based on the initial materials and the first batch of new materials in the canvas interface. The step of generating new process materials based on existing process materials and adding them to the canvas interface can be repeated, continuously enriching the process materials in the canvas interface.

[0057] By iteratively generating materials through intelligent agents, the quantity and diversity of process materials in the canvas interface can be quickly expanded, providing a richer selection space for subsequent video editing and making the final video creation more flexible and more in line with personalized needs.

[0058] Optionally, the canvas interface includes a material quantity specification control. This control is an interactive element in the canvas interface used to set the desired quantity of new process materials to be generated, such as a numeric input box or quantity increase / decrease buttons. Correspondingly, through an intelligent agent, new process materials are generated based on existing process materials in the canvas interface. This includes: determining the target quantity of new process materials to be generated based on the material quantity specification control in the canvas interface; and generating the target quantity of new process materials based on existing process materials in the canvas interface through the intelligent agent.

[0059] Optionally, the canvas interface includes an agent-designated control. This control is an interactive element within the canvas interface used to select the specific agent responsible for generating new materials, such as a drop-down menu or a selection button. Correspondingly, generating new process materials based on existing process materials in the canvas interface, using the agent, includes: determining the target agent for generating the new process materials based on the agent-designated control in the canvas interface; and generating the new process materials using the target agent based on existing process materials in the canvas interface.

[0060] This application embodiment provides controls for specifying the number of materials and the intelligent agent, allowing users to independently specify the number of materials and the intelligent agent to be used. This enables precise control over the scale of material output to match editing needs, and flexible selection of suitable intelligent agents to ensure material quality. Ultimately, it achieves on-demand generation and efficient and controllable video material creation, reducing operational complexity while improving the targeting of material generation.

[0061] Optionally, the canvas interface includes multiple material display areas. Different material display areas are used to display different types of process materials, including video and image types. The material display area for video-type process materials contains video clips and the dialogue corresponding to the characters' lip movements in the video clips, or icons for the sound effects of the video clips, or the intelligent agent prompts used to generate the video clips; the material display area for image-type process materials contains images and the intelligent agent prompts used to generate the images.

[0062] Among them, the intelligent agent prompt is the instruction text that the user inputs to the intelligent agent to clarify the requirements for material generation, or the intelligent agent prompt is the instruction text obtained by processing the instruction text input by the user through a prompt processing program. The intelligent agent generates corresponding video or image materials based on the description of the intelligent agent prompt.

[0063] Sound effect icons are graphic symbols used to visually identify the sound effects accompanying a video clip. Clicking or viewing the icon will provide information about the video's sound effects. For example, a sound effect icon can be shaped like a record, with different colored records representing different sound effects.

[0064] This solution categorizes and displays video and image-based process materials by dividing the canvas interface into different areas, and simultaneously presents key related information such as dialogue, sound effect icons, and intelligent agent prompts. This not only makes the material types clearly identifiable and management more efficient, but also allows users to quickly obtain the generation logic and additional information of the materials, reducing information search costs and providing clear information support for subsequent material editing and combination creation, thereby improving overall creation efficiency.

[0065] Step S30: In response to an application instruction on at least one process material in the canvas interface, replace the target video segment in the target video with the process material indicated by the application instruction.

[0066] For example, each process material in the canvas interface has a corresponding application control. When a user performs a trigger operation on the application control of at least one process material, the target video segment in the target video can be replaced with the process material corresponding to the triggered application control.

[0067] For example, the terminal displays a target video clip in the video generation interface. In response to an editing operation on the target video clip, it jumps to the canvas interface corresponding to the target video clip. After generating and displaying new process material based on the canvas interface, in response to an application instruction on at least one process material in the canvas interface, it returns to the video generation interface and replaces the target video clip in the target video displayed in the video generation interface with the process material indicated by the application instruction.

[0068] Understandably, this solution supports both "single-material replacement" and "multi-material replacement" scenarios. When a user issues an application command to any one or more process materials in the canvas interface, the terminal will replace the target video segment in the target video with one or more process materials specified by the user command. This significantly improves the flexibility of video segment replacement, enhancing overall editing efficiency and user experience.

[0069] The video processing solution provided in this application, in response to editing instructions for a target video segment within a target video, displays a canvas interface containing process materials associated with the target segment, providing users with an editing foundation and enabling them to deeply intervene in the intermediate stages of video segment generation. By having an intelligent agent generate new process materials based on existing process materials in the canvas interface and add them to the canvas, the diversity of process materials is enhanced, providing users with rich options. Users can select the optimal process materials from the canvas interface to replace the target video segment in the target video with the selected process materials, achieving refined and personalized adjustments to the video content. This solution, through the visualization and editability of the process materials for video segments, enhances the user's control over the video generation process, compensates for the shortcomings of intelligent agents in semantic understanding and scene logic, effectively improves the quality and customization level of the final video, thereby enhancing the practicality of the target video.

[0070] Based on the first embodiment described above, a second embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 3 In the second embodiment, step S20 includes at least one of steps S201 to S207: In step S201, in response to the video regeneration instruction, the agent re-converts the video based on the reference image to obtain a new video segment.

[0071] Optionally, the canvas interface includes a video regeneration control, and the terminal generates a video regeneration instruction based on the trigger operation of the video regeneration control. (See reference) Figure 4 The canvas interface displays a reference image at the top, below which is a reference video clip generated based on that image. To the right of this video clip is its corresponding toolbar, where a "Regenerate" control is displayed; this is the video regeneration control. Clicking this control will display... Figure 5 The task waiting area shown can be displayed to the right of a reference video clip. After successful video generation, the task waiting area will switch to display the newly generated video clip, such as... Figure 6 The effect shown.

[0072] In step S202, in response to the lip-sync video generation instruction, the agent converts the lip movements of the characters in the reference video clip from those adapted to the original lines to those adapted to the new lines, thereby obtaining a new video clip.

[0073] Optionally, the canvas interface includes a lip-sync video generation control. The terminal obtains new lines of dialogue based on the trigger operation of the lip-sync video generation control, and generates a lip-sync video generation instruction based on the new lines of dialogue.

[0074] Continue to refer to Figure 4 Clicking the lip-sync control in the toolbar will display the original lip-sync result of the reference video clip on the canvas interface, such as... Figure 7 As shown, the original line in the reference video clip, matched with the character's lip movements, is "Mommy, I'm going to play now." Clicking the lip-sync control in the lower right corner will display the canvas interface. Figure 8 The task waiting area shown displays the new dialogue entered by the user, such as "Mom, I'm going to find a map today!". After successful video generation, the task waiting area will switch to displaying the newly generated video clip, i.e., the new lip-sync result, as shown below. Figure 9 As shown.

[0075] In step S203, in response to the sound effect generation instruction, the original sound effect in the reference video clip is converted into a new sound effect through the intelligent agent to obtain a new video clip.

[0076] Optionally, the canvas interface includes a sound effect generation control, and the terminal generates sound effect generation instructions based on the trigger operation of the sound effect generation control. (See also...) Figure 4 The control displayed in the toolbar with "Generate Sound Effects" is the sound effect generation control. Clicking this control will display a task waiting area on the canvas interface, which can be shown to the right of the reference video clip. After successfully generating the video, the task waiting area will switch to display the newly generated video clip, such as... Figure 10 The effect shown is as follows. This new video clip contains four different sound effects; clicking on the different sound effect icons will allow you to learn about the different sound effects.

[0077] In step S204, in response to the multi-image video generation instruction, the agent performs video conversion based on the reference image and the new image to obtain a new video clip.

[0078] Optionally, the canvas interface includes a multi-image video generation control. Based on the triggering operation of this control, the terminal retrieves the uploaded new image and generates a multi-image video generation instruction based on the reference image and the new image. (Continue to refer to...) Figure 4 The control displayed in the toolbar that says "Generate Video from Multiple Images" is the multi-image video generation control. Clicking this control will display the canvas interface. Figure 11 The image upload area shown contains the original reference image. After uploading a new image in this area and clicking the "Generate Video" control in the lower right corner, the image upload area will switch to a task waiting area. Once the video is successfully generated, the task waiting area will switch to display the newly generated video clip, such as... Figure 12 The effect shown.

[0079] In step S205, in response to the video frame to video conversion instruction, the agent performs video conversion based on at least one frame of the reference video segment to obtain a new video segment.

[0080] Optionally, the canvas interface includes a video frame to video control. Based on the trigger operation of the video frame to video control, the terminal obtains at least one frame image selected from the reference video segment, and generates a video frame to video instruction based on the at least one frame image.

[0081] For example, the at least one frame may be the first frame or the last frame of a reference video clip. Alternatively, the at least one frame may include the first frame of a reference video clip and any frame selected from the reference video clip; this application does not impose any limitations on this embodiment.

[0082] Continue to refer to Figure 4 The toolbar to the right of the reference video clip may also contain a video frame to video control. Clicking this control will display the canvas interface. Figure 13The video frame selection area is shown. This area includes the first frame of the reference video clip. Users can select any frame from the reference video clip as the last frame using the last frame selection control. After selection, clicking the "Generate Video" control in the lower right corner will switch the display of the video frame selection area to the task waiting area. After successful video generation, the task waiting area will switch to display the newly generated video clip, such as... Figure 14 The effect shown.

[0083] Step S206: In response to the video element update instruction, the target element in the reference video segment is adjusted by the intelligent agent to obtain a new video segment. The adjustment of the target element includes: adjusting at least one of the number, layout and dynamic effects of the target element, or replacing the target element with a new element.

[0084] Optionally, the canvas interface includes a video element update control. Based on the trigger operation of the video element update control, the terminal obtains the element modification conditions. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, a video element update instruction is generated.

[0085] Element modification conditions are the information entered by the user to adjust elements in a reference video clip, clearly specifying what needs to be changed and how it should be changed. They are the core basis for generating video element update instructions. The target element to be modified is the specific visual element specified in the element modification conditions that needs to be adjusted in the reference video clip, such as characters, background props, or text titles. The modification objective of the target element is the final effect that the element to be modified must achieve, as clearly stated in the element modification conditions, such as increasing its quantity, shifting its position to the right, or replacing it with a new appearance.

[0086] Example 1: Adjusting the number of elements: After the user triggers the video element update control, they enter "increase the number of flowers in the video from 3 to 5" in the modification prompt input box. The terminal uses an agent to perform semantic recognition on this information, determining that the target element to be modified is flowers, and the modification target is to increase from 3 to 5. Based on this element modification condition, a video element update instruction is generated. Then, the agent adjusts the number of flowers in the reference video clip based on this element modification condition to obtain a new video clip.

[0087] Example 2: Adjusting Element Layout: After the user triggers the video element update control, they enter "move the product logo in the lower left corner of the video frame to the upper right corner" in the modification prompt input box. The terminal uses an intelligent agent to perform semantic recognition on this information, determining that the target element to be modified is the product logo, and the modification target is to move it from the lower left corner to the upper right corner. Based on this element modification condition, a video element update instruction is generated. Then, through the intelligent agent, the position of the product logo in the reference video clip is adjusted according to the element modification condition, resulting in a new video clip.

[0088] Example 3: Adjusting Element Dynamics: After the user triggers the video element update control, they enter "Slow down the falling speed of the snowflakes in the video by 50%" in the modification prompt input box. The terminal uses an agent to perform semantic recognition on this information, determining that the target element to be modified is the falling snowflakes, and the modification target is to slow down the falling speed by 50%. Based on this element modification condition, a video element update instruction is generated. Then, through the agent, the falling speed of the snowflakes in the reference video clip is adjusted based on this element modification condition to obtain a new video clip.

[0089] Example 4: Replacing the Target Element: After the user triggers the video element update control, they enter "Replace the red car in the video with a blue car" in the modification prompt input box. The terminal uses an agent to perform semantic recognition on this information, determining that the target element to be modified is the red car, and the modification target is to replace it with a blue car, thus obtaining the element modification condition. Based on this element modification condition, a video element update instruction is generated. Then, through the agent, the red car in the reference video clip is adjusted according to the element modification condition, resulting in a new video clip where the red car has been replaced by a blue car.

[0090] Optionally, the terminal generates a video element update instruction based on the selection operation of target elements in the reference video segment. The selection operation of target elements includes various methods such as box selection, point selection, and circle selection.

[0091] For example, a user selects a target element in a reference video clip, such as a person or prop in the scene. After selection, a quick operation menu automatically pops up. This menu can include options such as adjusting the quantity, modifying the layout, replacing elements, and changing dynamic effects. The user selects the corresponding option and sets specific modification parameters, such as changing the quantity from 2 to 4. The terminal generates a video element update instruction based on the selected target element, the selected modification item, and the specific modification parameters. Then, the intelligent agent adjusts the target element in the reference video clip according to the selected modification item and specific modification parameters to obtain a new video clip.

[0092] For example, after a user selects a target element, they can click the voice input button on the interface and speak their desired modification, such as "replace the selected red balloon with a yellow star." The terminal uses an intelligent agent to perform semantic analysis on the selected target element and the spoken words to determine the target element to be modified and its modification target, thereby generating a corresponding video element update instruction. Alternatively, after selecting a target element, the user can also directly enter the text description "replace the selected red balloon with a yellow star" through the modification prompt input box on the canvas interface. In this case, the terminal uses an intelligent agent to perform semantic analysis on the selected target element and the text description to determine the target element to be modified and its modification target, thereby generating a corresponding video element update instruction.

[0093] In step S207, in response to the video style conversion instruction, the video style of the reference video clip is adjusted by the intelligent agent to obtain a new video clip.

[0094] Optionally, the canvas interface includes a video style conversion control, and the terminal generates a video style conversion instruction based on the trigger operation of the video style conversion control.

[0095] For example, the video style conversion control is presented as a "style list dropdown" or a "style card group," which may include preset options such as "cartoon style," "retro film style," "cyberpunk style," and "ink wash style." After the user selects a style, the terminal directly generates a video style conversion instruction based on that selection, instructing the video style to be converted to the user-selected style. Then, the terminal uses an intelligent agent to convert the video style of the reference video clip to the user-selected style, resulting in a new video clip.

[0096] For example, a video style conversion control is associated with a style description input box. Based on a trigger operation on the video style conversion control, the terminal displays the style description input box, where the user enters a custom style description, such as "a warm and healing Japanese anime style, with an orange-pink hue and soft lines." The terminal uses an intelligent agent to recognize this custom style description, determine the target style, and then generate a video style conversion instruction matching the target style. Next, the terminal uses the intelligent agent to convert the video style of the reference video clip to the target style, resulting in a new video clip.

[0097] For example, the video style conversion control includes a style reference image upload button. The user uploads an image with the target style, such as a painting in the style of Van Gogh or a screenshot from a retro movie, using this button. The terminal analyzes the target style of the image using an intelligent agent and generates a video style conversion instruction that matches that style. Then, the terminal uses the intelligent agent to convert the video style of the reference video clip to the target style, resulting in a new video clip.

[0098] This application provides multi-dimensional intelligent command functions, including video regeneration, lip-syncing to new dialogue, sound effect replacement, multi-image video generation, video frame-to-video conversion, video element adjustment, and style conversion, covering core needs in video creation from content reconstruction and detail optimization to style adjustment. It can flexibly meet diverse creative goals and significantly improve video processing efficiency and convenience.

[0099] This application embodiment sets up various types of controls on the canvas interface, such as video regeneration, lip-syncing, and sound effect generation, and clarifies the command generation logic after each control is triggered, such as inputting new lines, uploading new images, selecting video frames, and selecting target elements. This allows users to generate the required commands without complicated operations, simply by triggering the corresponding controls and supplementing simple information. This significantly reduces the operational threshold of video editing, making video modification and innovation more intuitive and convenient, and comprehensively improving the efficiency and experience of video creation.

[0100] Based on the first embodiment of this application described above, a third embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. See also... Figure 15 In the third embodiment, step S20 includes at least one of steps S208 to S213, and step S214.

[0101] In step S208, in response to the image regeneration instruction, a new reference image is generated by the intelligent agent based on the reference text, where the reference text is the basis for generating the existing reference image in the canvas interface.

[0102] Optionally, the canvas interface includes an image regeneration control, and the terminal generates an image regeneration instruction based on the trigger operation of the image regeneration control. (See reference) Figure 16 The top of the canvas displays a reference image and its corresponding reference text. Below the reference image is a reference video clip generated based on that image. To the right of the reference image is its corresponding toolbar, which includes a "Regenerate" control for regenerating the image. Clicking this control will display a task waiting area on the canvas, which can be shown to the right of the reference image. After successfully generating the image, the task waiting area will switch to displaying the newly generated reference image.

[0103] Step S209: In response to the text-to-image instruction, a new reference image is generated based on the new reference text by the intelligent agent.

[0104] Optionally, the canvas interface includes a text-to-image control. The terminal, based on the trigger operation of the text-to-image control, obtains new input reference text and generates text-to-image instructions based on the new reference text. (Continue to the next section...) Figure 16 Clicking the text image control in the toolbar will display the canvas interface. Figure 17In the text input area shown, enter the new reference text "Ultraman, clad in armor standing under the blue sky and white clouds, his chest reactor glowing," and click the image generator control in the lower right corner. The text input area will then switch to the task waiting area. After successfully generating the image, the task waiting area will switch to display the newly generated reference image, such as... Figure 18 The effect shown.

[0105] In step S210, in response to the image-to-image command, a new reference image is generated based on the uploaded image by the intelligent agent.

[0106] Optionally, the canvas interface includes an image-generating control. The terminal, based on a trigger operation on the image-generating control, retrieves the uploaded image and generates an image-generating command based on the uploaded image. The uploaded image can be at least one image. The uploaded image can be of any type, such as a face image.

[0107] For example, the terminal, based on a trigger operation on the image-to-image control, obtains the uploaded image and image modification prompts, and generates an image-to-image command based on the uploaded image and image modification prompts. Then, the terminal, through an intelligent agent, modifies the uploaded image according to the image modification prompts to obtain a new reference image.

[0108] Continue to refer to Figure 16 Clicking the graphic control in the toolbar will bring up the canvas interface. Figure 19 The image upload area is shown below. Upload your image in this area, enter a suggestion for image modification in the prompt input box, and click the generate control in the lower right corner. The image upload area will then switch to a task waiting area. After successfully generating the image, the image upload area will switch to display the newly generated reference image.

[0109] Continue to refer to Figure 16 Clicking the multi-image generator control in the toolbar will display a canvas interface. Figure 20 The image upload area is shown below. This area can upload multiple images. After uploading multiple images, enter your desired image editing suggestions in the prompt input box, and click the "Generate Image" control in the lower right corner. The image upload area will then switch to a task waiting area. Once the image is successfully generated, the image upload area will switch to display the newly generated reference image.

[0110] Continue to refer to Figure 16 Clicking the face image control in the toolbar will display a canvas interface. Figure 21The image upload area is shown below. Upload a face image in this area, enter your desired image modification suggestions in the prompt input box, and click the "Generate Image" control in the lower right corner. The image upload area will then switch to a task waiting area. After successfully generating the image, the image upload area will switch to display the newly generated reference image.

[0111] Step S211: In response to the image modification command, the intelligent agent modifies the existing reference image in the canvas interface based on the input image modification prompt to obtain a new reference image.

[0112] Optionally, the canvas interface includes an image modification control. The terminal obtains the input image modification prompt based on the trigger operation of the image modification control, and generates an image modification command based on the image modification prompt.

[0113] Continue to refer to Figure 16 The control displayed in the toolbar with "Edit Image in One Sentence" is the image editing control. Clicking it will open the canvas interface. Figure 22 The image editing area is shown below. A reference image is displayed in this area. Enter the image editing prompt "Little Rabbit Transforms into Ultraman" in the prompt box below the reference image, and click the "Generate Image" control in the lower right corner. The image editing area will then switch to a task waiting area. After successfully generating the image, the image editing area will switch to display the newly generated reference image, such as... Figure 23 The effect shown.

[0114] In step S212, in response to the region redraw instruction, the target region of the existing reference image in the canvas interface is redrawn by the intelligent agent to obtain a new reference image.

[0115] Optionally, the canvas interface includes a region redrawing control. Based on the trigger operation of the region redrawing control, the terminal determines the target region selected from the existing reference image, and generates a region redrawing instruction based on the target region. The target region in the reference image can be selected in various ways. For example, the target region can be drawn with a brush, outlined with lines, or enclosed with a rectangle, etc. This embodiment does not limit the selection.

[0116] For example, based on the trigger operation of the region redraw control, the terminal determines the target region selected in the existing reference image and the redraw prompt information for the target region, and generates a region redraw instruction based on the target region and the redraw prompt information. Then, the terminal, through an intelligent agent, redraws the target region of the reference image according to the redraw prompt information to obtain a new reference image.

[0117] Continue to refer to Figure 16 Clicking the area redraw control in the toolbar will display the canvas interface. Figure 24The image editing area is shown below. A reference image is displayed in this area. The user selects the target area within the reference image (i.e., the area where the rabbit on the right is located), then enters the redraw prompt "Remove the rabbit on the right" in the prompt input box below. Clicking the redraw control in the lower right corner will switch the image editing area to a task waiting area. After successfully generating the image, the image editing area will switch to display the newly generated reference image, such as... Figure 25 The effect shown.

[0118] Step S213: In response to the image element update instruction, the target element in the existing reference image in the canvas interface is adjusted by the intelligent agent to obtain a new reference image. The adjustment of the target element includes: adjusting at least one of the quantity and layout of the target element, or replacing the target element with a new element.

[0119] Optionally, the canvas interface includes an image element update control. Based on the trigger operation of this control, the terminal obtains the element modification conditions. These conditions indicate the target element to be modified and its modification objective. Based on these conditions, an image element update instruction is generated. Then, through an intelligent agent, the terminal adjusts the target element in the existing reference image to conform to the modification objective, resulting in a new reference image.

[0120] Continue to refer to Figure 16 The control displayed in the toolbar with "Edit Element" is the image element update control. Clicking this control will open the canvas interface. Figure 26 The image editing area is shown below. A reference image is displayed in this area. Enter the desired modification conditions in the prompt box below the reference image, such as "remove the rabbit on the right." Clicking the "Generate Image" control in the lower right corner will switch the image editing area to a task waiting area. After successfully generating the image, the image editing area will switch to display the newly generated reference image, such as... Figure 25 The effect shown.

[0121] Optionally, the terminal generates an image element update instruction based on the selection operation of target elements in an existing reference image. The selection operation of target elements includes various methods such as box selection, point selection, and circle selection.

[0122] For example, a user selects a target element in a reference image, such as a person or prop. After selection, a quick operation menu automatically pops up. This menu can include options such as adjusting the quantity, modifying the layout, and replacing elements. The user selects the corresponding option and sets specific modification parameters, such as changing the quantity from 2 to 4. The terminal generates an image element update instruction based on the selected target element, the selected modification item, and the specific modification parameters. Then, an intelligent agent adjusts the target element in the reference image according to the selected modification item and specific modification parameters to obtain a new reference image.

[0123] For example, after a user selects a target element in a reference image, they click the voice input button on the interface and speak their desired modification, such as "replace the selected red balloon with a yellow star." The terminal uses an intelligent agent to perform semantic analysis on the selected target element and the spoken words to determine the target element to be modified and its modification target, thus generating a corresponding image element update instruction. Alternatively, after selecting a target element, the user can directly enter the text description "replace the selected red balloon with a yellow star" through the modification prompt input box on the canvas interface. In this case, the terminal uses an intelligent agent to perform semantic analysis on the selected target element and the text description to determine the target element to be modified and its modification target, thereby generating a corresponding image element update instruction. Then, the terminal uses the intelligent agent to adjust the target element in the existing reference image to conform to the modification target, resulting in a new reference image.

[0124] Step S214: Generate a new video clip based on the new reference image.

[0125] For example, a new reference image in the canvas interface displays a corresponding video conversion control. Clicking this control causes the terminal to convert the new reference image into a new video clip via an intelligent agent. All reference images and video clips generated by the terminal are recorded in the canvas interface for the user to select and use.

[0126] This application's embodiments support obtaining new reference images through various methods, including image regeneration, text-to-image, image-to-image, image modification based on prompts, region redrawing, and image element updates. This provides full-scene flexibility for image creation, enabling everything from completely new generation to partial modifications. Furthermore, it can generate new video clips based on the new reference images, forming a complete chain from image material innovation to video material derivation. This enriches the diversity and relevance of process materials in the canvas, ultimately lowering the barrier to cross-type material creation and improving the overall efficiency and innovation of content generation.

[0127] This application's embodiments feature diverse controls on the canvas interface, including image regeneration, text-to-image, image-to-image, image modification, region redrawing, and image element updating. The logic for generating instructions after each control is clearly defined, such as uploading images, inputting modification prompts, selecting target areas, and selecting target elements. This allows users to quickly generate the necessary image creation instructions simply by triggering the corresponding controls and providing basic information, flexibly meeting the needs of all scenarios, from complete image generation to detailed optimization. This significantly lowers the technical barrier to image design and comprehensively improves the convenience and efficiency of image creation.

[0128] Based on the first embodiment of this application described above, a fourth embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. See also... Figure 27 In the fourth embodiment, step S20 includes steps S215 to S216.

[0129] Step S215: Obtain the modification instruction information input for the existing process material in the canvas interface. The modification instruction information indicates the content to be modified.

[0130] The existing process materials in the canvas interface include reference images and reference video clips. Correspondingly, the modification instructions can indicate the content to be modified in the reference images or reference video clips.

[0131] The content to be modified in the reference image can include the style, quantity, color, layout, and style of specific elements in the reference image.

[0132] The content to be modified in the reference video clip may include the lip-syncing of the characters in the reference video clip, the content of the lines, the composition elements of the video frame, the style, quantity, layout, and dynamic effects of specific frame elements, the sound effects of the reference video clip, the video style, and the camera movement effects, etc.

[0133] For example, the modification instruction information can be input in various forms such as text and voice, and this application embodiment does not limit this.

[0134] Step S216: Using an intelligent agent, based on the modification instruction information, adjust the content to be modified in the existing process material to obtain new process material.

[0135] The modification instruction is expressed in natural language. The terminal uses an intelligent agent to perform semantic recognition on the modification instruction, determine the content to be modified, and then adjust the content to be modified in the existing process material to obtain new process material.

[0136] Optionally, the modification instruction information also includes the modification target of the content to be modified. Based on the modification instruction information, the intelligent agent adjusts the content to be modified in the existing process material to obtain new process material, including: adjusting the content to be modified in the existing process material to conform to the modification target based on the modification instruction information to obtain new process material.

[0137] The modification target of the content to be modified refers to the final expected state or effect set for the content to be modified during the adjustment process. It clarifies what the content to be modified should look like and is the core basis for the intelligent agent to accurately adjust the content, ensuring that the modification result meets the user's needs.

[0138] For example, for a reference image, if the content to be modified is the sky color in the reference image, the modification goal can be set to change the gray sky to a gradient pink-orange sunset. For a reference video clip, if the content to be modified is the character's dialogue in the reference video clip, the modification goal can be set to replace the original dialogue "Welcome" with "It's nice to meet you," and the character's lip movements should match the new dialogue.

[0139] For example, in a reference video clip on the canvas interface, the scene where the character is located features slowly falling raindrops. The user inputs modification instructions for this reference video clip, such as entering "change the falling raindrops in the scene to cherry blossom petals falling from the sky" in the prompt input box below the reference video clip. The terminal then uses an intelligent agent to perform semantic recognition on the modification instructions, determining that the content to be modified is the background dynamic element of raindrops, and the modification target is to change the raindrops to falling cherry blossom petals. Subsequently, the intelligent agent processes the reference video clip to adjust the background dynamic element of raindrops to falling cherry blossom petals, resulting in a new video clip.

[0140] This application embodiment obtains modification instructions from the user regarding existing process materials, and the intelligent agent precisely adjusts the materials based on this information, achieving a direct connection between user needs and intelligent execution. This avoids the blind modification by the intelligent agent, ensuring that new materials accurately match user expectations, and simplifies the operation process. Users only need to specify the direction of modification to generate new process materials, significantly improving the accuracy and efficiency of material generation and lowering the threshold for personalized creation.

[0141] This application's embodiments, by clearly specifying the content to be modified and the modification target in the modification instruction information, allow the intelligent agent to directly and accurately adjust existing process materials, avoiding modification deviations caused by ambiguous requirements. This further improves the accuracy and success rate of material modification, and can efficiently meet users' customized material needs.

[0142] Another point to note is that the above examples are only for understanding this application and do not constitute a limitation on the video processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0143] This application also provides a video processing apparatus, please refer to... Figure 28 The video processing device includes: The canvas display module 10 is used to display a canvas interface in response to editing instructions for a target video segment in the target video. The canvas interface contains process materials associated with the target video segment. The material generation module 20 is used to generate new process materials based on the existing process materials in the canvas interface through the intelligent agent, and add the new process materials to the canvas interface. The media replacement module 30 is used to replace the target video segment in the target video with the media indicated by the application instruction in response to an application instruction for at least one process media in the canvas interface.

[0144] Optionally, the existing process materials in the canvas interface include reference video clips and / or reference images on which the reference video clips are generated, and the reference video clips include the target video clip and / or candidate video clips of the target video clip; The new process material includes new video clips generated based on reference video clips and / or reference images.

[0145] Optionally, the new video clip differs from the reference video clip in at least one of the following ways: Sound effects, dialogue corresponding to the character's lip movements, compositional elements of the video frame, number of elements in the video frame, layout of elements in the video frame, dynamic effects of elements in the video frame, camera movement effects of the video frame, and style of the video frame.

[0146] Optionally, the material generation module 20 is configured to perform at least one of the following: In response to the video regeneration instruction, the agent re-converts the video based on the reference image to obtain a new video clip; In response to the lip-sync video generation command, the intelligent agent converts the lip movements of the characters in the reference video clip from those adapted to the original lines to those adapted to the new lines, thus obtaining a new video clip; In response to the sound effect generation command, the agent converts the original sound effect in the reference video clip into a new sound effect to obtain a new video clip; In response to the multi-image video generation command, the intelligent agent performs video conversion based on the reference image and the new image to obtain a new video clip; In response to a video frame to video instruction, an intelligent agent performs video conversion based on at least one frame from a reference video segment to obtain a new video segment. In response to a video element update instruction, the agent adjusts the target element in the reference video clip to obtain a new video clip. The adjustment of the target element includes: adjusting at least one of the number, layout and dynamic effects of the target element, or replacing the target element with a new element. In response to a video style conversion command, the agent adjusts the video style of a reference video clip to obtain a new video clip.

[0147] Optionally, the canvas interface includes various types of controls, and the material generation module 20 is also used to perform at least one of the following: Based on the trigger operation of the video regeneration control, a video regeneration instruction is generated; Based on the trigger operation of the lip-sync video generation control, new lines are obtained and lip-sync video generation instructions are generated based on the new lines. Based on the trigger operation of the sound effect generation control, a sound effect generation command is generated; Based on the trigger operation of the multi-image video generation control, the newly uploaded image is obtained, and a multi-image video generation instruction is generated based on the reference image and the new image. Based on the trigger operation of the video frame to video control, at least one frame image selected from the reference video segment is obtained, and a video frame to video instruction is generated based on the at least one frame image. Based on the trigger operation of the video element update control, the element modification conditions are obtained. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, the video element update instruction is generated. Based on the selection operation of target elements in the reference video clip, generate video element update instructions; Based on the trigger operation of the video style conversion control, a video style conversion instruction is generated.

[0148] Optionally, the new process material also includes new reference images generated by the intelligent agent, and new video clips generated based on the new reference images; wherein, the material generation module 20 is further configured to perform at least one of the following: In response to the image regeneration command, a new reference image is generated based on the reference text through the intelligent agent. The reference text is the basis for generating the existing reference image in the canvas interface. In response to the text-to-image command, a new reference image is generated based on the new reference text through an intelligent agent; In response to the image-to-image command, a new reference image is generated based on the uploaded image through an intelligent agent; In response to the image modification command, the intelligent agent modifies the existing reference image on the canvas interface based on the input image modification prompts to obtain a new reference image; In response to the region redraw command, the intelligent agent redraws the target region of the existing reference image in the canvas interface to obtain a new reference image. In response to an image element update command, the agent adjusts the target element in an existing reference image on the canvas to obtain a new reference image. Adjusting the target element includes: adjusting at least one of the quantity and layout of the target element, or replacing the target element with a new element.

[0149] Optionally, the canvas interface includes various types of controls, and the material generation module 20 is also used to perform at least one of the following: Based on the trigger operation of the image regeneration control, an image regeneration instruction is generated; Based on the trigger operation of the text-to-image control, obtain the new reference text input, and generate text-to-image instructions based on the new reference text; Based on the trigger operation of the image-generating control, the uploaded image is obtained, and an image-generating command is generated based on the uploaded image; Based on the trigger operation of the image modification control, obtain the input image modification prompt, and generate the image modification command based on the image modification prompt; Based on the trigger operation of the region redraw control, the target region selected in the existing reference image is determined, and a region redraw instruction is generated based on the target region. Based on the trigger operation of the image element update control, obtain the element modification conditions. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, generate the image element update instruction. Based on the selection operation of target elements in the existing reference image, an image element update instruction is generated.

[0150] Optionally, the material generation module 20 includes: The instruction acquisition unit is used to acquire modification instruction information input in the canvas interface for existing process materials, and the modification instruction information indicates the content to be modified. The material adjustment unit is used by an intelligent agent to adjust the content to be modified in existing process materials based on modification instructions, so as to obtain new process materials.

[0151] Optionally, the modification instruction information may also include the target of the modification to be made. The material adjustment unit is used by an intelligent agent to adjust the content to be modified in existing process materials to meet the modification target, thereby obtaining new process materials.

[0152] Optionally, the material generation module 20 is used to iteratively execute the steps of generating new process materials based on existing process materials in the canvas interface through the intelligent agent, and adding the new process materials to the canvas interface.

[0153] Optionally, the material generation module 20 is used to determine the target number of new process materials to be generated based on the material quantity specification control in the canvas interface, and generate the target number of new process materials based on the existing process materials in the canvas interface through an intelligent agent; and / or, to determine the target intelligent agent for generating new process materials based on the intelligent agent specification control in the canvas interface, and generate new process materials based on the existing process materials in the canvas interface through the target intelligent agent.

[0154] Optionally, the canvas interface includes multiple material display areas, with different material display areas used to display different types of process materials, including video and image types; The material display area corresponding to the process material of the video type includes video clips and the lines corresponding to the characters' lip movements in the video clips, or icons of the sound effects of the video clips, or intelligent agent prompts used to generate the video clips; The material display area corresponding to the image-type process material includes the image and the intelligent agent prompt words used to generate the image.

[0155] The video processing apparatus provided in this application, employing the video processing method described in the above embodiments, can solve the technical problem of low practicality caused by the inconsistency between the video generated by the intelligent agent and the user's expectations. Compared with the prior art, the beneficial effects of the video processing apparatus provided in this application are the same as those of the video processing method described in the above embodiments, and other technical features in the video processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0156] This application provides a video processing apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the video processing method described above.

[0157] The following is for reference. Figure 29The diagram illustrates a structural schematic of a video processing device suitable for implementing embodiments of this application. The video processing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 29 The video processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0158] like Figure 29 As shown, the video processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the video processing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the video processing equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows video processing equipment with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0159] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0160] The video processing device provided in this application, employing the video processing method described in the above embodiments, can solve the technical problem of low practicality caused by the inconsistency between the video generated by the intelligent agent and the user's expectations. Compared with the prior art, the beneficial effects of the video processing device provided in this application are the same as those of the video processing method provided in the above embodiments, and other technical features of the video processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0163] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the video processing method described in the above embodiments.

[0164] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0165] The aforementioned computer-readable storage medium may be included in the video processing device; or it may exist independently and not assembled into the video processing device.

[0166] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a video processing device, cause the video processing device to: display a canvas interface containing process material associated with the target video segment in response to an editing instruction for a target video segment in the target video; generate new process material based on existing process material in the canvas interface using an intelligent agent, and add the new process material to the canvas interface; and replace the target video segment in the target video with the process material indicated by the application instruction in response to an application instruction for at least one process material in the canvas interface.

[0167] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0169] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0170] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video processing method. This solves the technical problem in related technologies where the video generated by the intelligent agent does not match user expectations and has low practicality. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video processing method provided in the above embodiments, and will not be repeated here.

[0171] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video processing method described above.

[0172] The computer program product provided in this application can solve the technical problem that the videos generated by intelligent agents in related technologies do not meet user expectations and have low practicality. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the video processing method provided in the above embodiments, and will not be repeated here.

[0173] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A video processing method, characterized in that, The method includes: In response to an editing instruction on a target video segment in a target video, a canvas interface is displayed, the canvas interface containing process materials associated with the target video segment; Using an intelligent agent, new process materials are generated based on existing process materials in the canvas interface, and the new process materials are added to the canvas interface. In response to an application instruction on at least one process material in the canvas interface, the target video segment in the target video is replaced with the process material indicated by the application instruction.

2. The method as described in claim 1, characterized in that, The existing process materials in the canvas interface include reference video clips and / or reference images on which the reference video clips are generated, and the reference video clips include the target video clip and / or candidate video clips of the target video clip; The new process material includes new video clips generated based on the reference video clips and / or the reference images.

3. The method as described in claim 2, characterized in that, The new video clip differs from the reference video clip in at least one of the following ways: Sound effects, dialogue corresponding to the character's lip movements, compositional elements of the video frame, number of elements in the video frame, layout of elements in the video frame, dynamic effects of elements in the video frame, camera movement effects of the video frame, and style of the video frame.

4. The method as described in claim 3, characterized in that, The process of generating new process materials based on existing process materials in the canvas interface through an intelligent agent includes at least one of the following: In response to the video regeneration instruction, the intelligent agent re-converts the video based on the reference image to obtain the new video segment; In response to the lip-sync video generation command, the intelligent agent converts the lip movements of the characters in the reference video clip from those adapted to the original dialogue to those adapted to the new dialogue, thereby obtaining the new video clip; In response to the sound effect generation command, the agent converts the original sound effect in the reference video clip into a new sound effect to obtain the new video clip; In response to the multi-image video generation instruction, the intelligent agent performs video conversion based on the reference image and the new image to obtain the new video segment; In response to a video frame to video conversion instruction, the agent performs video conversion based on at least one frame of the reference video segment to obtain the new video segment. In response to a video element update instruction, the agent adjusts the target element in the reference video segment to obtain the new video segment. The adjustment of the target element includes: adjusting at least one of the number, layout, and dynamic effects of the target element, or replacing the target element with a new element. In response to a video style conversion instruction, the agent adjusts the video style of the reference video segment to obtain the new video segment.

5. The method as described in claim 4, characterized in that, The canvas interface contains various types of controls, and the method further includes at least one of the following: Based on the trigger operation of the video regeneration control, the video regeneration instruction is generated; Based on the trigger operation of the lip-sync video generation control, new lines are obtained, and the lip-sync video generation instruction is generated based on the new lines; Based on the trigger operation of the sound effect generation control, the sound effect generation instruction is generated; Based on the trigger operation of the multi-image video generation control, the uploaded new image is obtained, and the multi-image video generation instruction is generated based on the reference image and the new image. Based on the trigger operation of the video frame to video control, at least one frame image selected from the reference video segment is obtained, and based on the at least one frame image, the video frame to video instruction is generated. Based on the trigger operation of the video element update control, the element modification conditions are obtained. The element modification conditions indicate the target element to be modified and the modification target of the target element. Based on the element modification conditions, the video element update instruction is generated. Based on the selection operation of the target element in the reference video segment, the video element update instruction is generated; The video style conversion instruction is generated based on the trigger operation of the video style conversion control.

6. The method as described in claim 2, characterized in that, The new process materials also include new reference images generated by the intelligent agent, and new video clips generated based on the new reference images; wherein, the method for obtaining the new reference images includes at least one of the following: In response to the image regeneration instruction, the agent generates the new reference image based on the reference text, where the reference text is the basis for generating the existing reference image in the canvas interface. In response to the text-to-image command, the intelligent agent generates the new reference image based on the new reference text. In response to the image-to-image command, the intelligent agent generates the new reference image based on the uploaded image; In response to the image modification command, the intelligent agent modifies the existing reference image in the canvas interface based on the input image modification prompt to obtain the new reference image; In response to the region redraw command, the intelligent agent redraws the target region of the existing reference image in the canvas interface to obtain the new reference image. In response to an image element update command, the agent adjusts the target element in an existing reference image on the canvas interface to obtain the new reference image. The adjustment of the target element includes: adjusting at least one of the quantity and layout of the target element, or replacing the target element with a new element.

7. A video processing apparatus, characterized in that, The device includes: The canvas display module is used to display a canvas interface in response to editing instructions for a target video segment in the target video. The canvas interface contains process materials associated with the target video segment. The material generation module is used to generate new process materials based on the existing process materials in the canvas interface through an intelligent agent, and add the new process materials to the canvas interface; The media replacement module is used to replace the target video segment in the target video with the media indicated by the application instruction in response to an application instruction for at least one process media in the canvas interface.

8. A video processing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video processing method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the video processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the video processing method as described in any one of claims 1 to 6.

Images