Agent-based video generation method and device, agent, and electronic device

Abstract

The present disclosure provides an agent-based video generation method and device, an agent and an electronic device, relates to the technical field of artificial intelligence, and in particular to the technical field of computer vision, large models, agents and the like. The agent-based video generation method comprises: using an agent to perform the following operations: performing a target task on a reference video frame in a source video according to a received requirement text to obtain a prompt video frame, wherein a visual prompt element in the prompt video frame indicates editing information for a target picture area in the source video, and the target picture area and the editing information match an intention of the requirement text; and performing an editing task on the source video according to the prompt video frame to generate a target video.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, large models, and intelligent agents. Background Technology

[0002] In scenarios such as game development and film and television production, users edit video footage to obtain video content that meets their specific needs. Summary of the Invention

[0003] This disclosure provides a video generation method, apparatus, intelligent agent, and electronic device based on intelligent agents.

[0004] According to one aspect of this disclosure, a video generation method based on an intelligent agent is provided, comprising: using an intelligent agent to perform the following operations: performing a target task on a reference video frame in a source video according to a received request text to obtain a prompt video frame, wherein visual cue elements in the prompt video frame indicate editing information for a target screen area in the source video, and the target screen area and editing information match the intent of the request text; performing an editing task on the source video according to the prompt video frame to generate a target video.

[0005] According to another aspect of this disclosure, an agent-based video generation apparatus is provided, comprising: a first execution module, configured to perform a target task on a reference video frame in a source video according to a received request text to obtain a prompt video frame, wherein visual cue elements in the prompt video frame indicate editing information for a target screen area in the source video, and the target screen area and editing information match the intent of the request text; and a second execution module, configured to perform an editing task on the source video according to the prompt video frame to generate a target video.

[0006] According to another aspect of this disclosure, an artificial intelligence agent is provided, configured to perform the methods provided in embodiments of this disclosure.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to an embodiment of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method provided according to an embodiment of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to embodiments of this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 The illustration schematically shows an exemplary system architecture for applying agent-based video generation methods and apparatus according to embodiments of the present disclosure.

[0013] Figure 2 A flowchart illustrating an agent-based video generation method according to an embodiment of the present disclosure is shown.

[0014] Figure 3 The diagram illustrates an application scenario of the agent-based video generation method according to an embodiment of the present disclosure.

[0015] Figure 4 The diagram illustrates an application scenario of another embodiment of the agent-based video generation method of this disclosure.

[0016] Figure 5 The schematic diagram illustrates the principle of the agent-based video generation method according to an embodiment of the present disclosure.

[0017] Figure 6 A block diagram of an agent-based video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 7 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0019] Figure 8 A schematic block diagram of an example electronic device that can be used to implement the agent-based video generation method of the embodiments of this disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In the technical solutions disclosed herein, the acquisition, storage, and application of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0022] An intelligent agent can be a software system with autonomous planning and decision-making capabilities, and the ability to invoke tools to complete complex tasks. For example, an intelligent agent can be used to invoke video editing tools to process videos to be edited, so as to conveniently obtain video content that meets actual needs. However, the edited videos may have quality defects such as errors in the video content, requiring users to perform multiple manual editing operations to correct the videos, resulting in long video editing times and making it difficult to accurately meet the user's actual needs.

[0023] Embodiments of this disclosure provide a video generation method, apparatus, agent, and electronic device based on an intelligent agent. The video generation method based on an intelligent agent includes: using an intelligent agent to perform the following operations: performing a target task on a reference video frame in a source video according to a received request text to obtain a prompt video frame, wherein visual cue elements in the prompt video frame indicate editing information for a target screen area in the source video, and the target screen area and editing information match the intent of the request text; performing an editing task on the source video according to the prompt video frame to generate a target video.

[0024] According to embodiments of this disclosure, by utilizing an intelligent agent to process requirement text and perform target tasks on reference video frames in the source video, a prompt video frame displaying visual cues is generated. The position of the visual cues in the prompt video frame can accurately indicate the target screen area that needs video editing, and the visual cues provide editing information for the editing operation on the source video. Thus, by converting requirement text based on natural language representation into prompt video frames as visual cues, the intelligent agent can be prompted to perform editing tasks. This allows the intelligent agent to edit the source video more accurately according to the intent of the requirement text, generating a target video that matches the intent more precisely, thereby improving the editing efficiency and quality of the target video.

[0025] Figure 1 The illustration schematically shows an exemplary system architecture for applying agent-based video generation methods and apparatus according to embodiments of the present disclosure.

[0026] It is important to note that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture applicable to the agent-based video generation method and apparatus may include a terminal device. However, the terminal device can implement the agent-based video generation method and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0029] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0031] It should be noted that the agent-based video generation method provided in this disclosure can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the agent-based video generation device provided in this disclosure can also be disposed in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0032] Alternatively, the agent-based video generation method provided in this disclosure can generally be executed by server 105. Correspondingly, the agent-based video generation apparatus provided in this disclosure can generally be located in server 105. The agent-based video generation method provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the agent-based video generation apparatus provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0034] Figure 2 A flowchart illustrating an agent-based video generation method according to an embodiment of the present disclosure is shown.

[0035] like Figure 2 As shown, the agent-based video generation method includes performing operations S210~S220 using an agent.

[0036] In operation S210, the target task is performed on the reference video frame in the source video according to the received request text, and the prompt video frame is obtained.

[0037] When operating S220, the source video is edited according to the prompt video frames to generate the target video.

[0038] In embodiments of this disclosure, the request text represents natural language text for editing the source video. For example, the request text could be "remove the table from the video" or "turn the car in the video red." The request text can be determined by the target object through any type of interactive operation, such as text input or voice input.

[0039] The source video includes multiple source video frames arranged in a temporal sequence. The agent determines the reference video frame from these multiple source video frames by invoking a large model or target tool. The reference video frame can be a source video frame that matches the intent of the demand text, and there can be one or more reference video frames.

[0040] In embodiments of this disclosure, visual cue elements in the prompt video frame indicate editing information for a target area in the source video, and the target area and editing information match the intent of the demand text.

[0041] The agent performs visual semantic understanding on reference video frames and textual semantic understanding on the intent of the request text to determine the target screen region matching the intent of the request text, as well as the editing information for editing operations on the target screen region represented by the textual semantics of the request text. The target screen region can be, for example, an image patch in the reference video frame, or it can be an image region in the reference video frame related to the screen object to be edited.

[0042] Visual cues may include icon elements displayed in a reference video frame. The display position of the visual cues or the position indicated by the visual cues represents a target screen area. For example, the visual cues may be displayed in the target screen area of ​​the reference video frame, or the visual cues may be displayed in a target display position in the reference video frame that corresponds to the target screen area.

[0043] Editing information represents the execution parameters for editing operations such as deleting, adding, modifying shapes, and changing colors. Visual cue elements can represent editing information for a target image area using any type of visual attribute, such as shape or texture. Therefore, visual cue elements can be used to represent the operation information for editing a reference video frame through visual semantics.

[0044] In the embodiments of this disclosure, the intelligent agent processes the cue video frame and the source video by invoking a video editing tool. This allows the visual content of the source video to be edited based on the visual semantics of the visual cue elements displayed in the cue video frame as execution parameters. This reduces information loss and semantic understanding errors caused by editing the source video with cross-modal cue information, improves the accuracy of editing operations on the source video, and enables the video content of the target video to match the intent of the demand text more accurately, thereby improving the video quality of the target video.

[0045] In some embodiments, performing editing tasks on the source video based on cue video frames may further include: using a large model to perform editing tasks on the source video based on cue video frames to obtain a target video. This allows the large model to more accurately process the content related to the target area in one or more source video frames according to the editing information, based on the visual cue elements in the cue video frames and the visual semantic content of the reference video frames. This avoids the large model's difficulty in accurately editing the content of the source video frames due to cross-modal cue information understanding, reduces errors in the large model's execution of editing operations and editing information, and improves the video quality of the target video.

[0046] In some embodiments, performing editing tasks on the source video based on prompt video frames may further include: using a large model to perform editing tasks on the source video based on prompt video frames and editing information.

[0047] The editing information and the prompt video frames are correlated. By understanding the correlation between the editing information and the prompt video frames, the large model can accurately determine the source video frame to be edited corresponding to the prompt video frame from the source video. Thus, the large model can accurately locate the target screen area in the source video frame to be edited based on the visual semantic cues of the visual cue elements in the prompt video frame, and edit the source video frame to be edited based on the visual attributes of the visual cue elements. At the same time, based on the associated editing information, it can further accurately edit the screen content of the target screen area, so as to supplement and explain the editing information of the visual semantic indication, improve the editing quality of the source video for complex editing operations, and improve the video quality of the target video and the degree of matching with the intent of the demand text.

[0048] In some embodiments, the element attributes of the visual cue element are used to indicate the execution parameters of the editing task, and the execution parameters include at least one of the following: the operation type of the editing operation, the texture attribute parameters related to the editing task, and the movement direction of the object to be edited in the target video.

[0049] The element attributes of visual cues can be any visual attributes that can be displayed, such as shape, color, and display position.

[0050] Editing operations include deleting, adding, changing, and adjusting the shape and color of objects in the image. By utilizing the element attributes of visual cues to indicate the operation type, the agent can more accurately perform editing operations on the target area in the source video based on the element attributes of the visual cues. This avoids defects such as incorrect editing operation types and mismapping between complex editing operations and the target area, thereby improving the video quality of the target video.

[0051] Texture attribute parameters indicate how to edit the texture properties of the source video frame's content. For example, texture attribute parameters might indicate modifying the colors of objects like tables or buildings in the source video frame. Alternatively, texture attribute parameters might indicate converting the source video frame's content to a two-dimensional style. Therefore, texture attribute parameters can be combined with other editing information to improve the accuracy of editing the source video frame's content.

[0052] In some embodiments, visual cues can indicate the direction of movement using display elements such as arrows or fingers. The object to be edited is a scene object displayed in the target video. Therefore, the display position of the visual cues indicates the display or movement position related to the object to be edited. The movement direction indicated by the visual cues prompts the agent to move the object to be edited within the target video, ensuring that the target video displays video content showing the object moving in the desired direction. This avoids misinterpretation of cross-modal cues by the agent, preventing a mismatch between the movement direction of the object to be edited and the intent of the request text.

[0053] It should be noted that, in the embodiments of this disclosure, the element attributes of the visual prompt elements can represent one or more execution parameters, and the number of element attributes representing execution parameters can also be one or more. The embodiments of this disclosure do not limit the number of element attributes or the number of execution parameters corresponding to the element attributes.

[0054] In some embodiments, performing a target task on a reference video frame in a source video based on a received request text may include: performing visual semantic understanding on the reference video frame based on the received request text to obtain editing instructions; generating visual cue elements based on the editing instructions using a large model; and updating the target screen area of ​​the reference video frame based on the visual cue elements to obtain a cue video frame.

[0055] In embodiments of this disclosure, the editing instruction includes associated region information and editing information, whereby the region information indicates a target frame region of a reference video frame. The region information, for example, represents the image coordinates of a region in the reference video frame that matches the editing intent of the request text for the source video. Alternatively, the region information may also be the vertex coordinates of the target frame region in the reference video frame that matches the editing intent.

[0056] The editing information indicates the execution parameters for editing operations on the target screen area. By understanding the relationship between the editing information and the area information, and by semantically understanding the content of the reference video frame, the large model can determine the display position of the visual cue element in the cue video frame and the element attributes corresponding to the editing information. Furthermore, the large model can be used to generate visual cue elements that can represent editing instructions based on element attributes by processing editing commands.

[0057] For example, the editing instructions include location information "location coordinates C", editing information "Brand A vehicle", "modify", and "red". "Location coordinates C" represents the outline boundary coordinates of the object to be edited, "Brand A vehicle," in the reference video frame. Therefore, the large visual model is used to process the editing instructions to perform the image generation task, outputting a red border that matches the location coordinates C, making this red border a visual cue element. Furthermore, a red border can be added to the target image area corresponding to "Brand A vehicle" in the reference video frame based on location coordinates C, resulting in a cue video frame. The display area of ​​the red border in the cue video frame matches the outline boundary of the target image area corresponding to "Brand A vehicle". Thus, the large visual model can be used to perform video editing tasks on the source video based on the cue video frame, outputting a target video where the target image area displays "Brand A vehicle" in red.

[0058] In the embodiments of this disclosure, by utilizing a large model to process the request text to obtain structured editing instructions, the editing instructions can more accurately represent the editing intent of the request text for the source video. Thus, the agent, by using the large model to generate visual cue elements based on the editing instructions, can ensure that the element attributes of the visual cue elements accurately display the execution parameters of the editing information. Furthermore, based on region information, the visual cue elements are added to the reference video frame, enabling the resulting cue video frame to accurately indicate the editing information for the target area of ​​the source video. Moreover, the visual semantics of the visual cue elements in the cue video frame and the content of the reference video frame can be used to prompt the execution parameters of the editing operation, thereby improving the accuracy of the editing operation on the source video frame, improving the video quality of the target video, reducing the interaction time and computational overhead caused by user modifications to the target video, and enhancing the user experience.

[0059] It should be noted that the large model involved in this embodiment can be a multimodal vision large model used to process multimodal data. The information output by the large model can be data of any modality, such as images, text, symbols, and videos.

[0060] In some embodiments, performing visual semantic understanding on a reference video frame based on the received request text to obtain editing instructions may include: performing intent understanding on the request text to obtain structured editing information; and using a large visual model to perform visual semantic understanding on the reference video frame based on the structured editing information to determine region information.

[0061] For example, the agent invokes a Large Language Model (LLM) to understand the intent of the request text and outputs structured editing information. The editing information can be structured data; for example, the editing information may include multiple parts, such as the object to be edited, the type of editing operation, and the color attribute to be edited.

[0062] The visual big model performs visual semantic understanding of reference video frames based on structured editing information. It processes the visual semantic content of reference video frames according to structured editing information that precisely matches the intent of the required text. This allows it to determine region information that matches the structured editing information, indicating the target screen area corresponding to the editing task. Thus, the associated region information and editing information can accurately represent the editing intent for the reference video frame, improving the accuracy of visual cues in representing editing information. Furthermore, the visual cues displayed in the prompt video frame can guide the agent to accurately invoke the visual big model to perform editing tasks on the source video based on more accurate visual prompts, outputting a target video that accurately meets the editing intent.

[0063] In some embodiments, the method further includes using an agent to perform the following operations: determining tool execution parameters based on structured editing information, and using a target tool to determine reference video frames from source video frames of the source video based on the tool execution parameters.

[0064] The target tool can be a tool with video frame filtering capabilities. For example, the target tool can be a visual inspection tool with image content recognition capabilities. The agent calls upon the visual inspection tool resources to perform visual inspection on the source video frames based on the text in the structured editing information, and obtains reference video frames that match the object to be edited and the target image area to be edited as indicated in the editing information.

[0065] For example, an intelligent agent can use visual inspection tools to detect source video frames that are related to the structured edit information "Brand A vehicles" as reference video frames.

[0066] In some embodiments, the method further includes using an agent to perform the following operation: using a large model to process at least one source video frame based on structured editing information to obtain a reference video frame in the source video that matches the intent.

[0067] For example, the agent uses structured editing information as input prompts to the large model, and uses the large model to perform visual semantic understanding on multiple source video frames based on the prompts, so as to obtain source video frames that match the objects and regions to be edited as indicated by the editing information.

[0068] In the embodiments of this disclosure, by utilizing an agent to invoke a target tool or a large model to determine reference video frames from the source video based on structured editing information, the agent's task planning and decision-making capabilities can be leveraged to determine source video frames that can more accurately meet the intent of the required text as reference video frames. This allows the visual content related to visual cue elements in the prompt video frame to provide accurate prompts to the large model performing the visual editing task. This avoids significant differences between the content of the prompt video frame and the source video frame to be edited, which could lead to illusions and execution parameter errors in the large model during the editing task. This improves the control accuracy of the editing task, reduces editing task execution errors, and enhances the quality of the target video.

[0069] In some embodiments, generating visual cue elements using a large model based on editing instructions may include: using a large model, performing a generation task based on editing information and reference video frames to obtain visual cue elements.

[0070] For example, the large model performs a generation task based on the object to be edited indicated by the editing information and the visual semantics of the blue lane in the reference video frame, outputting yellow visual cue elements. This allows the large model performing the editing task to understand the editing information indicated by the visual cue elements and the content of the reference video frame more accurately and clearly, thereby improving the accuracy of the editing task execution.

[0071] In some embodiments, the large model can also understand the content of the reference video frame based on the operation parameters of the editing operation indicated by the editing information, so that the shape, color, and other element attributes of the output visual cue elements can be adapted to the target image area in the reference video frame. This ensures that the cue elements in the cue video frame can be understood more clearly and accurately during the editing task, avoiding confusion between the visual cue elements in the cue video frame and the content of the reference video frame, which could lead to illusions in the editing task's understanding of the visual semantics of the cue video frame. This further improves the accuracy of the editing task and enhances the video quality of the target video.

[0072] In some embodiments, performing the generation task using a large model based on editing information and reference video frames may further include: using object information in the editing information as prompting information, performing visual semantic understanding on the reference video frames using a large model, and outputting visual prompting elements.

[0073] The object information indicates the object to be edited in the reference video frame, and the shape of the visual cue element matches the outline of the object to be edited. For example, if the object to be edited is "Brand A vehicle," the object information prompts the large model to focus on the shape and outline of "Brand A vehicle" in the reference video frame, and then outputs a visual cue box that closely resembles the shape and outline of "Brand A vehicle" as the visual cue element. Thus, by adding the visual cue element to the reference video frame, a cue video frame is obtained, which allows the large model to more accurately guide the large model to edit the target image area within the outline of "Brand A vehicle" in the source video, reducing editing errors caused by editing other image areas in the source video.

[0074] In some embodiments, multiple objects in a reference video frame occlude each other. By using a large model to perform visual semantic understanding of the reference video frame based on object information in the editing information, the table object among the multiple occluded objects can be identified more accurately. The shape of the output visual cue information matches the specific rectangular shape of the table object in the reference video frame, improving the accuracy of the visual cue element in representing the outline edge of the object to be edited in the reference video frame. This improves the accuracy of editing the content of the source video during the editing task, reduces editing operation errors, and improves the quality of the target video.

[0075] In some embodiments, the visual cue element includes an object trajectory element that indicates the movement trajectory of a target image object within a frame of the source video. The object trajectory element can be, for example, a movement trajectory displayed in a reference video frame. The starting point of the object trajectory element can be, for example, the display area of ​​the object to be edited, and the ending point of the object trajectory element can be, for example, the destination area to which the object to be edited needs to be moved, as indicated by the editing information.

[0076] It should be understood that the target screen area in the reference video frame is the area related to the object trajectory element. Thus, during the execution of the editing task, the large model can generate video content of the object to be edited moving according to the object trajectory element based on the target screen area indicated by the object trajectory element in the prompt video frame, thereby obtaining the target video.

[0077] In some embodiments, performing editing tasks on the source video based on the prompt video frame may further include: using the object trajectory element and the background content of the prompt video frame relative to the object trajectory element as prompt information, performing visual semantic understanding on the source video frame using a large model, and outputting motion parameters; and using the motion parameters to edit the source video frame to generate a target video.

[0078] Figure 3The diagram illustrates an application scenario of the agent-based video generation method according to an embodiment of the present disclosure.

[0079] like Figure 3 As shown, the first cue video frame 310 includes a reference video frame from the first source video 320, and visual cue elements displayed in the reference video frame. The visual cue elements include a first bounding box 3121 indicating the drone object 311 and an object trajectory element 3122. The drone object 311 is the object to be edited. The object trajectory element 3122 and other screen content displayed in the first cue video frame 310 represent the editing intent of the request text "Make the drone fly up and down in a U-shape" for the first source video 320.

[0080] By using video editing tools to process the first cue video frame 310 and the first source video 320, a first target video 330 is output. Multiple target video frames played sequentially in the first target video can show the drone object 311 moving along the movement trajectory indicated by the object trajectory element 3122.

[0081] Video editing tools can be built on a visual big model. By using the first prompt video frame 310 as a prompt, the visual big model can learn the editing intention of the first source video 320 based on the object trajectory elements and reference video frames through visual semantic understanding. This allows the tool to execute the editing task more accurately according to the intention of the prompt text, thereby enabling the output first target video 330 to meet the actual intention of moving the object to be edited as represented by the textual semantics of the prompt text. This avoids the shortcomings of cross-modal prompt information in accurately controlling the big model to edit the video according to the intention of the prompt text, reduces video quality defects in the target video, and improves the generation efficiency and quality of the target video.

[0082] In some embodiments, the visual cue element includes an action element, the display position of which in the cue video frame indicates the part of the target image object performing the target action.

[0083] The target image object can be an object displayed in the reference video frame, or it can be an object that does not appear in the reference video frame but the editing information indicates needs to be added to the source video.

[0084] For example, the source video is a shot of the sea surface, and the editing information includes: Person A, right hand raised. The target image is Person A, and the target body part is the right hand. Therefore, a visual cue element can be used to display Person A's image. The visual cue element also includes an action element corresponding to the position of Person A's right hand. The action element can be an arrow element pointing upwards, and the color of the action element can indicate the action type "raised hand".

[0085] Therefore, by using a large model to process visual cues containing action elements displayed in the prompt video frames and the background content relative to the action elements in the prompt video frames as prompt information, the large model can more accurately perform editing tasks on the original video frames based on the visual semantic cues of the prompt video frames. This allows the target video to display the screen content of the target object performing the target action according to the target screen area corresponding to the intent of the demand text, avoiding errors or inaccuracies in fine-grained actions such as finger movements in the target video. This improves the accuracy of the large model in performing editing tasks, reduces the illusion of semantic understanding of prompt information, and ultimately improves the accuracy of the target video's representation of the target action.

[0086] In some embodiments, performing editing tasks on the source video based on the prompt video frame may include: using the action element and the background content of the prompt video frame relative to the action element as prompt information, using a large model to perform visual semantic understanding on the source video frame, and outputting action parameters; and using the action parameters to edit the source video frame to generate a target video.

[0087] In embodiments of this disclosure, the background content in the prompt video frame relative to the action element includes the content of the reference video frame, or the background content in the prompt video frame relative to the action element, and may also include other visual cue elements different from the action element.

[0088] Because the background content of the prompt video frame relative to the action element has a high visual similarity to the content of the source video frame to be edited, the large model can clearly and accurately understand the editing intent of the editing information regarding the content of the source video frame based on the action element in the prompt video frame. This allows the action parameters to more accurately describe the action type, target body part, and direction of the target action performed by the target object. Therefore, the large model can be used to perform editing tasks on the source video frame based on the action parameters, outputting a target video that meets the required textual intent.

[0089] In some embodiments, the visual cue elements include multiple action elements corresponding to multiple target actions. The multiple action elements may correspond to one or more object parts.

[0090] In some embodiments, the visual cue elements also include execution order elements for multiple action elements. For example, visual numerical markers such as 1, 2, 3, etc., can be used as target markers to indicate the execution order of multiple target actions. Alternatively, visual markers such as English letters can also be used as target markers to indicate the execution order of multiple target actions. By utilizing a large model to process the editing instructions obtained from the requirement text, the execution order of multiple target actions can be included. This allows the large model to generate multiple action elements that accurately represent multiple target actions while precisely understanding the editing instructions. This improves the accuracy of the visual representation of the editing intent of the requirement text by the action elements in the cue video frame, thereby improving the execution accuracy of the editing task. It also improves the smoothness and naturalness of the target video in performing multiple target actions on the target screen object, avoiding content that defies physical common sense or physiological structure principles, reducing model illusions generated during the large model's editing of the source video, and improving the quality of the target video. The target video is used to display at least one object part of the target screen object, performing multiple target actions in multiple execution orders. For example, the target video can display the right hand of the target screen object performing multiple target actions in sequence, such as raising the hand, shaking the finger, and lowering it.

[0091] In some embodiments, the visual cue element includes multiple action elements corresponding to multiple target actions. The action attributes in the editing information are used to indicate the execution order of the target actions corresponding to the action elements. The multiple action elements in the visual cue element can be associated with multiple editing attributes. Thus, during the process of the large model performing the editing task on the source video, the target screen object can be driven to perform actions according to the associated editing attributes. This enables the display of screen content in the target video in which the target screen object performs multiple actions in an execution order that matches the intent, thereby improving the video quality of the target video.

[0092] Using action elements and background content relative to action elements in the prompt video frame as prompt information, and using a large model to perform visual semantic understanding on the source video frames in the source video, can also include: using action elements, background content, and action attributes in the prompt video frame as prompt information, using a large model to perform visual semantic understanding on the source video frames in the source video, and outputting multiple action parameters.

[0093] The execution order attribute of multiple action parameters is matched with the execution order of multiple target actions. Therefore, a large model can be used to perform editing tasks on the source video based on the execution order attribute of multiple action parameters, so that the edited target video can represent the screen content of multiple target actions being performed in the execution order.

[0094] Figure 4 The diagram illustrates an application scenario of another embodiment of the agent-based video generation method of this disclosure.

[0095] like Figure 4 As shown, the second cue video frame 410 includes a reference video frame in the second source video 420, and visual cue elements displayed in the reference video frame. The visual cue elements include a second bounding box 4121 indicating a person object and multiple motion elements associated with the second bounding box 4121. The person object is a target scene object that the request text indicates needs to be added to the target video.

[0096] The requirement text could be, for example, "Add a character who stands and performs a gesture of resting their chin on their hand, while also placing their other hand on their hip and spreading their legs." A large model can be used to understand the requirement text, generate multiple editing instructions, and execute the image generation task based on these instructions. This outputs a second bounding box 4121 for the visual cue element, and multiple action elements associated with the second bounding box 4121. These action elements correspond to target markers a, b, and c, respectively, indicating the execution order of actions for the character's right hand, left leg, and left hand, thus executing multiple target actions. The visual editing tool processes the second cue video frame (second target video 410) and the second source video 420, outputting the editing task execution result for the second source video 420 as the second target video 430. The second target video 430 demonstrates the execution process of multiple target actions as the character transitions from a standing posture to the posture specified in the requirement text during the timeline development: "one hand supporting the chin, the other hand on the hip, and legs apart." These target actions are executed in the order of right hand, left leg, and left hand, improving the naturalness and fluency of the character's execution of these actions. This achieves the conversion of cross-modal semantic cues from the requirement text into visual semantic cues, precisely controlling the large visual model to perform editing tasks, reducing model illusions and comprehension errors, and ultimately improving the quality of the target video.

[0097] In the embodiments of this disclosure, the execution order of multiple target actions can be determined by semantic understanding of the request text, and the order attribute among multiple editing information represents the execution order attribute of multiple target actions. Alternatively, the execution order of multiple target actions can also be determined by the large model processing the content of the editing information and the reference video frame based on action knowledge information. For example, the large model performs visual semantic understanding of the editing information "backflip" and the content of the reference video frame based on action knowledge information to determine the execution order of the actions of multiple object parts such as legs, arms, and torso during the "backflip" action of a person. Thus, by decomposing the action indicated by the request text into a combination of multiple target actions, and indicating the execution order of multiple target actions based on multiple action elements, the large model can be more accurately prompted to edit the source video according to the intent of the request text during the editing task, thereby improving the naturalness and smoothness of the target video's content.

[0098] In some embodiments, the agent-based video generation method provided in this disclosure is used in e-commerce video editing scenarios. For example, if the requirement text is: "Change the beverage in the person's hand to coffee," the method provided in this disclosure can edit the source video without masking it, outputting a target video of a person holding coffee and talking.

[0099] In some embodiments, the agent-based video generation method provided in this disclosure is used in advertising video production scenarios. By using demand text to perform editing operations such as deleting, replacing, and modifying the colors of target screen objects displayed in the source video, such as background advertising boards, trademarks, and advertising slogans, the efficiency of advertising video production can be improved.

[0100] In some embodiments, the agent-based video generation method provided in this disclosure is used in short video creation scenarios. It understands editing information represented by requirement text, such as deleting passersby, adding props, modifying the weather, and changing the color of clothing, and outputs prompt video frames that accurately guide the visual big model. This allows the visual big model to process the prompt video frames and the source video to edit the source video and output short video content that satisfies the user.

[0101] Editing operations such as deleting, replacing, and changing the color of target screen objects such as background billboards, trademarks, and advertising slogans displayed in the source video can improve the efficiency of advertising video production.

[0102] In some embodiments, the agent-based video generation method provided in this disclosure is used in digital human video production scenarios such as animation video production and game video production. By recognizing editing information such as clothing, accessories, background elements, and digital human actions represented by the input requirement text, it outputs prompt video frames that match the intent of the requirement text, thereby enabling quick and accurate editing of the source video according to the intent of the requirement text and outputting the target video.

[0103] In some embodiments, the agent-based video generation method further includes using an agent to perform the following operations: reviewing a target video based on a requirement text to obtain a review result; and performing at least one of a target task and an editing task based on defect information in the review result to obtain an updated target video.

[0104] For example, the intelligent agent reviews the target video by processing the request text and the source video. For instance, the agent reviews the video quality of the target video and the meaning of the target video in relation to the request text by invoking review rules. Figure 1 Upon approval, the system outputs the review result. If the review result indicates failure, it outputs defect description information corresponding to the review rules. This defect description information can be used as defect information in the review result to prompt the agent to re-execute at least one of the target task and editing task. This allows for timely correction of intermediate execution results such as editing instructions and prompting video frames to generate the target video, and the regeneration of a new target video, until the review result for the target video indicates approval, resulting in a target video that meets the actual requirements of the requirement text and the video quality requirements.

[0105] Figure 5 The schematic diagram illustrates the principle of the agent-based video generation method according to an embodiment of the present disclosure.

[0106] like Figure 5 As shown, the agent-based video generation method provided in this disclosure is executed collaboratively by a visual agent and a review agent. Multiple agents can collaborate to execute the method provided in this disclosure.

[0107] The requirement text and source video are input into the visual agent. The visual agent calls a large language model to process the requirement text, outputting structured editing information and region information. The visual agent can also call a large visual model to process reference video frames, structured editing information, and region information in the source video to determine the target image region in the reference video frame, and perform an image generation task to output visual cue elements that can represent the editing information. A cue video frame is generated by adding visual cue elements to the target image region of the reference video frame.

[0108] The video editing module receives a prompt video frame and the source video, and performs editing tasks on the source video based on the prompt video frame, outputting an edited target video. The review agent processes the requirement text, source video, and target video according to review rules to verify the consistency between the target video and the editing intent of the requirement text, review the video quality of the target video, and output a review result. If the review score does not meet a preset score threshold, the module transmits defect information from the review result to the visual agent, prompting the visual agent to re-execute at least one of the target task and editing task to correct the editing defects in the source video and output an updated target video. This process continues until the updated target video meets the review rule requirements, resulting in the target video provided as feedback to the user.

[0109] Figure 6 A block diagram of an agent-based video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0110] like Figure 6 As shown, the agent-based video generation device 600 includes: a first execution module 610 and a second execution module 620.

[0111] The first execution module 610 is used to perform a target task on a reference video frame in the source video according to the received demand text, and obtain a prompt video frame. The visual cue elements in the prompt video frame indicate editing information for the target screen area in the source video, and the target screen area and editing information match the intent of the demand text.

[0112] The second execution module 620 is used to perform editing tasks on the source video according to the prompt video frames to generate the target video.

[0113] According to embodiments of this disclosure, the first execution module includes: an editing instruction acquisition unit, a visual cue element acquisition unit, and a cue video frame acquisition unit.

[0114] The editing instruction obtaining unit is used to perform visual semantic understanding on the reference video frame based on the received demand text to obtain editing instructions. The editing instructions include associated region information and editing information, and the region information indicates the target screen region of the reference video frame.

[0115] The visual cue element acquisition unit is used to generate visual cue elements based on editing instructions from a large model.

[0116] The prompt video frame acquisition unit is used to update the target screen area of ​​the reference video frame based on the visual cue elements to obtain the prompt video frame.

[0117] According to embodiments of this disclosure, the visual cue element obtaining unit includes a visual cue element obtaining subunit.

[0118] The visual cue element acquisition subunit is used to perform generation tasks based on editing information and reference video frames using a large model to obtain visual cue elements.

[0119] According to embodiments of this disclosure, the visual cue element obtaining subunit is configured as follows: based on object information in the editing information as cue information, a large model is used to perform visual semantic understanding on the reference video frame, and a visual cue element is output, wherein the object information indicates the object to be edited in the reference video frame, and the element shape of the visual cue element is adapted to the outline of the object to be edited.

[0120] According to embodiments of this disclosure, the editing instruction obtaining unit includes: an editing information obtaining subunit and a region information determining subunit.

[0121] The editing information acquisition sub-unit is used to understand the intent of the requirement text and obtain structured editing information.

[0122] The region information determination subunit is used to determine region information by performing visual semantic understanding of reference video frames based on structured editing information using a large visual model.

[0123] According to embodiments of this disclosure, the agent-based video generation apparatus further includes: a first reference video frame acquisition module and a second reference video frame acquisition module.

[0124] The first reference video frame acquisition module is used to determine the tool execution parameters based on structured editing information, and to use the target tool to determine the reference video frame from the source video frame based on the tool execution parameters.

[0125] The second reference video frame acquisition module is used to process at least one source video frame based on structured editing information using a large model to obtain a reference video frame in the source video that matches the intent.

[0126] According to embodiments of this disclosure, the element attributes of the visual cue element are used to indicate the execution parameters of the editing task, and the execution parameters include at least one of the following: the operation type of the editing operation, the texture attribute parameters related to the editing task, and the movement direction of the object to be edited in the target video.

[0127] According to embodiments of this disclosure, the visual cue element includes an object trajectory element that indicates the movement trajectory of a target image object within the frame of the source video.

[0128] The second execution module includes a movement parameter acquisition unit and a first generation unit.

[0129] The motion parameter acquisition unit is used to take the object trajectory element and the background content in the prompt video frame relative to the object trajectory element as prompt information, and use the large model to perform visual semantic understanding on the source video frame to output motion parameters.

[0130] The first generation unit is used to edit the source video frames using motion parameters to generate the target video.

[0131] According to embodiments of this disclosure, the visual cue element includes an action element, the display position of which in the cue video frame indicates the object part of the target screen object performing the target action.

[0132] The second execution module includes an action parameter acquisition unit and a second generation unit.

[0133] The motion parameter acquisition unit is used to take the background content of the motion element and the prompt video frame relative to the motion element as prompt information, and use the large model to perform visual semantic understanding on the source video frame to output motion parameters.

[0134] The second generation unit is used to edit the source video frames using motion parameters to generate the target video.

[0135] According to embodiments of this disclosure, the visual cue element includes multiple action elements corresponding to multiple target actions; the target video is used to display at least one object part of the target screen object, and the multiple target actions are executed in an execution order.

[0136] According to embodiments of this disclosure, the visual cue element includes multiple action elements corresponding to multiple target actions, and the action attributes in the editing information are used to indicate the execution order of the target actions corresponding to the action elements.

[0137] According to embodiments of this disclosure, the action parameter acquisition unit includes an action parameter acquisition subunit.

[0138] The motion parameter acquisition subunit is used to take the motion elements, background content and motion attributes in the prompt video frame as prompt information, use the large model to perform visual semantic understanding on the source video frame in the source video, and output multiple motion parameters. The execution order attribute of the multiple motion parameters matches the execution order of multiple target actions. Among them, the target video is used to display at least one object part of the target screen object, and execute multiple target actions according to multiple execution orders.

[0139] According to embodiments of this disclosure, the second execution module includes an execution unit.

[0140] The execution unit is used to perform editing tasks on the source video based on the prompt video frames and editing information using a large model.

[0141] According to embodiments of this disclosure, the agent-based video generation apparatus further includes: an auditing module and an updated target video acquisition module.

[0142] The review module is used to review target videos based on the required text and obtain the review results.

[0143] The updated target video acquisition module is used to perform at least one of the target task and editing task based on the defect information in the review results to obtain the updated target video.

[0144] It should be noted that in the embodiments of this disclosure, the modules, units, and sub-units of the video generation device based on intelligent agents can all be executed by intelligent agents. For example, the modules, units, and sub-units in the video generation device based on intelligent agents can be set by a server that has deployed intelligent agents.

[0145] Figure 7 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0146] In embodiments of this disclosure, such as Figure 7 As shown, the AI ​​agent 700 may include an input module 710, a processing module 720, and an output module 730.

[0147] Input module 710 is used to receive input information;

[0148] The processing module 720 is used to determine the target task based on the input information received by the input module, determine the large model based on the target task, and execute the agent-based video generation method provided in the embodiments of this disclosure by calling the large language model to obtain output information;

[0149] Output module 730 is used to output the output information obtained by the processing module.

[0150] According to embodiments of this disclosure, the input module 710 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 700 can understand and process. The input module 710 is the primary link for the AI ​​agent 700 to interact with the outside world, enabling the AI ​​agent 700 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0151] In the example, input module 710 can input the requirement text, source video, reference video frame, etc. described above.

[0152] In the example, processing module 720 is the core support for the AI ​​agent 700's ability to handle complex tasks. Processing module 720 can execute the agent-based video generation method described above.

[0153] In the example, the performance of the processing module 720 is closely related to the large model on which the AI ​​agent 700 is based. To fully leverage the capabilities of the large model, the internal structure of the processing module 720 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0154] In the example, after the AI ​​agent 700 obtains the requirement text, the processing module 720 can use a large visual model to process the requirement text and reference video frames to obtain a prompt video frame, use the large visual model to process the prompt video frame and the source video to obtain the target video, and then pass the target video to the output module 730.

[0155] Understandably, while large language models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. Once the AI ​​agent 700 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to create weather forecasts.

[0156] In the example, output module 730 can output the target video described above.

[0157] The AI ​​agent 700 according to embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.

[0158] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0159] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0160] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0161] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0162] Figure 8A schematic block diagram of an example electronic device that can be used to implement the agent-based video generation method of embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0163] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0164] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0165] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as an agent-based video generation method. For example, in some embodiments, the agent-based video generation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the agent-based video generation method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the agent-based video generation method by any other suitable means (e.g., by means of firmware).

[0166] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0168] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0170] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0171] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0172] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0173] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method based on intelligent agents, comprising: The following operations are performed using an intelligent agent: Based on the received request text, the target task is performed on the reference video frame in the source video to obtain the prompt video frame, wherein the visual cue element in the prompt video frame indicates the editing information for the target screen area in the source video, and the target screen area and the editing information match the intent of the request text; Based on the prompted video frames, the source video is edited to generate the target video.

2. The method according to claim 1, wherein, The step of performing the target task on the reference video frames in the source video according to the received demand text includes: Based on the received demand text, the reference video frame is visually semantically understood to obtain editing instructions. The editing instructions include associated region information and editing information, and the region information indicates the target screen region of the reference video frame. The visual cue elements are generated using a large model based on the editing instructions; and The target screen area of ​​the reference video frame is updated based on the visual cue elements to obtain the cue video frame.

3. The method according to claim 2, wherein, The process of generating the visual cue elements using a large model based on the editing instructions includes: Using the large model, a generation task is performed based on the editing information and the reference video frame to obtain the visual cue elements.

4. The method according to claim 3, wherein, The step of utilizing the large model to perform a generation task based on the editing information and the reference video frames includes: Using the object information in the editing information as prompt information, the large model is used to perform visual semantic understanding on the reference video frame, and the visual prompt element is output. The object information indicates the object to be edited in the reference video frame, and the shape of the visual prompt element is adapted to the outline of the object to be edited.

5. The method according to claim 2, wherein, The step of performing visual semantic understanding on the reference video frame based on the received demand text to obtain editing instructions includes: The intent of the request text is understood to obtain structured editing information; The visual big data model is used to perform visual semantic understanding on the reference video frame based on the structured editing information to determine the region information.

6. The method of claim 5, further comprising using the agent to perform the following operations: Based on the structured editing information, tool execution parameters are determined, and the target tool is used to determine the reference video frame from the source video frames based on the tool execution parameters; or Using a large model, at least one of the source video frames is processed based on the structured editing information to obtain a reference video frame in the source video that matches the intent.

7. The method according to claim 1, wherein, The element attributes of the visual cue element are used to indicate the execution parameters of the editing task, and the execution parameters include at least one of the following: The operation type of the editing operation, the texture attribute parameters related to the editing task, and the direction of movement of the object to be edited in the target video.

8. The method according to claim 1, wherein, The visual cue elements include object trajectory elements, which indicate the movement trajectory of the target image object within the frame of the source video. The step of performing editing tasks on the source video based on the prompted video frames includes: The object trajectory element and the background content in the prompt video frame relative to the object trajectory element are used as prompt information. The large model is used to perform visual semantic understanding on the source video frame and output the motion parameters. as well as The source video frame is edited using the motion parameters to generate the target video.

9. The method according to claim 1, wherein, The visual cue element includes an action element, and the display position of the action element in the cue video frame indicates the part of the target image object that performs the target action. The step of performing editing tasks on the source video based on the prompt video frame includes: Using the action element and the background content relative to the action element in the prompt video frame as prompt information, a large model is used to perform visual semantic understanding on the source video frame, and action parameters are output; and The source video frame is edited using the motion parameters to generate the target video.

10. The method according to claim 9, wherein, The visual cue elements include multiple action elements corresponding to multiple target actions; The target video is used to display at least one part of the target image object, and to execute the multiple target actions in an execution order.

11. The method according to claim 9, wherein, The visual cue elements include multiple action elements corresponding to multiple target actions, and the action attributes in the editing information are used to indicate the execution order of the target actions corresponding to the action elements; Using the action element and the background content in the prompt video frame relative to the action element as prompt information, a large model is used to perform visual semantic understanding on the source video frames in the source video, including: The action elements in the prompt video frame, the background content, and the action attributes are used as prompt information. A large model is used to perform visual semantic understanding on the source video frames in the source video, and multiple action parameters are output. The execution order attribute of the multiple action parameters is matched with the execution order of the multiple target actions. The target video is used to display at least one part of the target image object, and the multiple target actions are executed in a multiple execution order.

12. The method according to any one of claims 1 to 9, wherein, Perform editing tasks on the source video according to the prompted video frames, including: The large model is used to perform the editing task on the source video based on the prompt video frames and the editing information.

13. The method according to claim 1, wherein, The method further includes using the agent to perform the following operations: The target video is reviewed based on the stated requirements text, and the review result is obtained; and Based on the defect information in the audit results, perform at least one of the target task and the editing task to obtain an updated target video.

14. A video generation device based on intelligent agents, comprising: The first execution module is used to perform a target task on a reference video frame in the source video according to the received demand text to obtain a prompt video frame, wherein the visual cue elements in the prompt video frame indicate editing information for a target screen area in the source video, and the target screen area and the editing information match the intent of the demand text; The second execution module is used to perform editing tasks on the source video according to the prompt video frame to generate the target video.

15. An artificial intelligence agent configured to perform the method of any one of claims 1 to 13.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 13.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images