A video nonlinear editing method and system based on multi-agent self-supervised learning

Abstract

This invention proposes a nonlinear video editing method and system based on multi-agent self-supervised learning, belonging to the fields of computer vision and artificial intelligence. The method includes: constructing an editing context encompassing scripts, source materials, and timeline states; deploying editing agents to generate multiple candidate video timelines; evaluating and iteratively optimizing the timelines from semantic, logical, and rhythmic perspectives through review agents; introducing a multimodal evaluation model to score and rank the candidate timelines based on quality; and constructing a self-supervised learning mechanism using the evaluation results and editing data to continuously update the multi-agent strategy and output the final video. This invention achieves high-quality automatic editing of long, multi-source videos through multi-agent collaboration and self-supervised learning, applicable to scenarios such as short video production, live stream replay summaries, course editing, and promotional video generation, significantly reducing human intervention and improving editing efficiency and quality stability.

Description

Technical Field

[0001] This invention relates to a video nonlinear editing method and system based on multi-agent self-supervised learning, belonging to the interdisciplinary technical fields of computer vision, multimedia content analysis and artificial intelligence large model applications. Background Technology

[0002] Traditional video editing typically involves professional editors using non-linear editing software. Editors spend a significant amount of time selecting and annotating numerous raw clips, repeatedly importing and replacing footage, cropping, previewing, and constructing a timeline that meets the script's requirements. They then combine transitions, subtitles, and sound effects to complete the final product. This video production process heavily relies on human experience and subjective aesthetics, is cumbersome and time-consuming, and its quality is difficult to guarantee, making it unsuitable for the rapid production demands of today's short videos and online videos.

[0003] With the development of computer vision and multimedia intelligent analysis technologies, methods for automated video editing have emerged. These methods process original footage through shot segmentation, face detection, scene recognition, and rhythm analysis, splicing video segments according to preset scripts or rules to generate simple videos. While these script-driven or rule-driven video editing methods are relatively easy to implement, they have limited capabilities in controlling complex narrative structures, semantic relationships between multiple sources, and personalized styles, making it difficult to meet the diverse needs of video creation.

[0004] To enhance the intelligence of video editing, deep learning models are being introduced into video production. These models jointly model video frames, audio signals, and accompanying text, using semantic feature vectors to retrieve and match video materials, and automatically selecting video segments based on input themes, keywords, or script requirements. Some methods integrate speech recognition and automatic subtitle modules to form a processing flow from material selection to video editing output. However, these methods typically use a single model for material selection. Video editing is essentially a mapping from text to a set of video segments, lacking detailed modeling of the timeline structure.

[0005] To address the aforementioned issues, there is an urgent need to develop a method for non-linear video editing processes. Based on a clear understanding of the modeling timeline and material environment, this method utilizes a multi-agent structure with role division and collaboration capabilities. Different agents can interact through natural language or structured information. Under a self-supervised learning mechanism, the video editing strategy can be improved, and combined with a multimodal large model, automated quality evaluation of the generated video can be achieved. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a video nonlinear editing method based on multi-agent self-supervised learning. This method collaboratively models the agent responsible for performing editing operations with the agent responsible for evaluation and feedback. Through multiple rounds of interaction and self-supervised exploration, it realizes the automatic construction and optimization of the video timeline, reduces human intervention, and improves the semantic relevance, structural rationality and overall visual appeal of the video generation results. It also improves the performance of automatic video editing in terms of timeline construction, semantic understanding and editing quality control.

[0007] To solve the above technical problems, the present invention adopts the following technical solution: This invention proposes a nonlinear video editing method based on multi-agent self-supervised learning, which mainly includes the following steps: Step 1: Formalize the editing environment as a triple ,in, This refers to the semantic representation of the plot script and narration text in the edited video. Indicates the current editing round. This represents a set of candidate video clips. Indicates in The timeline status of each editing round; the above editing environment is the foundation of multi-agent video editing; Step 2: Based on multi-agent collaborative generation and optimization of the video timeline, two types of video editing agents with different responsibilities are set up in the editing context: an editing agent that proposes specific editing operations and a reviewing agent that evaluates the current timeline and provides modification suggestions. Through multiple rounds of interaction between the two agents, the selection and arrangement of video segments in the timeline are continuously adjusted. The update of the timeline state can be represented as follows: ,in In the round The selected editing operations are used until a video editing solution that meets the preset goals is obtained; Step 3: Optimize the multi-agent collaborative strategy using a self-supervised learning mechanism. In the absence of large-scale manually labeled samples, the multi-agent intelligently explores the editing environment and collects interactions including contextual states, editing operation sequences and corresponding evaluation results. Based on this data, a self-supervised learning model is constructed to update the internal strategies and interaction rules of each agent, thereby gradually improving editing quality and collaborative efficiency during the continuous editing process. Step 4: Evaluate and filter candidate timelines based on a multimodal evaluation model, converting the candidate timeline videos generated by the multi-agent system into multimodal input; Let the timeline... The corresponding keyframe set is The corresponding text summary is Multimodal evaluation model The quality assessment is expressed as: ,in, To evaluate the model parameters, For timeline The algorithm calculates the overall quality score and selects the video with the highest score from multiple candidate timeline videos. It then feeds the evaluation information back to the self-supervised learning module to iteratively update the multi-agent system and further improve the editing strategy.

[0008] Furthermore, the specific steps of step 1 are as follows: Step 1.1: Unify the encoding of multi-dimensional attributes of video materials, analyze the video script, extract semantic elements such as video theme, scene, characters, and time, perform frame-level or clip-level analysis on candidate video materials, and provide attributes such as video content, action, shot type, and rhythm, and then classify the video clips. The eigenvectors are represented as ,in This is a video feature extraction function that obtains the attribute codes of the candidate video set. ; Step 1.2: Structure the sequential relationship, start and end times, overlaps and subtitle text of each segment along the video playback timeline, so that the video editing agent can clearly understand the connection between the video clips used and the deviation from the target video duration or rhythm; Step 1.3: Combine the video generation requirements, material attributes, and playback timeline status into a data structure that the agent can read. ,in For constructor, Video data, represented in vector, matrix, list, or other structured form, is used to drive subsequent decisions by multiple agents in the video generation process.

[0009] Furthermore, the specific steps of step 2 are as follows: Step 2.1: Based on the current editing context, the editing agent outputs suggested operations for the playback timeline. These operations include adding, deleting, moving, replacing, cropping, and transitioning video clips, and can plan the operation sequence in combination with constraints such as the target video length and rhythm. Step 2.2: The review agent obtains the current video playback timeline and the video script of the editing target, comprehensively evaluates the matching degree between the timeline and the script, the naturalness of the scene transitions, and the overall rhythm, and provides feedback in structured or natural language form, pointing out the segments that need improvement and how to edit them; Step 2.3: Based on the feedback from the review agent, call the editing agent to perform a new round of editing and adjustment on the video playback timeline, forming a closed loop of editing modifications, evaluation suggestions, and adjustment operations; the termination conditions include the evaluation score threshold, target video length, and upper limit of the number of editing rounds, etc. The agent interaction decides whether to stop editing based on the set termination conditions.

[0010] Furthermore, the specific steps of step 3 are as follows: Step 3.1: The multi-agent system learns editing strategies by training on various types of video footage, recording structured data of the decision-making process, editing operation sequences, and evaluation scores to form strategies that can be used to guide editing training; Step 3.2: Based on the existing evaluation scores and corresponding vector videos during training, set up different operation sequences to automatically generate quality strategies; Step 3.3: Extract representative triples {initial timeline, modification suggestion, modified timeline} from the training process and use them as training samples in the agent's decision-making process to update the editing agent and the reviewing agent. This allows the method of the present invention to reuse previous video generation experience when generating videos, thereby improving editing efficiency and quality stability.

[0011] Furthermore, the specific content of step 4 may include, but is not limited to: Step 4.1: Based on the position and duration of the video segments on the playback timeline, select multiple representative keyframes and combine them with the corresponding script segments and narration text to form multimodal samples for evaluating the model; Step 4.2: Input the above multimodal samples into the pre-trained evaluation model. This model has the ability to jointly understand visual and textual content. It can give quality scores and preference rankings for candidate timelines based on preset evaluation indicators and determine whether the edited and generated video meets the standards. Step 4.3: Select the timeline with the highest score or the best ranking as the output result, and feed the evaluation result back to the multi-agent to adjust the loss function and sample weights in the self-supervised learning stage, so that the method can continuously improve the video editing level.

[0012] On the other hand, this invention provides a video nonlinear editing system based on multi-agent self-supervised learning, including: The editing context construction unit is configured to: align and preprocess scripts, narration texts and candidate video clips to generate structured input containing timestamps, semantic tags and clip attributes, and model the current timeline state based on this input to obtain the editing context; The candidate timeline generation unit is configured to: deploy an editing agent in the editing context, generate a set of editing operations including segment selection, order adjustment, duration trimming and transition insertion according to the context, and construct multiple candidate video editing timelines; The review and optimization unit is configured to: deploy a review agent in the editing context, evaluate candidate timelines in terms of semantic integrity, logical coherence and rhythmic smoothness, provide modification suggestions and call the editing agent to iteratively optimize the timelines; The multimodal evaluation and screening unit is configured to: introduce a multimodal evaluation model, jointly evaluate keyframes and their corresponding text summaries in the candidate timeline, output quality scores and ranking results, and screen out the target video timeline; The self-supervised learning and output unit is configured to: construct self-supervised training based on structured data from multimodal evaluation results and the editing process, update multi-agent policies, and output the final video corresponding to the target video timeline.

[0013] Meanwhile, the present invention also provides an electronic system, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, characterized in that the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method proposed in the present invention.

[0014] Finally, the present invention also proposes a computer-readable storage medium storing computer instructions for causing the computer to perform the steps of the method proposed in the present invention.

[0015] The present invention adopts the above technical solution and has the following technical effects compared with the prior art: This invention utilizes a self-supervised learning mechanism to use multimodal evaluation results as training data for updating multi-agent strategies. It can continuously accumulate data and preference samples from the editing process without large-scale manual annotation, thereby reducing human intervention while improving the stability of the final video in terms of semantic relevance, structural rationality, rhythmic coherence, and overall visual appeal. This method can be integrated with existing nonlinear editing software or cloud-based editing services, supporting various application scenarios such as short video production, live replay summarization, online course editing, and promotional video production, demonstrating good scalability and application value. Attached Figure Description

[0016] Figure 1 This is a multi-agent nonlinear editing framework diagram.

[0017] Figure 2 This is a flowchart of the two-stage context learning process for multi-agent communication. Detailed Implementation

[0018] This invention proposes a video nonlinear editing method based on multi-agent self-supervised learning. The following detailed description of the invention is provided in conjunction with specific implementation steps.

[0019] Example 1: This example provides a video nonlinear editing method based on multi-agent self-supervised learning, including the following steps: Collect the script, narration text, and candidate video clip materials of the video to be edited, align and preprocess them to form a structured input containing timestamps, semantic tags, and clip material attributes. Based on the structured input, construct non-linear video editing, model the current timeline state, and obtain the editing context for multi-agent decision-making. In the editing context, an editing agent is deployed to generate a set of selectable editing operations, including segment selection, order adjustment, duration trimming, and transition insertion, based on the editing context, and to construct multiple candidate video editing timelines. In the editing context, a review agent is deployed to evaluate candidate timelines in terms of semantic integrity, logical coherence, and rhythmic smoothness, provide modification suggestions, and call the editing agent to iteratively optimize the timelines; A multimodal evaluation model is introduced to jointly evaluate keyframes and their corresponding text summaries in candidate timelines, output quality scores and ranking results, and select timelines with high ranking scores as target video timelines. The multimodal evaluation results and structured data from the editing process are used to construct a self-supervised training method, update the multi-agent policy, and output the final video corresponding to the target video timeline.

[0020] Furthermore, as a further specific implementation of this embodiment, it includes: 1. Video generation context construction and initialization: The input consists of a video script and user requirements as raw text, which are then processed using a text encoding function to obtain a semantic vector. , It includes information such as project theme, setting, characters, and timeline. The collection of candidate video editing clips is denoted as... ,in, For the first Each video segment is processed using a video feature extraction function to calculate its features, thus obtaining a set of video materials. Based on the above elements, video generation is formalized into triples. This provides unified and structured contextual information for subsequent multi-agent decision-making.

[0021] 2. Multi-agent collaborative generation of candidate playback timelines: In video generation, two types of agents are introduced: editing agents that perform specific editing operations, and reviewing agents that evaluate the timeline and provide modification suggestions. In the... In the editing process, the editing agent first generates operations based on the current environment and feedback from the previous round. ,in It is the policy function adopted by the editing agent. yes The system provides round-robin feedback, allowing users to review and edit video clips, including adding, deleting, moving, replacing, and cropping them. The timeline updates based on these actions. The agent reviews the updated environment and provides feedback for this round. ,in The policy function used by the reviewing agents. Multiple agents iterate in this manner, ending the editing process and obtaining candidate video playback timelines when a termination condition is met. Repeating this process yields multiple candidate video playback timelines.

[0022] 3. Multimodal quality assessment and candidate solution screening: To automatically select higher-quality edits, a multimodal evaluation model is introduced to score and filter candidate timelines generated by multiple agents. For any candidate timeline... Based on the position and duration of each segment on the timeline, several representative frames are selected to form a set of key images. Simultaneously, text summaries were extracted from the video script, narration text, and subtitles corresponding to the timeline. Feature vectors of key images are obtained using visual feature extraction functions and text feature extraction functions, respectively. With evaluation text features The multimodal evaluation model is denoted as... Its quality assessment of the timeline can be formalized as ,in, To evaluate the model parameters, a score is calculated for each set of candidate timelines generated by the multi-agent system. The system then sorts and selects the best editing scheme for the current video editing process. Simultaneously, the overall quality score of this set of candidate timelines is used for subsequent self-supervised learning.

[0023] 4. Self-supervised interactive data recording and sample construction: Record and save the editing behavior of multi-agents in different types of videos to build the data needed for self-supervised learning.

[0024] For each editing task, the multiple rounds of interaction from the initial timeline to the final timeline form a sequence. ,in, This could be a score calculated by the evaluation model for the current timeline. Data from all tasks is aggregated to form an interactive dataset, including multiple candidate timelines for video editing. and If its score meets Then the preference pair is automatically constructed. And based on this preference relationship, a ranking loss is defined. This is used to guide the model to favor higher-quality editing schemes in subsequent policy updates. Simultaneously, it automatically selects significantly improved video examples from the dataset to help the agent learn effective editing and modification patterns.

[0025] 5. Self-supervised multi-agent policy update: Using the dataset and quality assessment constructed in step 4, the policy configuration parameters of the editing and reviewing agents are updated, enabling the multi-agent system to continuously improve editing quality and collaborative efficiency during ongoing operation. For the editing agent, a quality-weighted loss function is constructed to increase the contribution of high-quality samples during training, gradually leading the editing agent to produce higher-scoring editing operations. For the reviewing agent, its scoring output is aligned with the final quality, generating instructive feedback that is closer to the target video quality and more guiding for the editing agent.

[0026] Example 2: This example provides a video nonlinear editing system based on multi-agent self-supervised learning, including... The editing context construction unit is configured to: align and preprocess scripts, narration texts and candidate video clips to generate structured input containing timestamps, semantic tags and clip attributes, and model the current timeline state based on this input to obtain the editing context; The candidate timeline generation unit is configured to: deploy an editing agent in the editing context, generate a set of editing operations including segment selection, order adjustment, duration trimming and transition insertion according to the context, and construct multiple candidate video editing timelines; The review and optimization unit is configured to: deploy a review agent in the editing context, evaluate candidate timelines in terms of semantic integrity, logical coherence and rhythmic smoothness, provide modification suggestions and call the editing agent to iteratively optimize the timelines; The multimodal evaluation and screening unit is configured to: introduce a multimodal evaluation model, jointly evaluate keyframes and their corresponding text summaries in the candidate timeline, output quality scores and ranking results, and screen out the target video timeline; The self-supervised learning and output unit is configured to: construct self-supervised training based on structured data from multimodal evaluation results and the editing process, update multi-agent policies, and output the final video corresponding to the target video timeline.

[0027] Example 3: According to another aspect of the present invention, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor, and the processor loads and executes the computer program using the method proposed in this invention.

[0028] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0029] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual use, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0030] Furthermore, the memory can be an internal storage unit of the terminal device, such as the hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. In addition, the memory can also be a combination of the internal storage unit and the external storage device of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0031] Furthermore, any one of the methods described in the above embodiments can be stored in the memory of the terminal device and loaded and executed on the processor of the terminal device for convenient use.

[0032] Example 4: This application also discloses a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, it employs any of the methods described in the above embodiments.

[0033] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0034] In summary, this invention proposes a video nonlinear editing method based on multi-agent self-supervised learning, realizing a closed loop from editing environment construction, multi-agent collaborative generation of candidate timelines, multimodal automatic evaluation, and self-supervised strategy updating, automatically completing video nonlinear editing with reduced human intervention. Those skilled in the art should understand that the above embodiments do not limit the scope of protection of this invention in any way, and all technical solutions obtained by equivalent substitution or other means fall within the scope of protection of this invention. Parts not covered in this invention are the same as or can be implemented using existing technologies.

Claims

1. A video nonlinear editing method based on multi-agent self-supervised learning, characterized in that, Including the following steps: S1. Constructing the editing context: Based on the script, narration text and candidate video clips, alignment and preprocessing are performed to generate a structured input containing timestamps, semantic tags and clip material attributes. Based on this input, the current timeline state is modeled to obtain the editing context. S2. Generate candidate timelines: Deploy an editing agent in the editing context to generate a set of editing operations including segment selection, order adjustment, duration trimming, and transition insertion based on the context, and construct multiple candidate video editing timelines; S3. Review and Optimization: Deploy a review agent in the editing context to evaluate candidate timelines in terms of semantic integrity, logical coherence, and rhythmic smoothness, provide modification suggestions, and call the editing agent to iteratively optimize the timelines; S4. Multimodal evaluation and screening: A multimodal evaluation model is introduced to jointly evaluate the key frames and their corresponding text summaries in the candidate timelines, output quality scores and ranking results, and screen out the target video timelines; S5. Self-supervised learning and output: Self-supervised training is constructed based on the multimodal evaluation results and structured data of the editing process to update the multi-agent policy and output the finished video corresponding to the target video timeline.

2. The method according to claim 1, characterized in that, In step S1, the editing environment is formalized into triples. ,in, This refers to the semantic representation of the plot script and narration text in the edited video. Indicates the current editing round. This represents a set of candidate video clips. Indicates in The timeline status of each editing round.

3. The method according to claim 1, characterized in that, The specific process of constructing the editing context in step S1 includes: (1.1) Perform frame-level or clip-level analysis on candidate video materials to extract video content, action, shot type and rhythm attributes, and encode them into feature vectors; (1.2) Structured representation of the sequential relationship, start and end times and overlap information of video segments in the timeline; (1.3) Combine video generation requirements, material attributes and timeline status into a data structure that can be read by the agent.

4. The method according to claim 3, characterized in that, video clips The eigenvectors are represented as: ,in This is a video feature extraction function that obtains the attribute codes of the candidate video set. ; Data structures that can be read by intelligent agents ,in For constructor, Video data, represented in vector, matrix, list, or other structured form, is used to drive subsequent decisions by multiple agents in the video generation process.

5. The method according to claim 1, characterized in that, In step S3, the editing agent and the reviewing agent update the timeline state through multiple rounds of interaction until a preset termination condition is met, as follows: Step S301: Based on the current editing context, the editing agent outputs suggested operations for the playback timeline, including adding, deleting, moving, replacing, cropping, and transitioning video clips, and can plan the operation sequence in combination with the target video duration and rhythm constraints; Step S302: The review agent obtains the current video playback timeline and the video script of the editing target, comprehensively evaluates the matching degree between the timeline and the script, the naturalness of the scene transitions and the overall rhythm, and provides feedback in structured or natural language form, pointing out the segments that need improvement and editing strategies. Step S303: Based on the feedback from the review agent, the editing agent is invoked to perform a new round of editing and adjustment on the video playback timeline, forming a closed loop of editing, evaluation suggestions, and adjustment operations. The agent interaction determines whether to stop editing based on the set termination conditions, including: the evaluation score reaches the threshold, the video length meets the requirements, or the number of editing rounds reaches the upper limit.

6. The method according to claim 1, characterized in that, In step S4, the multimodal evaluation and screening are specifically as follows: The candidate timelines are evaluated and screened based on a multimodal evaluation model, and the candidate timeline videos generated by the multi-agent are converted into multimodal inputs. Set timeline The corresponding keyframe set is The corresponding text summary is Multimodal evaluation model The quality assessment is expressed as: ,in, To evaluate the model parameters, For timeline The overall quality score is used to select the video with the highest score from multiple candidate timeline videos.

7. The method according to claim 1, characterized in that, The specific process of updating the multi-agent policy in step S5 is as follows: Step 501: The multi-agent system learns editing strategies by training on various types of video footage, records structured data of the decision-making process, editing operation sequences and evaluation scores, and forms strategies to guide editing training; Step 502: Based on the evaluation scores and corresponding vector videos already obtained during training, set up different operation sequences to automatically generate quality strategies; Step 503: Extract representative triples {initial timeline, suggested modification, modified timeline} from the training process and use them as training samples in the agent's decision-making process to update the editing agent and the reviewing agent.

8. A video nonlinear editing system based on multi-agent self-supervised learning, characterized in that, include The editing context construction unit is configured to: align and preprocess scripts, narration texts and candidate video clips to generate structured input containing timestamps, semantic tags and clip attributes, and model the current timeline state based on this input to obtain the editing context; The candidate timeline generation unit is configured to: deploy an editing agent in the editing context, generate a set of editing operations including segment selection, order adjustment, duration trimming and transition insertion according to the context, and construct multiple candidate video editing timelines; The review and optimization unit is configured to: deploy a review agent in the editing context, evaluate candidate timelines in terms of semantic integrity, logical coherence and rhythmic smoothness, provide modification suggestions and call the editing agent to iteratively optimize the timelines; The multimodal evaluation and screening unit is configured to: introduce a multimodal evaluation model, jointly evaluate keyframes and their corresponding text summaries in the candidate timeline, output quality scores and ranking results, and screen out the target video timeline; The self-supervised learning and output unit is configured to: construct self-supervised training based on structured data from multimodal evaluation results and the editing process, update multi-agent policies, and output the final video corresponding to the target video timeline.

9. An electronic system comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, characterized in that the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the steps of the method according to any one of claims 1-7.

Images