Method and apparatus for constructing video clip agent training samples

Abstract

The present disclosure relates to the technical field of video processing and machine learning, and discloses a method and device for constructing a video clip agent training sample. The method comprises: inversely mapping a native clip project to an executable clip action sequence, recording a conversion loss, and obtaining a gold clip action sequence through forward reconstruction, normalized comparison and quality access control; constructing a causal next action supervision sample based on the gold clip action sequence, wherein the action space comprises a clip action, a recall action carrying a search query and a termination action; a recall result is injected into a next state, and a termination action is added to a complete qualified track; a supervised fine-tuning model is run in a clip sandbox to collect reinforcement learning tracks, and rewards are calculated according to a recall hit, actual use, recall cost, structural legality and film quality. The present disclosure can construct a video clip agent training sample that is replayable, causally non-leaky and covers tool calls and task termination.

Description

Technical Field

[0001] This disclosure relates to the fields of video processing and machine learning technologies, and in particular to techniques for constructing supervised fine-tuning samples and reinforcement learning trajectory samples for training video editing agents. Background Technology

[0002] The descriptions in this section are intended to provide background information for the implementation of this disclosure and should not be construed as an admission or implication that they constitute prior art.

[0003] In recent years, training models capable of automatically generating video editing operations has become an important research direction in the field of intelligent editing. However, model training relies on a large number of high-quality training samples. In existing technologies, video editing results are usually saved in the form of native editing projects. These projects contain a large number of objects such as footage, tracks, clips, effects, transitions, and keyframes. These objects reference each other through identifiers, forming a complex object graph. If native editing projects are directly used as training data, the model struggles to learn stable editing operation rules from the complex object graph. If editing projects are manually compiled into training samples, the annotation cost is high and the efficiency is low, making it difficult to meet the needs of large-scale training. Furthermore, while parsing editing project files can extract some editing information, information loss and semantic bias may occur during the parsing process, and there is a lack of effective means to verify the extraction results, making it difficult to guarantee the quality of training samples. Summary of the Invention

[0004] This disclosure provides a method and apparatus for constructing training samples for a video editing agent. One technical problem it can solve is how to automatically obtain a high-fidelity and executable sequence of editing actions from a native editing project, and further construct supervised fine-tuning samples and reinforcement learning trajectory samples that enable the video editing agent to learn when to perform editing, when to retrieve footage, and when to terminate the task, while suppressing training bias caused by future state leakage, invalid recall, and cross-dataset distribution of homologous samples.

[0005] According to one aspect of this disclosure, a method for constructing training samples for a video editing agent is provided, comprising: The original editing project is parsed to obtain the project object reference diagram, which includes material objects, track objects, clip objects and auxiliary effect objects, as well as the hierarchical inclusion relationship and reference relationship between each object; A global object index is established for each object in the project object reference diagram, with the object identifier as the key; Based on the global object index, time semantics, track semantics, and resource semantics are extracted from the project object reference graph. Supported objects are mapped to editing actions to obtain an executable editing action sequence. Conversion loss records are generated for unsupported objects, unknown tracks, and unresolved references. Each editing action includes an action type, track information, time information, resource information, and action parameters. The time information includes at least one of the following based on the action type: source material cut-out interval, target time interval, or action time. For editing actions on media that can be cut, the time information includes both the source material cut-out interval and the target time interval. The editing action sequence is forward-converted into a reconstructed editing project; The original editing project and the reconstructed editing project are normalized to obtain feature sequences after removing non-semantic fields. The feature sequences of the original editing project and the reconstructed editing project are matched and semantically compared to obtain quality indicators. Based on the quality index and the conversion loss record, a quality gate is determined, and the editing action sequences that fail the quality gate are isolated. The editing action sequences that pass the quality gate are determined as gold editing action sequences, and training samples for training the video editing agent are constructed based on the gold editing action sequences.

[0006] The video editing task is modeled as a next action prediction task. The action space of the next action includes editing actions, recall actions, and termination actions. Editing actions correspond to atomic editing operations or action groups with atomic submission semantics in the editing intermediate representation language; recall actions include recall flags and recall queries, which are used to trigger material retrieval and add the retrieval results to the editing context of the next state; termination actions indicate that the current editing state has passed structural verification, reference closure verification, and renderability verification.

[0007] Based on the golden clip action sequence, recall and termination actions are inserted, and supervised fine-tuning samples are constructed using causal position sampling. This ensures that the model input only includes the task, state, history, and clip context available before the target action, and the next action at the target position is used as the supervision label.

[0008] In a preferred embodiment, a supervised fine-tuned video editing agent generates state-action trajectories in an editing sandbox, calculates rewards based on recall hits and subsequent usage, recall costs, action legality, and final output quality, and stores the state, action, reward, next state, and termination flag as reinforcement learning trajectory samples.

[0009] In a preferred embodiment, parsing the native editing project includes: when the root node of the native editing project is a container project, selecting an actual project from the projects contained in the container project for parsing based on the number of valid tracks, the number of audio and video materials, and the valid duration; The extraction of time semantics, track semantics, and resource semantics includes: uniformly converting the time amount in the reference diagram of the project object into a preset time unit, distinguishing the source material extraction interval and the target time interval; identifying the main track and overlay track according to the preset track recognition rules, and determining the track information and layer order of each editing action.

[0010] In a preferred embodiment, the method further includes: Replace the replaceable media resource information and text content in the editing action sequence with placeholders to form replaceable input slots; A protection tag is set for objects that are inherent resources of the template in the native editing project. Resources with the protection tag are not replaced with placeholders.

[0011] In a preferred embodiment, the method further includes: performing sequential normalization on the editing action sequence before forward converting the editing action sequence into a reconstructed editing project; The sequence normalization includes: sorting the editing actions within the same track according to the start time of the target time interval; arranging the editing actions that generate the basic material clips before the effect actions and keyframe actions that depend on the basic material clips, to obtain a replayable editing action sequence.

[0012] In a preferred embodiment, the normalization includes: projecting the native editing project and the reconstructed editing project to the same supported semantic subspace; removing the object identifier field, the generation time field, and the machine-dependent absolute path field; generating stable placeholders for similar resources based on at least one of content hash, perceptual hash, input slot identifier, or first-use semantic key; quantizing the time quantity to a preset precision; and performing stable sorting of the field keys of the objects. The feature sequence includes track type, track level, segment start point, segment duration, source material extraction range, resource type, and supported effect parameters; The matching and semantic comparison include: performing a minimum cost one-to-one matching on the segments in the feature sequences of the original editing project and the reconstructed editing project, allowing some segments to be unmatched; applying a preset cost to the unmatched segments; and jointly calculating the segment coverage, normalized time error, and attribute consistency rate based on the matched and unmatched segments.

[0013] In a preferred embodiment, the quality metric is calculated by weighting at least two of the following: reverse skip rate, unresolved reference rate, structure check error rate, forward conversion failure rate, unmatched fragment rate, normalized time error, and attribute inconsistency rate; each item is normalized to zero to one, the weights are non-negative and the sum of the weights is one, and the attribute inconsistency rate is one minus the attribute consistency rate. The quality access control determination includes the following hard conditions, and the corresponding editing action sequence is isolated when it meets any of the hard conditions: the number of structural verification errors is greater than zero; there are unknown actions or unresolved necessary references; any necessary editing action in the forward conversion fails; the rights metadata associated with the corresponding native editing project is missing, the license period has expired, or the license purpose does not indicate that it is allowed to be used for model training.

[0014] In a preferred embodiment, constructing training samples for training a video editing agent based on the editing action sequence determined by the quality gate includes: The edited action sequence that passes the quality access control determination is used as a candidate sample; The normalized edited action sequence is serialized and a fingerprint hash value is calculated. The candidate samples are then precisely deduplicated based on the fingerprint hash value. Based on the weighted multiset similarity of ordered action n-tuples formed by adjacent clip action types, track information, and quantization time information, or the action sequence alignment similarity, the candidate samples after precise deduplication are clustered into near-duplicate clusters; within the same cluster, candidate samples are selected for retention based on the quality indicators, source integrity, and track coverage, and the retained candidate samples are used to construct the training samples.

[0015] In a preferred embodiment, the method further includes: grouping candidate samples according to their source project, project family, template family, or creator, treating each group as an indivisible unit, and dividing it as a whole into one of the training set, validation set, and test set; and classifying supervised samples, recall variant samples, and reinforcement learning trajectory samples generated from the same golden clip action sequence into the same dataset.

[0016] In a preferred embodiment, the method further includes: recording lineage information for each training sample, the lineage information including sample identifier, source project file hash value, rights source information, converter version, clipping action mode version, quality index, de-clustering identifier, dataset partitioning identifier, and deletion status; In response to the revocation of the source project's rights, training samples derived from the source project are deleted based on the lineage information.

[0017] In a preferred embodiment, a supervised fine-tuning sample is constructed based on the gold clip action sequence: the gold clip action sequence is normalized according to dependencies; a recall action is inserted by controlled removal of material required for subsequent clip actions and generation of a recall query; a termination action is appended to the end of the fully renderable sequence; a target position is sampled from the enhanced sequence, using the task instructions available before the target position, the current clip state, the action history, and the clip context as inputs, and the clip action, recall action, or termination action at the target position as the next action label.

[0018] The target position sampling method used in the supervised fine-tuning stage is similar to that used in mask data construction, but the action or state after the target position is not retained in the model input; each sampled position independently forms a causal next action prediction sample, and the clipping model still uses the autoregressive next term or next action prediction target.

[0019] In a preferred embodiment, a supervised fine-tuned video editing agent performs editing, recall, and termination actions in an editing sandbox, and generates rewarded trajectory samples based on action execution feedback and final output evaluation. Controlled perturbations such as missing fields, time conflicts, or broken references can also be applied to the golden editing action sequence to construct auxiliary structure repair samples labeled with the minimum repair action or the set of legal repair actions.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising: Memory, used to store computer-executable instructions; and, A processor, coupled to the memory, is configured to implement the steps of the method described above when executing the computer-executable instructions.

[0021] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the steps of the method described above.

[0022] In the embodiments of this disclosure, the original project is converted into a verified golden clip action sequence through engineering object graph inversion, deterministic forward loop closure, normalized semantic comparison, and quality gating. The clipping task is then modeled as a next action prediction task including three types of actions: clipping, recalling, and terminating. Supervised fine-tuning samples and clipping sandbox reinforcement learning trajectory samples are constructed from the golden clip action sequence, thus forming a verifiable closed loop from the actual completed project to the agent's training data. This process reduces manual reconstruction and annotation for each action, but does not exclude the inclusion of manually sourced task instructions, rights metadata, thresholds, and evaluation rules.

[0023] Furthermore, by selecting the actual project based on the number of valid tracks, the number of audio and video materials, and the valid duration when the root node is a container project, and by unifying the time unit, distinguishing between the source material extraction interval and the target time interval, and identifying the main track and the overlay track, it is possible to avoid parsing container projects with no substantial content and ensure the accuracy of time semantics and track semantics extraction.

[0024] Furthermore, by replacing replaceable media resource information and text content with placeholders to form input slots, and setting protection tags for template-specific resources, user-replaceable content can be distinguished from template-specific assets, preventing the model from mislearning template-specific assets as user input.

[0025] Furthermore, by sorting the editing actions within the same track by their start time and arranging the editing actions that generate the basic material clips before the effects actions and keyframe actions that depend on them, it can be ensured that the editing action sequence can be replayed in order, avoiding conversion failures caused by missing dependencies.

[0026] Furthermore, by projecting both projects onto the same supported semantic subspace, aligning resources using content or semantic stability keys, and explicitly including unmatched segments in partial one-to-one matching, we can reduce mismatches caused by differences in resource order and the number of segments, enabling quality metrics to simultaneously reflect structural preservation, time error, and attribute consistency.

[0027] Furthermore, by unifying the quality indicators into a normalized error term where a larger value indicates a larger deviation, constraining the non-negative weights and weight sums, and truncating the quality score to zero to one, we can avoid the attribute consistency rate and error rate moving in opposite directions or the weighted result going out of bounds. By setting hard gates through independent validators and associated rights metadata, we can prevent serious structural defects or authorization deficiencies from being masked by the average score.

[0028] Furthermore, by using precise deduplication based on fingerprint hash values ​​and near-repetition detection based on weighted multisets or sequence alignment of ordered action n-tuples, the order of actions and the number of repetitions can be preserved in the similarity score, reducing the risk of being misjudged as completely identical after the order of actions is swapped.

[0029] Furthermore, by treating the same source project, the gold clip action sequence and its derived supervised fine-tuning samples, recall variant samples and reinforcement learning trajectory samples as indivisible groups and assigning them only to one of the training set, validation set and test set, the inflated evaluation caused by trajectory-level homogeneous information leakage can be reduced.

[0030] Furthermore, by recording lineage information for each training sample, including the source of rights, version information, and quality indicators, and deleting derived samples based on lineage information when the source of rights is revoked, the entire process of training sample source and processing can be traced, supporting targeted management of derived samples.

[0031] Furthermore, by using causal location sampling to ensure that the next action sample does not contain the target action and future state, answer leakage can be reduced; by controlling the removal of materials required for subsequent actions and inserting recall actions, supervised samples can be constructed to determine whether to retrieve, with what query to retrieve, and how to continue editing after retrieval; by adding termination actions only in the fully renderable state and setting recall costs, hit and use rewards, illegal action penalties, and final state sparse rewards in the sandbox trajectory, duplicate recalls and early termination can be suppressed.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Various aspects and embodiments of this disclosure can be combined in different ways. This part of the disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to aid in determining the scope of the claimed subject matter. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a method for constructing video editing training samples according to an embodiment of the present disclosure.

[0034] Figure 2 This is a schematic diagram of the state transition of a video editing agent according to an embodiment of the present disclosure.

[0035] Figure 3 This is a schematic diagram of the process for constructing two-stage training samples according to an embodiment of the present disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1 This embodiment provides a method for constructing video editing training samples, which can be executed by electronic devices such as servers, cloud computing nodes, or workstations. The overall process of this method is as follows: Figure 1 As shown, the process mainly includes six steps: parsing the native editing project, establishing a global object index, extracting the editing action sequence, forward conversion, semantic comparison, and quality gate judgment. Each step is explained in detail below.

[0038] Step 101: parse the original editing project to obtain the project object reference graph.

[0039] A native editing project refers to the project data saved by the video editor, such as a project data file saved in JSON format and its associated resource directory. A native editing project contains a large number of objects, which reference each other through object identifiers. During parsing, the project data file is read, and the objects within it are reconstructed into a project object reference graph. The project object reference graph can be represented as follows: ,in, Represents a diagram of project object references. For a collection of objects, This is the set of edges. The set of objects can be further divided into... ,in, This represents a collection of material objects, which record the attributes and storage paths of materials such as videos, images, audio, and text. Represents a collection of track objects, which are containers for holding fragments and have track type and rendering order; This represents a collection of clip objects, which record the placement of footage on the final timeline. This represents a collection of auxiliary effect objects, such as speed adjustment objects, transition objects, filter objects, mask objects, animation objects, and keyframe objects. (Side collection) Record two types of relationships: one is the hierarchical containment relationship, such as a project containing tracks, and tracks containing segments; the other is the reference relationship, such as a segment referencing material objects and auxiliary effect objects through object identifiers.

[0040] Step 102: Create a global object index for each object in the project object reference diagram, with the object identifier as the key.

[0041] In native editing projects, clip objects typically only record object identifiers when referencing source material and effects; the actual content of the referenced objects is scattered across different locations in the project data. To enable tracing back to all referenced objects from any clip object during subsequent extraction, this step creates a global object index for all objects, categorized by type (e.g., media, text, speed, effects, transitions, animations, masks, etc.), using the object identifier as the key and the object content as the value. Once the global object index is established, any object identifier can be used to directly retrieve the corresponding object without repeatedly traversing the project data.

[0042] Step 103: Based on the global object index, extract time semantics, track semantics, and resource semantics from the project object reference graph, map supported objects to clip actions to obtain an executable clip action sequence, and generate conversion loss records for unsupported objects, unknown tracks, and unresolved references.

[0043] This step restores the materialized object states in the original editing project to operational semantics for the final cut timeline. Each extracted editing action can be represented as a six-tuple. ,in, Indicates the first One editing action, As an action type, For track information, The target time interval is the range in which this content appears on the final video's timeline. The target time interval is the range extracted from the source footage. Together with the source footage's extracted time interval, it constitutes the timing information for the editing action. For resource information, For action parameters. Temporal semantic extraction includes reading the source interval, target interval, and speed change information recorded in the clip object, while maintaining the distinction between the source material cut interval and the target time interval; track semantic extraction includes determining the track information and layer order of each clip action based on the type and rendering order of the track object; resource semantic extraction includes parsing the material objects referenced by the clip object into resource information through the global object index. Supported objects are mapped to clip actions according to predefined mapping rules. For example, video clip objects are mapped to the add video action, image clip objects are mapped to the add image action, audio clip objects are mapped to the add audio action, text objects are mapped to the add text action, keyframe objects are mapped to the add keyframe action, and supported transition objects and mask objects are mapped to the transition parameters, mask parameters, or corresponding auxiliary actions of the corresponding clip action. At the same time, for unsupported objects that do not belong to the preset mapping rules, unknown tracks whose type or affiliation cannot be determined, and unresolved references of referenced objects that cannot be found in the global object index, a conversion loss record is generated. The conversion loss record records the object identifier, category, and reason of the skipped or isolated object. "Executable" means that the resulting sequence of clip actions can be replayed sequentially by the forward converter without relying on other information in the original clip project.

[0044] Step 104: Convert the editing action sequence into a reconstructed editing project.

[0045] Specifically, a deterministic forward converter is used to process each editing action sequentially according to the order of the editing action sequence. Based on the action type, each editing action is expanded into objects such as tracks, clips, footage, and auxiliary effects, along with their reference relationships, thereby reconstructing a rebuilt editing project with the same format as the original editing project. The forward conversion forms the basis for loop-based verification of the reverse extraction results: if the editing action sequence has structural defects or semantic deviations, the rebuilt editing project will fail to generate, or will exhibit detectable differences from the original editing project.

[0046] Step 105: Normalize the original editing project and the reconstructed editing project respectively to obtain the feature sequence after removing non-semantic fields. Match and semantically compare the feature sequence of the original editing project and the feature sequence of the reconstructed editing project to obtain the quality index.

[0047] Even if the original and reconstructed editing projects are semantically identical, they will naturally differ in fields such as object identifiers, generation times, and file paths. These fields are unrelated to the editing content and are considered non-semantic fields. Normalization removes these non-semantic fields, transforming each project into a feature sequence that retains only the editing semantics. Then, the two feature sequences are matched, comparing each corresponding element to determine semantic equivalence, thus obtaining a quality metric reflecting the fidelity of the reverse extraction. This quality metric can include, for example, temporal errors and attribute-level consistency.

[0048] Step 106: Perform quality gate judgment based on quality indicators and conversion loss records, isolate editing action sequences that fail the quality gate judgment, determine editing action sequences that pass the quality gate judgment as gold editing action sequences, and construct supervised fine-tuning samples and reinforcement learning trajectory samples for the video editing agent based on the gold editing action sequences.

[0049] The quality gate judgment comprehensively considers the quality index obtained from semantic comparison and the transformation loss record generated in step 103: the quality index reflects the degree of deviation between the reconstructed action sequence and the original project in the supported semantic subspace, and the transformation loss record reflects the degree of information loss during the reverse extraction process. Editing action sequences that fail the judgment, along with their transformation loss records, enter the isolation zone; sequences that pass the judgment are designated as the gold editing action sequences. The video editing task is modeled as the next action prediction task, with the action space including editing actions, recall actions, and termination actions; supervised fine-tuning samples use causal position sampling that only contains information prior to the target action, and reinforcement learning trajectory samples are formed by the actual state transitions of the supervised fine-tuned agent in the editing sandbox.

[0050] In supervised fine-tuning data, recall actions are constructed by controlling the removal of footage required for subsequent gold actions, and the search results are added to the editing context in the next round; the corresponding recall is marked as a valid positive sample only when the search results contain qualified target footage and that footage is actually used by subsequent editing actions. Termination actions are only added when the complete state passes structure, reference closure, and renderability checks.

[0051] In reinforcement learning data, each recall action incurs a call cost, and a validity reward is generated when the reference material or semantically equivalent material is hit and subsequently used. Penalties are generated for repeated recalls, illegal editing actions, and early termination. When a valid termination occurs, a sparse final reward is calculated based on structural legality, task semantics or similarity to the reference film, and aesthetic quality.

[0052] Through steps 101 to 106 above, the complex native engineering is transformed into a sequence of golden clipping actions that has undergone loopback verification and quality gating. This sequence is then constructed as causal supervised fine-tuning samples covering three decision categories: clipping, recall, and termination, as well as clipping sandbox reinforcement learning trajectory samples. This process eliminates the need for manual reconstruction and annotation of each clipping action, and controls training bias through future information isolation, recall cost versus actual usage conditions, final state renderability verification, and homogeneous grouping.

[0053] The following describes several optional improvements to this embodiment.

[0054] In some optional implementations, the parsing in step 101 may also include the selection of valid projects. For some native editing projects, the root node is simply a container project. A deterministic candidate score can be constructed based on the number of valid tracks, the number of audio / video materials, the valid duration, and the track rendering order. The actual project is then selected from the container projects for parsing. Main track identification can comprehensively consider track media type, track rendering order, and the duration of visible content; when the preset editor format provides an explicit main track marker, that marker is used preferentially. Correspondingly, step 103 uniformly converts the time amount to a preset time unit, distinguishes between the source material extraction interval and the target time interval, and determines the track information and layer order based on the track identification results.

[0055] Optionally, placeholder processing can be performed on the editing action sequence: replace replaceable media resource information and text content in the editing action sequence with placeholders to form replaceable input slots. For example, replace the material path of a video clip with a placeholder like Media Slot 1, and replace a title text with a placeholder like Text Slot 1. At the same time, protection tags are set for objects that are inherent resources of the template in the native editing project, such as the template's own background music and decorative patches. Resources with protection tags are not replaced with placeholders. The editing action sequence after placeholder processing becomes a reusable parameterized template, which also avoids the model mislearning the template's inherent assets as user input. When the training objective does not involve material replacement, placeholder processing can be omitted, and training samples can be constructed directly using specific material information.

[0056] As a preferred but not mandatory improvement, the sequence of edited actions can be normalized before performing the forward transformation in step 104. First, a directed acyclic dependency graph is constructed based on the target references and dependency references of the actions, and then a deterministic topological sorting is performed. For actions with the same dependency level, the sorting keys are sequentially determined by the track rendering order, target time start point, action type priority, and intra-sample logical identifier. Multiple actions constituting the same visual effect are recorded with a unified action group identifier and replayed as an atomic group. When multiple legal next actions exist in the same pre-execution state, the training sample records the set of legal actions, or the above sorting rules are used to determine the normalization linearization target to avoid incorrectly labeling other legal actions as negative samples.

[0057] Example 2 Based on Embodiment 1, this embodiment further explains a specific implementation of normalization, matching, and semantic comparison in step 105, as well as a specific implementation of quality access control determination.

[0058] In terms of normalization, the original editing project and the reconstructed editing project are first projected to the same supported semantic subspace, and then non-semantic fields such as object identifiers, generation time, and machine-related absolute paths are removed. Resources are no longer renamed simply according to their order of appearance, but instead, stable placeholders are generated based on content hashes, perceptual hashes, input slot identifiers, or first-time use semantic keys. Time quantities are quantized to a preset precision, and field keys are stably sorted. The feature sequence obtained after normalization includes at least track type, track level, segment start point, segment duration, source material cut range, resource type, resource semantic key, and supported effect parameters.

[0059] For matching and semantic comparison, a minimum-cost partial one-to-one matching of segments in two feature sequences is performed, allowing virtual nodes to be matched. The matching cost is determined by track differences, time differences, resource type differences, and resource semantic differences. Unmatched segments in the original or reconstructed project are added to the unmatched set and subject to a preset penalty. Let M be the set of real segment matches, U_O and U_R be the sets of unmatched segments on both sides, and N be the sum of the number of matched pairs and the number of unmatched segments on both sides. Segment coverage, normalized time error, and attribute consistency rate are calculated based on both matched and unmatched segments. Unmatched segments are counted as unit errors in the time error and as zero consistency in the attribute consistency rate; N is the zero-time-isolated space-time project sample.

[0060] For the comprehensive quality metrics, at least two of the following are used as error terms: reverse skip rate, unresolved reference rate, structure check error rate, forward conversion failure rate, unmatched fragment rate, normalized time error, and attribute inconsistency rate. Each error term is normalized to zero to one, and the attribute inconsistency rate is calculated by subtracting the attribute consistency rate from one. Each weight is a non-negative number, and the sum of the weights is one. The quality score is calculated by subtracting the weighted error sum from one and truncating it to zero to one. The weights and thresholds are determined using a validation set isolated from the training set.

[0061] In terms of quality gate detection, in addition to screening based on quality scores, hard conditions are set. A corresponding edit sequence is directly isolated if it meets any of these conditions: the number of structural validation errors is greater than zero; there are unknown actions or unresolved necessary references; any necessary edit action fails during forward conversion; the independent validator detects unclosed references, track constraints, or time constraints; or the rights metadata associated with the original edit project is missing, the license period has expired, or the licensed purpose does not indicate permission for model training. The authorization status is provided by the associated metadata and is not inferred from the project content.

[0062] Example 3 This embodiment describes optional improvements to deduplication, dataset partitioning, and lineage management in the training sample construction process, based on the above embodiments.

[0063] In some optional implementations, edited action sequences that pass quality gate detection are used as candidate samples. Deterministic serialization and fingerprint calculation are performed on the normalized action sequences to achieve accurate deduplication. For near-duplicate detection, action type, trajectory, quantization time, target reference, and action group information are encoded into ordered action n-tuples. Weighted multiset similarity or sequence alignment similarity are used to ensure that both the number of repetitions and the order of actions are compared. Samples with similarity exceeding a threshold are clustered together, and retained samples are selected based on quality score, source integrity, and trajectory coverage.

[0064] Building upon the above scheme, further grouping and segmentation can be implemented: candidate samples are grouped according to their source project, project family, template family, or creator. All samples within the same group are assigned to only one set in the training, validation, or test sets, without cross-set splitting. Supervised fine-tuning samples generated from the same gold clip action sequence at different target locations, recalled variant samples generated through different controlled removal methods, and reinforcement learning trajectory samples generated from the same task and reference project all inherit the grouping key of the gold clip action sequence. Random segmentation can be used when sample sources are independent and have no common origin.

[0065] Optionally, lineage information can also be recorded for each training sample. This lineage information includes sample identifier, source project file hash value, rights source information, converter version, clipping action mode version, quality metrics, deduplication clustering identifier, dataset partitioning identifier, and deletion status. The rights source information records the scope of authorization, permitted purpose, and license period of the sample's source project; the converter version and clipping action mode version are used to trace the sample's generation environment after the evolution of conversion rules or action modes; and the deletion status indicates whether the sample has been removed from the dataset. When the rights source of a source project is revoked, all training samples derived from that source project are located based on the lineage information and deleted, thereby achieving traceable management of training data.

[0066] Example 4 This embodiment, based on the above embodiments, further explains the specific methods for constructing supervised fine-tuning samples based on gold clip action sequences and reinforcement learning trajectory samples based on clip sandbox running trajectories. The overall process of constructing training samples in the two stages is as follows: Figure 3 As shown, the supervised fine-tuning phase is used to enable the video editing agent to learn the basic distribution of the next action, while the reinforcement learning phase is used to enable it to learn the trade-offs between recall cost, action legality, and final output quality in an executable environment.

[0067] Training samples can be categorized into different levels based on whether they contain traceable task instructions. For native editing projects without traceable task instructions, structural samples are constructed. These structural samples consist of a metadata summary of the project, source material information, and editing action sequences. The metadata summary includes, for example, canvas, frame rate, duration, and track overview. For projects with traceable task instructions, such as those retaining original creative requirements or editing records, the task instructions can be combined with the above content to form instruction samples. When the model generates task descriptions, the generated task descriptions should be labeled as synthetic instructions and counted separately from real instructions to avoid impersonating genuine user instructions.

[0068] In some optional implementations, the action space of the video editing agent is defined as three categories: editing actions, recall actions, and termination actions. An editing action corresponds to an atomic operation or a group of actions with atomic submission semantics in the intermediate representation of the edit; a recall action represents a combination of a recall flag and a retrieval query, used to perform a retrieval on the vector library and add the result to the retrieval context of the next state; a termination action indicates that the current editing project has passed the quality gate and is ready for rendering. The state transition process of the video editing agent driven by these three types of actions is as follows: Figure 2As shown. Based on the example action trajectory, the target position is selected, and the previous action is replayed to obtain the state before execution. The task instruction, current project state, used materials, action history, and retrieval context are used as inputs, and the next action at the target position is used as the supervision target. The occlusion is only used for offline selection of supervision positions and hiding future trajectories. The model still uses autoregressive next action or next term prediction for the target, and the input does not include the target action and its subsequent state information.

[0069] Optionally, repair samples can also be constructed: controlled perturbations such as missing fields, time conflicts, or broken references are applied to the action sequence, the perturbed state is used as input, the minimum repair increment or the set of valid repairs is used as the supervision target, and the loss is calculated only for the repair position, avoiding requiring the model to re-output the complete action sequence.

[0070] When constructing recall action samples, target footage required for subsequent gold actions is removed from the initial retrieval context in a controlled manner. A query without a unique identifier for the footage is generated based on the content description, scene tags, and editing purpose of the target footage. This recall action is then used as the current supervision target. The retrieval system returns candidate footage based on the query and injects it into the next state. Only when the candidates contain qualified target footage and that footage is actually used in subsequent editing actions is this recall marked as a valid positive sample.

[0071] For complete gold example tracks that pass the quality gate, a termination action is appended after the last clip action. The supervisory state of the termination action must pass action structure, reference closure, timeline, and renderability checks; termination actions are not appended to tracks that are truncated or still have unmet dependencies.

[0072] Example 5 This embodiment describes the construction of reinforcement learning trajectory samples based on supervised fine-tuning samples. The supervised fine-tuned video editing agent is placed in an editing sandbox. The sandbox state includes task instructions, the current intermediate representation of the edit, action history, retrieval context, and remaining recall budget. After the model outputs an action, the sandbox performs editing operations, vector retrieval, or termination verification, and records the state, action, next state, immediate reward, and final reward.

[0073] Immediate rewards are used to constrain recall behavior: a hit reward is given when the recall result contains the material required for the gold trajectory and that material is subsequently used; each recall deducts the invocation cost; penalties are imposed for duplicate queries, invalid queries, exceeding the recall budget, and illegal actions in the output structure. Using actual use as the hit condition can prevent the model from accumulating rewards through repeated recalls.

[0074] The final reward is calculated only when the model outputs the termination action. After the current project passes the structural verification, reference closure verification, and renderability verification, a sparse reward is calculated based on the semantic consistency of the task, the similarity of the audiovisual features with the target, and the aesthetic quality; penalties are imposed if the verification fails, the project is terminated early, or the task is not completed after the budget is exhausted.

[0075] Example 6 This embodiment provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for constructing video editing training samples provided in any of the above embodiments. The electronic device can be a server, a cloud virtual machine, or a workstation. The processor can be a central processing unit (CPU). When batch processing of large-scale engineering data is required, a multi-core CPU combined with distributed task scheduling can be used to distribute the parsing and loopback verification of different native editing projects to multiple computing nodes for parallel execution. The memory can include RAM and non-volatile storage components such as hard disks and solid-state drives (SSDs). In addition to the computer program, the memory can also store native editing projects, isolation area data, training samples, and their lineage information.

[0076] One embodiment of this disclosure also provides a computer-readable storage medium storing computer-executable instructions (or a computer program) that, when executed by a processor, implement the steps in the above-described method embodiments. The computer-readable storage medium may include any type of volatile or non-volatile memory, or any combination thereof, storing computer program code. Specifically, the computer-readable storage medium may include, but is not limited to: magnetic storage devices (e.g., magnetic tape, hard disk, floppy disk), optical storage devices (e.g., optical disc (CD), digital versatile optical disc (DVD)), magneto-optical storage devices (e.g., magneto-optical disc), semiconductor memory (e.g., read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc.) or other storage technologies. The computer-readable storage medium in this disclosure refers to a non-transitory readable storage medium, excluding the transient propagation signal itself (e.g., modulated data signals, carrier waves, etc.).

[0077] The terms “comprising,” “including,” or any other variations thereof, as used herein, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The phrase "execute according to a certain element" as used in this specification means at least according to that element, including both "execute only according to that element" and "execute according to that element and other elements".

[0079] Those skilled in the art will understand that variations of the method steps described above can be obtained based on the embodiments of this disclosure without any inventive effort. For example, the sequence numbers of the steps described in the method embodiments of this specification do not themselves constitute a limitation on the execution order of these steps. Unless there is an explicit specific limitation in the context, the steps may be executed in a different order than that shown in the embodiments (e.g., executing the steps with larger sequence numbers first, then the steps with smaller sequence numbers), or they may be executed in parallel. Furthermore, other steps may be inserted between multiple steps with consecutively numbered sequence numbers.

[0080] Furthermore, it should be understood that after reading the contents of this disclosure, those skilled in the art can make various alterations or modifications to this disclosure, and these equivalent forms also fall within the scope of protection claimed by this disclosure.

Claims

1. A method for constructing training samples for a video editing intelligent agent, characterized in that, include: The original editing project is parsed to obtain the project object reference diagram, which includes material objects, track objects, clip objects and auxiliary effect objects, as well as the hierarchical inclusion relationship and reference relationship between each object; A global object index is established for each object in the project object reference diagram, with the object identifier as the key; Based on the global object index, time semantics, track semantics, and resource semantics are extracted from the project object reference graph. Supported objects are mapped to editing actions to obtain an executable editing action sequence. Conversion loss records are generated for unsupported objects, unknown tracks, and unresolved references. Each editing action includes an action type, track information, time information, resource information, and action parameters. The time information includes at least one of the following based on the action type: source material cut-out interval, target time interval, or action time. For editing actions on media that can be cut, the time information includes both the source material cut-out interval and the target time interval. The editing action sequence is forward-converted into a reconstructed editing project; The original editing project and the reconstructed editing project are normalized to obtain feature sequences after removing non-semantic fields. The feature sequences of the original editing project and the reconstructed editing project are matched and semantically compared to obtain quality indicators. A quality gate is determined based on the quality index and the conversion loss record, and editing action sequences that fail the quality gate are isolated. Training samples for training a video editing agent are constructed based on the editing action sequences that pass the quality gate. The action space of the video editing agent includes at least editing actions, recall actions carrying retrieval queries, and termination actions indicating the termination of the editing task. The training samples include at least causal next-action supervision samples, which take the pre-execution state of the target action as input and the target action as the supervision target. The target action is selected from the editing action sequences that pass the quality gate, and the input does not contain the state information corresponding to the target action and its subsequent actions.

2. The method according to claim 1, characterized in that, The parsing of the native editing project includes: when the root node of the native editing project is a container project, selecting an actual project from the projects contained in the container project for parsing based on the number of valid tracks, the number of audio and video materials, and the valid duration; The extraction of time semantics, track semantics, and resource semantics includes: uniformly converting the time amount in the reference diagram of the project object into a preset time unit, distinguishing the source material extraction interval and the target time interval; identifying the main track and overlay track according to the preset track recognition rules, and determining the track information and layer order of each editing action.

3. The method according to claim 1, characterized in that, The method further includes: Replace the replaceable media resource information and text content in the editing action sequence with placeholders to form replaceable input slots; A protection tag is set for objects that are inherent resources of the template in the native editing project. Resources with the protection tag are not replaced with placeholders.

4. The method according to claim 1, characterized in that, The method further includes: performing sequential normalization on the editing action sequence before forward converting the editing action sequence into a reconstructed editing project; The sequence normalization includes: sorting the editing actions within the same track according to the start time of the target time interval; arranging the editing actions that generate the basic material clips before the effect actions and keyframe actions that depend on the basic material clips, to obtain a replayable editing action sequence.

5. The method according to claim 1, characterized in that, The normalization includes: projecting the original editing project and the reconstructed editing project to the same supported semantic subspace; removing the object identifier field, the generation time field, and the machine-related absolute path field; generating stable placeholders for similar resources based on at least one of content hash, perceptual hash, input slot identifier, or first-use semantic key; quantizing the time quantity to a preset precision; and performing stable sorting of the object's field keys. The feature sequence includes track type, track level, segment start point, segment duration, source material extraction range, resource type, and supported effect parameters; The matching and semantic comparison include: performing a minimum cost one-to-one matching on the segments in the feature sequences of the original editing project and the reconstructed editing project, allowing some segments to be unmatched; applying a preset cost to the unmatched original segments and the unmatched reconstructed segments respectively; and calculating the segment coverage, normalized time error and attribute consistency rate based on the matched segments and the unmatched segments.

6. The method according to claim 1, characterized in that, The quality metrics are calculated by weighting at least two of the following: reverse skip rate, unresolved reference rate, structure check error rate, forward conversion failure rate, unmatched fragment rate, normalized time error, and attribute inconsistency rate; each metric is normalized to zero to one, the weights are non-negative and the sum of the weights is one, and the attribute inconsistency rate is one minus the attribute consistency rate. The quality access control determination includes the following hard conditions, and the corresponding editing action sequence is isolated when any of the hard conditions is met: the number of structural verification errors is greater than zero; there are unknown actions or unresolved necessary references; any necessary editing action in the forward conversion fails; a reference not closed, track constraint violation, or time constraint violation is detected based on the independent verifier; the training authorization in the rights metadata stored in association with the original editing project is missing or has expired.

7. The method according to claim 1, characterized in that, The step of constructing training samples for training a video editing agent based on the editing action sequence determined by the quality gate includes: The edited action sequence that passes the quality access control determination is used as a candidate sample; The normalized edited action sequence is serialized and a fingerprint hash value is calculated. The candidate samples are then precisely deduplicated based on the fingerprint hash value. The candidate samples after precise deduplication are clustered into near-repetition clusters based on the weighted multiset similarity of the n-tuples of the edited actions or the action sequence alignment similarity, so that the similarity simultaneously preserves the number of times the actions occur and the order of the actions; within the same cluster, the candidate samples to be retained are selected according to the quality index, and the retained candidate samples are used to construct the training samples.

8. The method according to claim 7, characterized in that, The method further includes: grouping the candidate samples according to the source project, project family, template family, or creator to which the candidate samples belong, and assigning all candidate samples in the same group to only one dataset in the training set, validation set, or test set; all training samples generated from the same editing action sequence that passes the quality gate are assigned to the same dataset; For each training sample, lineage information is recorded, including sample identifier, source project file hash value, rights source information, converter version, clipping action mode version, quality index, de-clustering identifier, dataset partition identifier, and deletion status. In response to the revocation of the source project's rights, training samples derived from the source project are deleted based on the lineage information.

9. The method according to claim 1, characterized in that, The editing action corresponds to an atomic editing operation or a group of editing actions with atomic submission semantics. The recall action includes a recall flag and the retrieval query. The termination action indicates that the current editing project can be sent for rendering. The construction of training samples for training the video editing agent based on the editing action sequence determined by the quality gate includes: Select a target position from the edit action sequence, replay the actions before the target position to obtain the pre-execution state, and construct a causal next action supervision sample with task instructions, current editing project state, used materials, action history, and current search context as inputs and the next action of the target position as the supervision target; the input does not include the state information corresponding to the target action and its subsequent actions; when there are multiple legal next actions in the same state, select a standard target according to the dependency graph and deterministic sorting rules, or record the set of legal actions together as the supervision target; When constructing recall action samples, the target material required for subsequent editing actions is removed from the initial context. A retrieval query without a unique identifier of the material is generated based on the content semantics of the target material. The recall action is used as a supervision target, and the retrieved candidate material is injected into the next state. The recall action is marked as a valid recall only when the recall result contains the target material or a material that is semantically equivalent to the target material, and the material is actually used by the subsequent editing action. For action sequences that pass the quality gate and are fully renderable, a termination action is appended to the end of the sequence.

10. The method according to claim 9, characterized in that, The method further includes: supervising and fine-tuning the video editing agent using the causal next action supervision sample; running the supervised and fine-tuned video editing agent in the editing sandbox, outputting editing action, recall action or termination action in response to the current state, executing the action and recording the state transition, and obtaining reinforcement learning trajectory samples; For recall actions, a hit reward is given when the recall result contains the material in the dataset that corresponds to the subsequent target clip action and the material is actually used. A call cost is imposed on each recall action, and penalties are imposed on duplicate queries, invalid queries, and behaviors that exceed the recall budget. For termination actions, after the current editing project passes structural verification, reference closure verification, and renderability verification, the final reward is calculated based on the semantic consistency of the task, the similarity of the audiovisual features with the target final cut, and the aesthetic quality; if the verification fails, a penalty is imposed on early termination.

11. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; as well as, A processor, coupled to the memory, is configured to implement the steps of the method according to any one of claims 1 to 10 when executing the computer-executable instructions.

12. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images