Video editing preview method and device based on editing intention graph and anchor constraint

Abstract

The application relates to a video editing preview method and device based on an editing intention graph and anchor point constraints. After a video editing draft is acquired, an editing intention graph composed of object nodes and relationship edges is generated, a hard anchor point set and a soft anchor point set are determined based on the graph, and anchor point constraint relationships are established, object-level hierarchical proxy playback expressions are generated for the object nodes, device resource states are acquired, and a target playback strategy combination that minimizes preview deviation cost is determined on the premise that resource constraints and hard anchor point timing alignment constraints are met, playback is performed under a unified master clock, and a preview result is output, preview deviations are monitored, and local correction is performed on related objects when the preview deviations exceed a preset deviation threshold. According to the application, when device resources are limited, the correct expression of key editing intentions can be preferentially maintained without relying on whole-frame uniform degradation, and key deviations between a preview result and an exported result are reduced.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video editing preview method and apparatus based on editing intent graphs and anchor point constraints. Background Technology

[0002] In video editing applications, users need to preview the edited effects in real time before exporting the final product. To maintain smooth previewing on mobile devices with limited computing power, memory, temperature, and battery, existing video editing preview methods typically reduce the resolution of the entire frame or skip frames when device resources are scarce, or pre-render the editing results as a proxy file to reduce the burden of real-time computing.

[0003] Because uniform downgrading of the entire frame does not distinguish the importance of each object in the picture, objects that carry the user's core editing intentions, such as subtitle entry points, music beats, and transition peak frames, are downgraded in the same way as background decorations. This leads to the distortion of key information that users use to judge the editing effect, and may even mislead users to make incorrect editing adjustments. While using pre-rendered proxy files can reduce real-time calculations, the proxy files need to be regenerated after the editing parameters change, so they cannot reflect the user's latest editing in real time. Moreover, it is also difficult to maintain the fidelity for individual objects. Summary of the Invention

[0004] In order to maintain the correct expression of key editing intents without relying on uniform degradation of the entire frame, and to reduce the key deviation between the preview result and the exported result, under the conditions of limited device computing power, memory, temperature and power, this application provides a video editing preview method and apparatus based on editing intent map and anchor point constraints.

[0005] Firstly, this application provides a video editing preview method based on editing intent graphs and anchor point constraints, which adopts the following technical solution:

[0006] A video editing preview method based on editing intent graph and anchor point constraints includes:

[0007] S1. Obtain the video editing draft, which includes track information, object information, parameter information, and user editing behavior information within a preset time window;

[0008] S2. Based on the video editing draft, generate an editing intent graph consisting of object nodes and relation edges;

[0009] S3. Determine the set of hard anchor points and the set of soft anchor points based on the editing intent map, and establish anchor point constraint relationships;

[0010] S4. For at least a portion of the object nodes in the editing intent map, generate object-level hierarchical proxy playback representations;

[0011] S5. Obtain the device resource status. Under the premise of satisfying the resource constraints of the device resource status and the time alignment constraints of the hard anchor set, determine the target playback strategy combination that minimizes the preview deviation cost based on the anchor constraint relationship and the hierarchical proxy playback expression.

[0012] S6. Under a unified master clock, perform original playback or proxy playback on the objects in the current playback window according to the target playback strategy combination, perform timing alignment control on multi-track objects and output preview results;

[0013] S7. Monitor the preview deviation of the preview result relative to the exported baseline expression, and perform local correction on the object corresponding to the preview deviation if the preview deviation exceeds the preset deviation threshold.

[0014] By adopting the above technical solutions, the editing intent graph structures the objects and relationships in the video editing draft, enabling the underlying player to recognize the user's editing goals. The hard anchor set and soft anchor set distinguish between objects that must be strictly maintained and objects that are allowed to be approximated. The hierarchical proxy playback expression at the object level generates differentiated approximate expressions for different objects, rather than uniformly downgrading the entire frame. The target playback strategy combination solves the problem with the goal of minimizing the preview deviation cost under the constraints of device resource constraints and the timing alignment constraints of the hard anchor set. The unified master clock ensures that the timing of the objects corresponding to the hard anchors does not drift beyond the threshold. When the preview deviation exceeds the preset deviation threshold, only local correction is performed on the relevant objects. Thus, when resources are limited, the correct expression of key editing intent is maintained first, and the key deviation between the preview result and the exported result is reduced.

[0015] Optionally, the set of hard anchor points includes objects or moments corresponding to at least one of the following: subtitle in point, subtitle out point, audio rhythm hit point, transition peak frame, speed change point, subject boundary key point, lip-sync point, and key control point of the user's current editing object; the set of soft anchor points includes objects or moments corresponding to at least one of the following: background blur area, decorative particle effect, non-critical background filter, and low-sensitivity object that is not directly related to the current editing target.

[0016] By adopting the above technical solution, highly sensitive user objects such as subtitles, rhythm hits, and transition peak frames are categorized into hard anchor point sets, while backgrounds and decorative effects are categorized into soft anchor point sets, thus aligning the protected objects with the user's editing focus when resources are limited.

[0017] Optionally, in S4, the hierarchical proxy playback expression includes at least one of temporal proxy, geometric proxy, edge proxy, color proxy, and special effects proxy, and generates a proxy that maintains key performance for the objects corresponding to the hard anchor set, and generates a low-level proxy for the objects corresponding to the soft anchor set.

[0018] By adopting the above technical solution, the proxy category is selected differently according to the anchor point affiliation of the object. Hard anchor point objects retain key performance while soft anchor point objects adopt low-level proxies, thus achieving a balance between fluency and the authenticity of key content.

[0019] Optionally, in S7, the monitoring of preview deviation is based on the prediction benchmark of historical keyframe features or preset error model; in response to the cumulative deviation of the object corresponding to the hard anchor point set exceeding the preset cumulative deviation threshold for multiple consecutive frames, a short window real recalculation is triggered for the object corresponding to the hard anchor point set, and the derived benchmark expression is updated with the result of the short window real recalculation.

[0020] By adopting the above technical solutions, routine monitoring is carried out with a low-cost prediction benchmark, and the high cost of real recalculation is only spent on objects that are associated with hard anchors and continuously deviate, thus aligning the high-fidelity budget of deviation monitoring with key editorial intentions.

[0021] Optionally, in S7, in response to multiple objects simultaneously triggering short-window real recalculation or local correction, multiple objects are sorted according to a composite sorting key consisting of the priority of the object corresponding to the hard anchor set and the preview deviation cost, and local correction is performed serially according to the sorting result. The resource reserve reserved for the current playback window by the target playback strategy combination is used as the correction quota. In response to the exhaustion of the correction quota, the objects that have not undergone local correction are postponed to the next playback window, and the sorting weight of the objects that have not undergone local correction is increased.

[0022] By adopting the above technical solutions, concurrent correction uses the reserved resource margin of the target playback strategy combination to avoid the correction itself from breaking through resource constraints and causing new lag. The delay and ranking improvement when the quota is exhausted prevent low priority objects from not being corrected for a long time.

[0023] Optionally, a hysteresis band and a minimum hold time are applied to the playback strategy switching and local correction of objects corresponding to the soft anchor point set, while no hysteresis band is applied to objects corresponding to the hard anchor point set, and the threshold of the hysteresis band is adaptively adjusted according to the device resource status.

[0024] By adopting the above technical solution, a hysteresis is applied to the policy switching of soft anchor objects to suppress jitter, and hard anchor objects are exempted to ensure immediacy. The hysteresis threshold is adaptive with the resource status, taking into account both stability and timely protection of critical objects.

[0025] Optionally, it also includes: recording the editing intent map, anchor point constraints, target playback strategy combination, and local correction actions used in this playback, and generating preview description data; the preview description data is used to reuse the target playback strategy combination when the same video editing draft is opened again.

[0026] By adopting the above technical solution, the decision made during a replay is condensed into preview description data, which can be reused directly when the same draft is opened again, avoiding repeated solutions and improving the reproducibility of the replay decision.

[0027] Optionally, when restoring video editing drafts across devices, the preview description data can be reused. When reusing, the anchor point constraint relationship corresponding to the hard anchor point set remains unchanged. Only the proxy playback level of the object corresponding to the soft anchor point set is scaled according to the device resource status of the target device, or the target playback strategy combination is re-determined based on the anchor point constraint relationship corresponding to the hard anchor point set.

[0028] By adopting the above technical solution, the hard anchor point constraints remain unchanged when reused across devices, and only the soft anchor point level scales with the computing power of the target device, so that the key editing intent remains consistent on different devices, rather than the key content adapts to the device capabilities.

[0029] Optionally, the data structure for editing intent graphs includes one or more of the following: graph database, adjacency list, adjacency matrix, relation table, and index tree.

[0030] By adopting the above technical solutions, the editing intent graph can select an appropriate data structure based on storage and query requirements, thereby improving the engineering flexibility of graph construction and retrieval.

[0031] Optionally, object nodes include one or more of the following: video clip nodes, audio clip nodes, subtitle nodes, special effects nodes, transition nodes, main area nodes, mask area nodes, lip-sync area nodes, and user focus nodes. Relationship edges include one or more of the following: time alignment relationship, occlusion relationship, spatial dependency relationship, action association relationship, perceived priority relationship, export consistency sensitive relationship, and user editing association relationship.

[0032] By adopting the above technical solution, multiple types of object nodes and relational edges are carried on the above data structure, enabling the graph to express the temporal, spatial and editing relationships between objects, and providing a structural basis for subsequent anchor point division.

[0033] Optionally, the division between the hard anchor set and the soft anchor set can be determined based on one or more of the following: object type, user editing behavior, object visual salience, audio rhythm relevance, and export consistency sensitivity level.

[0034] By adopting the above technical solution, the anchor point division is determined based on multi-dimensional attributes, making the identification of key objects more in line with the user's editing intent and the consistency requirements of export.

[0035] Optionally, in the early stages of playback before the user has made any editing moves, each object is assigned an initial value for anchor point classification and proxy playback level based on the prior weight corresponding to the object type. As the user's editing moves accumulate, the prior weights are gradually transitioned to weights based on editing moves according to a time window.

[0036] By adopting the above technical solution, key objects can be distinguished a priori based on object type and layered proxies can be applied during the cold start stage when there is no editing input, thus avoiding the degradation of the entire frame to a uniform degradation in the early stage of playback.

[0037] Optionally, preview deviations include one or more of the following: temporal deviation, spatial deviation, boundary deviation, color deviation, audio-visual synchronization deviation, and cumulative deviation across multiple consecutive frames.

[0038] By adopting the above technical solution, the deviation between the preview result and the exported benchmark expression is characterized from multiple dimensions, providing detailed criteria for triggering local correction.

[0039] Optionally, local correction may include one or more of the following: increasing the proxy playback level of the object corresponding to the preview deviation, increasing the priority of the object corresponding to the preview deviation, triggering keyframe recalculation, triggering short window precalculation, and performing anchor point correction.

[0040] By adopting the above technical solution, corresponding correction actions are selected for different types of deviations, and only local corrections are made to key objects without uniform adjustment of the entire frame.

[0041] Optionally, the exported baseline representation is determined based on one or more of the following: baseline results generated by the exported rendering link, historical keyframe features, local region features, and preset error models.

[0042] By adopting the above technical solutions, the derived benchmark expression can be selected from different sources according to accuracy and cost requirements, supporting the feasibility of deviation monitoring in different scenarios.

[0043] Optionally, device resource status includes one or more of the following: CPU utilization, GPU utilization, neural network processor utilization, memory availability, temperature status, battery status, single-frame processing time, and historical frame latency.

[0044] By adopting the above technical solution, multi-dimensional resource status provides complete resource constraint input for solving the target replay strategy combination.

[0045] Optionally, the combination of target replay strategies is determined by one or more of heuristic search, dynamic programming, constraint optimization, and near-optimal selection.

[0046] By adopting the above technical solutions, the solution method can be flexibly selected according to the problem size and real-time requirements, achieving a balance between solution quality and solution time.

[0047] Optionally, when the number of object nodes in the edit intent graph increases, fine-grained solutions are performed only on object nodes within the current playback window and adjacent to them, while graph pruning or block approximation solutions are performed on object nodes outside the playback window, while preserving their hard anchor point constraints.

[0048] By adopting the above technical solution, the fine solution is focused on the current playback window and adjacent objects, and objects outside the window are pruned while retaining hard anchor point constraints, so that the real-time performance of the solution can still be maintained even with very large drafts.

[0049] Secondly, the electronic device provided in this application adopts the following technical solution:

[0050] An electronic device comprising:

[0051] One or more processors;

[0052] Memory;

[0053] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to:

[0054] Perform the video editing preview method described above, which is based on the editing intent graph and anchor point constraints.

[0055] Thirdly, this application provides a computer-readable storage medium that adopts the following technical solution:

[0056] A computer-readable storage medium storing an application program that can be loaded by a processor and executed as described above.

[0057] The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following:

[0058] The video editing preview method based on editing intent graphs and anchor point constraints, as described above.

[0059] In summary, this application includes at least one of the following beneficial technical effects:

[0060] 1. By identifying key objects through editing intent graphs, prioritizing them by distinguishing between hard anchor point sets and soft anchor point sets, generating hierarchical proxy playback expressions for objects, and solving playback strategies with the goal of minimizing preview deviation costs under device resource constraints and hard anchor point timing alignment constraints, the correct expression of key editing intents such as subtitles, key points, main body, and transition peak frames is prioritized when device computing power, memory, temperature, and power are limited, avoiding preview misleading caused by uniform downgrading of the entire frame.

[0061] 2. By using a unified master clock, the timing of objects corresponding to hard anchor points is guaranteed not to drift beyond the threshold. When the preview deviation exceeds the preset deviation threshold, only local correction is performed on the relevant objects. Combined with the recording and reuse of preview description data, the key deviations between the preview results and the exported results are continuously controlled, and the playback decision remains consistent when the draft is reopened or restored across devices.

[0062] 3. By using a tiered deviation benchmark, the high cost of actual recalculation is only spent on objects with hard anchor points and continuous deviations. Shared resource margins are allocated and prioritized for concurrent corrections. Non-uniform hysteresis with anchor point awareness is applied to soft anchor point objects. When reusing across devices, hard anchor point constraints are maintained, and only the soft anchor point level is scaled. This ensures that deviation monitoring, concurrency management, and cross-device adaptation maintain key editing objectives. Figure 1 Further reduce resource consumption and jitter while ensuring consistency. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the functional module architecture of a video editing and preview method in one embodiment of the present invention.

[0064] Figure 2 This is a flowchart of a video editing and preview method in one embodiment of the present invention.

[0065] Figure 3 This is a schematic diagram of the structure of an editing intent map in one embodiment of the present invention.

[0066] Figure 4 This is a sub-flowchart of deviation monitoring and local correction in one embodiment of the present invention. Detailed Implementation

[0067] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the application and are not intended to limit the scope of the application.

[0068] This application provides a video editing preview method based on an editing intent graph and anchor point constraints. In the low-level preview stage of the video editing draft, the method structures the objects and their associations in the draft into an editing intent graph. Based on this, it distinguishes between hard anchor points that need to be strictly maintained and soft anchor points that allow approximation. It generates object-level hierarchical proxy playback representations for different objects, and determines a playback strategy with the goal of minimizing preview deviation costs when device resources are limited. It maintains the timing alignment of key objects by unifying the master clock, and performs local correction only on related objects when preview deviations exceed limits. Therefore, under conditions of limited computing power, memory, temperature, and power, it prioritizes the correct expression of key editing intents, as described in the reference... Figure 1The video editing preview method provided in this application embodiment can be achieved collaboratively by functional modules such as editing intent graph generation, anchor point determination, hierarchical proxy playback generation, budget-aware constraint solving, unified master clock and playback control, deviation monitoring and local correction, and preview description recording.

[0069] Before detailing the embodiments of this application, some terms will be explained first.

[0070] An editing intent graph is a data structure consisting of object nodes and relational edges. Object nodes represent editing objects in a draft, and relational edges represent the relationships between objects.

[0071] Hard anchors are objects or moments that need to be strictly timed or accurately represented during preview and playback.

[0072] Soft anchor points are objects or moments that can be approximated within a preset error range.

[0073] Layered proxy playback representation is an approximate playback representation generated for a single object, which maintains its key performance while simplifying non-critical computations, unlike whole-frame proxy which uniformly reduces the resolution of the entire frame.

[0074] As the baseline scenario throughout this article, we'll use a short video draft with a beat-matching template as an example. This draft includes a main video track, a music track, a subtitle track, and a transition track. The subtitle track has an in-frame subtitle at frame 30, the music track has a drum hit at frame 48, and the transition track has a transition with a peak frame at frame 90. Background particle effects and secondary filters are also overlaid on the video. Previewing this draft on a device with a frame rate of 60 frames per second, the estimated rendering time per frame is approximately 16.7 milliseconds. The following will use this scenario as an example to explain each step in detail. First, we need to obtain the video editing draft to get the raw information required for subsequent processing. (Refer to...) Figure 2 The video editing and preview method provided in this application includes steps S1 to S7. The following describes each step in detail using the above scenario as an example.

[0075] S1. Obtain the video editing draft. The video editing draft contains track information, object information, parameter information, and user editing behavior information within a preset time window. Track information indicates the composition of each track in the draft, object information indicates the editable objects on each track, and parameter information indicates the position, scaling, speed curve, transition parameters, and other attributes of each object. For example, in the benchmark scene of the beat-matching template, four tracks are read: the main video track, the music track, the subtitle track, and the transition track, as well as objects such as subtitles, drum hit points, transitions, and background particles. The user editing behavior information within the preset time window indicates the objects and their editing operations that the user has edited within the preset time window from the current time, such as the identifier of the most recently edited object, the type of editing operation, the editing time, and the duration of the pause. The above information is used as input for the subsequent generation of the editing intent map.

[0076] S2. Based on the video editing draft, generate an editing intent graph consisting of object nodes and relational edges. For example... Figure 3 As shown, during generation, each editable object in the draft is mapped to an object node, and the relationships between objects are mapped to relational edges. For example, in the baseline scene, the main video clip is mapped to a video clip node, the subtitle at the 30th frame inlet is mapped to a subtitle node, the drum hit point at the 48th frame is mapped to an audio rhythm node, the transition at the peak frame at the 90th frame is mapped to a transition node, and the background particle effects are mapped to effect nodes. A time alignment relationship is established between the subtitle inlet and the drum hit point because they need to be at the same moment, and a perceptual priority relationship is established between the subtitle node and the effect node because the former receives more user attention. Since the user edited the subtitle within the preset time window, a user editing association relationship is established between the subtitle node and the user focus node. In the early stages of playback when user editing behavior information is missing, initial associations can be assigned to each node according to object type. This process will be further explained later in conjunction with the cold start scenario. By generating an editing intent graph, the underlying player upgrades from only being able to see frame-by-frame pixels to being able to distinguish which objects are subtitles and which are background objects, providing a structural basis for subsequent anchor point recognition.

[0077] In some embodiments, the data structure of the edit intent graph is implemented using a graph database to facilitate the representation of complex relationships between nodes and edges and support fast adjacency queries. In other embodiments, the edit intent graph can also be implemented using data structures such as adjacency lists, adjacency matrices, relation tables, or index trees. Different data structures have their own advantages in terms of graph construction overhead and retrieval efficiency. For example, adjacency lists have a smaller footprint when there are sparse node relationships, while relation tables are easier to integrate with existing draft databases. The above data structures can all hold object nodes and relation edges, and therefore all belong to the implementation methods of edit intent graphs.

[0078] In some embodiments, object nodes include one or more of the following: video clip nodes, audio clip nodes, subtitle nodes, effects nodes, transition nodes, main area nodes, mask area nodes, lip-sync area nodes, and user focus nodes. Relationship edges include one or more of the following: temporal alignment relationships, occlusion relationships, spatial dependencies, action association relationships, perceptual priority relationships, export consistency sensitivity relationships, and user editing association relationships. Temporal alignment relationships describe the constraints that require two objects to align on the timeline, such as the alignment between a subtitle in-point and a drum beat. Occlusion and spatial dependencies describe the layering and positional relationships of objects in the frame. Perceptual priority relationships describe the relative importance of objects to the user. Through these various types of nodes and relationship edges, the editing intent graph can express the temporal, spatial, and editing relationships between objects.

[0079] In some embodiments, during the initial playback phase before user editing, since there is no user editing behavior information available, initial values ​​for anchor point classification and proxy playback level are assigned to each object based on the prior weight corresponding to the object type. For example, object types such as subtitles and transitions correspond to higher prior weights and are given higher attention by default, while object types such as background particles correspond to lower prior weights. As user editing behavior accumulates, the prior weights are gradually transitioned to weights based on editing behavior according to a time window. For example, when a user frequently adjusts a subtitle within a preset time window, the weight of the node corresponding to that subtitle increases accordingly. This method enables the differentiation of key objects and the application of layered proxies even in the initial playback phase when there is no editing behavior input, avoiding a uniform degradation of the entire frame in the initial playback phase. This is also an implementation method for determining anchor point classification.

[0080] S3. Based on the editing intent map, determine the sets of hard and soft anchor points and establish anchor point constraints. During determination, objects or moments in the map that require strict time synchronization or accurate representation are assigned to the hard anchor point set, while objects or moments that can be approximated within a preset error range are assigned to the soft anchor point set. For example, in the baseline scene, the caption in-frame at frame 30, the drum hit point at frame 48, and the transition peak frame at frame 90 are assigned to the hard anchor point set, while background particle effects and secondary filters are assigned to the soft anchor point set. The reason for distinguishing between hard and soft anchor points after establishing the map is that, with limited resources, it's impossible to guarantee equal protection for all objects; it's necessary to first determine which objects are least likely to err. When establishing anchor point constraints, the time alignment between the caption in-frame and the drum hit point is defined as a constraint, meaning they must remain at the same moment during playback. Hard anchor points require strict synchronization, while soft anchor points allow approximation within an error range. The boundary between the two is whether an error in the object will mislead the user's editing judgment. By distinguishing between hard and soft anchors, the protection targets when resources are limited are aligned with the user's editing focus.

[0081] In some embodiments, the hard anchor point set includes one or more objects or moments corresponding to subtitle in-point, subtitle out-point, audio rhythm hit point, transition peak frame, speed change point, subject boundary key point, lip-sync point, and key control point of the user's current editing object. The soft anchor point set includes one or more objects or moments corresponding to background blur area, decorative particle effects, non-critical background filters, and low-sensitivity objects not directly related to the current editing target. For example, in the baseline scene, subtitle in-point, drum hit point, and transition peak frame are classified into the hard anchor point set because they directly affect the timing, while background particle effects are classified into the soft anchor point set because even if they are approximated, they do not affect the user's judgment of the timing. The basis for classifying the above-mentioned high-sensitivity objects into the hard anchor point set is that errors in these objects will directly mislead the user's judgment of subtitle timing, timing, or transition effect.

[0082] In some embodiments, the division between hard and soft anchor sets is determined based on one or more of the following: object type, user editing behavior, object visual salience, audio rhythm relevance, and export consistency sensitivity level. For example, the object types corresponding to subtitles and transitions are inherently more sensitive; objects edited by the user within a preset time window are more likely to be included in the hard anchor set due to high user attention; and objects strongly related to music beats are more likely to be included in the hard anchor set due to beat-matching requirements. Combinations of different criteria will affect the final division result. For example, objects that simultaneously meet both the criteria of object type sensitivity and recent editing have a higher priority for being classified into the hard anchor set. The aforementioned multi-dimensional criteria are also an implementation method for dividing hard and soft anchor sets.

[0083] S4. Generate object-level hierarchical proxy playback representations for at least some object nodes in the editing intent map. Unlike whole-frame proxies that uniformly reduce the resolution of the entire frame, this step generates approximate representations for individual objects, preserving their key features while simplifying non-critical calculations. For example, in the baseline scene, a proxy is generated for subtitles that retains the text content, entry point timing, and position while simplifying shadows, outlines, and glows; a proxy is generated for transitions that retains the start and end points and the peak frame at frame 90 while simplifying intermediate particle calculations; and a proxy with a higher degree of simplification is generated for background particle effects. When generating proxy playback representations, proxies are generated for key object nodes in the map and object nodes related to the current editing target. Objects with very low computational overhead can be directly preserved in their original playback. Through object-level approximation, the most critical features of each object are preserved while reducing computational overhead, achieving a balance between smoothness and realism.

[0084] In some embodiments, the hierarchical proxy playback representation generated by S4 includes one or more of temporal proxies, geometric proxies, edge proxies, color proxies, and special effects proxies. For objects corresponding to hard anchor point sets, proxies that maintain key performance characteristics are generated; for objects corresponding to soft anchor point sets, low-level proxies are generated. Specifically, temporal proxies maintain the object's entry and exit times and key time boundaries; geometric proxies maintain the object's position, scaling, rotation, and occlusion relationships; edge proxies maintain the subject outline, mask boundaries, and keying edges; color proxies maintain tone and exposure trends; and special effects proxies maintain the special effects trigger times and main visual performance while simplifying intermediate calculations. In a quantifiable proxy level classification method, proxy levels are divided into levels 0 to 3. Level 0 is the original playback without simplification; Level 1 is a high-fidelity proxy with only a few simplified non-critical calculations; Level 2 is a proxy with moderate simplification of non-critical calculations; and Level 3 is the lowest-level proxy with the highest degree of simplification. For example, Level 1 proxies are generated for subtitles and transition objects corresponding to the set of hard anchor points to maintain key performance characteristics, while Level 3 proxies are generated for background particle objects corresponding to the set of soft anchor points. The proxy category and proxy level are selected differently based on the anchor point affiliation of the object, ensuring that the authenticity of key objects is prioritized.

[0085] S5. Obtain the device resource status. Under the premise of satisfying the resource constraints of the device resource status and the timing alignment constraints of the hard anchor point set, determine the target playback strategy combination that minimizes the preview deviation cost based on the anchor point constraint relationship and the hierarchical proxy playback expression. The target playback strategy combination indicates whether each object in the current playback window should use the original playback or a proxy playback of a certain level. When determining the strategy combination, the device resource not being exceeded and the timing alignment of hard anchor points are taken as two hard constraints. Under the premise that both are satisfied, the combination that minimizes the sum of preview deviation costs is selected from the optional proxy levels of each object. For example, in the benchmark scene, the device graphics processor utilization reaches 85%, and the single-frame processing time for three consecutive frames reaches 28 milliseconds, exceeding the single-frame budget of 16.7 milliseconds, entering the budget-limited state. At this time, the target playback strategy combination obtained is: use level 0 or level 1 proxy for the subtitles, drum hit points, and transition objects corresponding to the hard anchor point set to maintain timing and key performance, and use level 3 proxy for the background particles and secondary filters corresponding to the soft anchor point set. Compared to a uniform downgrade based on a fixed level, the replay strategy is dynamically solved based on resource status and deviation cost, making the replay strategy environmentally adaptive.

[0086] In some embodiments, device resource status includes one or more of the following: CPU utilization, GPU utilization, neural network processor utilization, memory availability, temperature status, battery status, single-frame processing time, and historical frame latency. Single-frame processing time and historical frame latency reflect the current and recent rendering load, while temperature status and battery status reflect the device's continuous power supply and heat dissipation capabilities. For example, in a baseline scene, GPU utilization and single-frame processing time together indicate that the device has entered a budget-constrained state. These resource statuses provide resource constraint inputs for solving the target playback strategy combination.

[0087] In some embodiments, the target replay strategy combination is determined through one or more of heuristic search, dynamic programming, constraint optimization, and near-optimal selection. The above describes a solution method aimed at minimizing bias cost. In other embodiments, the solution method can be flexibly selected based on the problem size and real-time requirements. For example, when the number of objects in the current replay window is small, constraint optimization is used to obtain a better strategy combination; when the number of objects is large and real-time requirements are high, heuristic search or near-optimal selection is used to shorten the solution time. The above solution methods can also determine the target replay strategy combination while satisfying resource constraints and hard anchor timing constraints; therefore, they all belong to implementation methods for solving the target replay strategy combination.

[0088] In one exemplary embodiment, the preview bias cost is quantified as follows. For objects in the edit intent map... In its adoption of proxy playback level At that time, the preview deviation cost of the object The result is obtained by normalizing and synthesizing the various deviations, including time deviation, spatial deviation, boundary deviation, color deviation, and audio-visual synchronization deviation, and then weighting them according to the anchor point affiliation of the object. ,in, For object anchor weights, For object At the agent replay level The combined deviation is the normalized result of the deviations of each component. Anchor point weights are determined based on the object's anchor point affiliation; objects corresponding to hard anchor point sets receive larger weights, while objects corresponding to soft anchor point sets receive smaller weights. For example, objects with hard anchor points receive... soft anchor point object take The synthesis bias increases with the agent replay level. For example, the synthesis bias corresponds to agent replay levels 0 to 3. The values ​​are 0, 0.1, 0.25, and 0.5 respectively. Therefore, hard anchor objects, due to their higher weight, are sensitive to increases in proxy playback level, while soft anchor objects, due to their lower weight, are insensitive to increases in proxy playback level. The preview bias cost of the current playback window is the sum of the preview bias costs of all objects within the window, i.e. ,in, For object The assigned agent replay level.

[0089] The target replay strategy combination is based on preview bias cost under two hard constraints. The goal is to minimize the resource consumption. The first hard constraint is a resource constraint, which states that the sum of the single-frame rendering times of all objects within the current playback window at their assigned proxy playback levels must not exceed the single-frame rendering time budget. , represented as ,in, For object At the agent replay level The first constraint is the single-frame rendering time. The second hard constraint is the hard anchor point timing alignment constraint, which means that the timing deviation of the objects corresponding to the set of hard anchor points does not exceed a preset timing deviation threshold. Since the higher the proxy playback level, the greater the timing deviation, this constraint is equivalent to limiting the proxy playback level of the hard anchor point objects to no more than the highest level that meets the timing deviation threshold. In an exemplary solution process, a cost-effective heuristic search is used to determine the target playback strategy combination, including: initializing the proxy playback level of each object in the window to level 0; when the sum of the single-frame rendering times of each object exceeds the single-frame rendering time budget. At that time, for each object that has not yet reached the upgrade limit, calculate the reduction in single-frame rendering time resulting from increasing its proxy playback level by one level. Increase in cost of deviation from preview Select ratio The largest object increases its proxy playback level by one level, with the upgrade cap for hard anchor objects being the highest level that meets the timing deviation threshold, and the upgrade cap for soft anchor objects being the highest proxy playback level; this upgrade process is repeated until the sum of the rendering times for each object per frame does not exceed [a certain value]. Alternatively, all objects may reach their upgrade cap; the final replay level of each object's proxy will be output as the target replay strategy combination.

[0090] The above solution process is illustrated using the aforementioned time-lapse template scenario as an example. Assume the main video clip in the current playback window maintains its original playback, with a single-frame rendering time of 8.0 milliseconds. The single-frame rendering times for the four objects—subtitles, transitions, background particle effects, and secondary filters—at proxy playback levels 0 to 3 are as follows: subtitles 3.0 ms, 2.2 ms, 1.6 ms, and 1.0 ms; transitions 7.0 ms, 4.5 ms, 3.0 ms, and 2.0 ms; background particle effects 6.0 ms, 4.0 ms, 2.5 ms, and 1.2 ms; and secondary filters 4.0 ms, 2.6 ms, 1.6 ms, and 0.8 ms. When all objects are initially at level 0, the sum of their single-frame rendering times is 28.0 milliseconds, exceeding the single-frame rendering time budget of 16.7 milliseconds. Subtitles and transitions are hard anchor objects, and their timing deviation cannot exceed the timing deviation threshold of 2 frames when the proxy playback level does not exceed level 1. Therefore, the upgrade limit is level 1. Background particle effects and secondary filters are soft anchor objects, and their upgrade limit is level 3. Upgrade sequentially according to cost-effectiveness: first, gradually upgrade background particle effects and secondary filters to level 3, then upgrade transitions to level 1, and finally upgrade subtitles to level 1. At this point, the sum of the rendering time per frame is... Milliseconds equal the single-frame rendering time budget; the solution is complete. The final target playback strategy combination is: subtitles and transitions use Level 1 proxy, background particle effects and secondary filters use Level 3 proxy, and the corresponding preview deviation cost is... This solution achieves the minimum preview deviation cost while satisfying resource constraints and hard anchor timing alignment constraints. This result is consistent with the aforementioned strategy of using level 0 or level 1 proxy for hard anchor objects and level 3 proxy for soft anchor objects.

[0091] In some embodiments, when the number of object nodes in the editing intent graph increases, fine-grained solving is performed only on object nodes within the current playback window and their adjacent nodes. For object nodes outside the playback window, graph pruning or block approximation is performed while preserving their hard anchor constraints. When the draft is a multi-material montage containing a large number of fragments, the number of graph nodes and relation edges increases accordingly. If fine-grained solving is performed on all nodes, the solution time will increase, affecting real-time performance. For example, the proxy level is solved one by one for object nodes within the current playback window and its adjacent windows. Object nodes far from the current playback position are pruned first, preserving only their hard anchor constraints, and then solved when they enter the vicinity of the window. This method allows the real-time performance of the solution to be maintained even with very large drafts, and it also belongs to the implementation method of determining the combination of target playback strategies.

[0092] S6. Under a unified master clock, perform original playback or proxy playback on objects in the current playback window according to the target playback strategy combination, control the timing alignment of multi-track objects, and output the preview results. The unified master clock provides a unified time reference for multi-track objects such as video, audio, subtitles, transitions, and effects. Even if some objects use proxy playback, all track objects still advance according to the same clock. For example, in the baseline scene, the subtitle in point of frame 30 on the subtitle track and the drum hit point of frame 48 on the music track are aligned with the unified master clock. Although the background particles use level 3 proxy, their presentation time is still anchored to the same clock, thus avoiding the subtitles being half a beat slower than the sound after being downgraded, which would cause the subtitles and drums to be misaligned. Through the unified master clock, the timing alignment offset of objects corresponding to hard anchor points using proxy playback does not exceed a preset threshold, and the timing of critical objects does not drift beyond the threshold.

[0093] S7. Monitor the preview deviation of the preview result relative to the exported baseline expression, and perform local correction on the object corresponding to the preview deviation if the preview deviation exceeds the preset deviation threshold.

[0094] Reference Figure 4 Here, the exported baseline expression is the baseline result obtained by exporting and rendering the object, and the preview deviation is the degree of deviation of the preview result relative to this baseline. The reason for monitoring deviation after playback output is that proxy playback is an approximation of the original playback, and it is necessary to continuously evaluate whether the approximation introduces unacceptable deviations in key objects. During monitoring, the preview result is compared with the exported baseline expression object by object to obtain the preview deviation of each object; when the preview deviation of an object exceeds a preset deviation threshold, only local correction is performed on that object, rather than a uniform adjustment of the entire frame. For example, in the baseline scene, the timing deviation of the peak transition frame at frame 90 reaches 3 frames, exceeding the preset deviation threshold of 2 frames. Therefore, only the transition object is triggered with short-window pre-calculation and anchor point correction is performed, while the rest of the image remains unchanged. By performing closed-loop correction only on key objects, the key differences between the preview result and the exported result are continuously controlled.

[0095] In some embodiments, the monitoring of preview deviation in S7 is based on a prediction benchmark of historical keyframe features or a preset error model. In response to the cumulative deviation of an object corresponding to a set of hard anchor points exceeding a preset cumulative deviation threshold across multiple consecutive frames, a short-window true recalculation is triggered for the object corresponding to the set of hard anchor points, and the derived benchmark expression is updated with the result of the short-window true recalculation. Regular monitoring uses a prediction benchmark constructed from historical keyframe features or a preset error model to avoid the high computational overhead of generating a true derived benchmark for each frame. Only when an object associated with a hard anchor point continuously deviates is a true benchmark for that object generated at a higher cost. For example, if a subtitle object's cumulative deviation reaches 4 frames within 5 consecutive frames, exceeding the preset cumulative deviation threshold of 3 frames, a short-window true recalculation of 8 frames is triggered for that subtitle object, and the derived benchmark expression for that object is updated with the recalculation result. This hierarchical approach ensures that the high cost of true recalculation is only spent on objects associated with hard anchor points that continuously deviate, aligning the high-fidelity monitoring budget with key editing intentions.

[0096] In some embodiments, in S7, in response to multiple objects simultaneously triggering short window real recalculation or local correction, multiple objects are sorted according to a composite sorting key consisting of the priority of the object corresponding to the hard anchor point set and the preview deviation cost, and local correction is performed serially according to the sorting result, with the resource reserve reserved for the current playback window using the target playback strategy combination as the correction quota; in response to the exhaustion of the correction quota, objects that have not undergone local correction are postponed to the next playback window, and the sorting weight of objects that have not undergone local correction is increased.

[0097] In one exemplary embodiment, the composite sorting key is configured as follows. For objects that simultaneously trigger short-window true recalculation or local correction... Its compound sort key ,in, For object Hard anchor point priority, For object The normalized current preview bias cost, and This is a preset weight. The hard anchor point priority of objects corresponding to the hard anchor point set is set to a larger value, making hard anchor point objects generally prioritized over soft anchor point objects in the sorting process. The preview deviation cost term is used to further sort objects with the same hard anchor point priority according to their current deviation degree. For example, taking... , Both subtitles and transitions are hard anchor point objects with a priority of 10. The normalization cost for the current preview of subtitles is 0.8, and for transitions it is 0.5. Therefore, the composite sorting key for the subtitles... Composite sort keys for transitions The compound sorting key for subtitles is relatively large, so subtitle correction is performed first.

[0098] For example, when subtitle objects and transition objects trigger real recalculation simultaneously within the same playback window, they are sorted according to their hard anchor point priority and deviation cost, with the object at the top of the sorting being corrected first. When the resource margin reserved for correction in this window is full, the objects that have not yet been corrected are postponed to the next playback window for processing, and their sorting weight is increased to prevent them from not being corrected for a long time. By sharing the target playback strategy and the reserved resource margin, concurrent correction will not exceed the established resource constraints and cause new stuttering.

[0099] In some embodiments, a hysteresis band and a minimum hold duration are applied to the playback strategy switching and local correction of objects corresponding to the soft anchor point set, while no hysteresis band is applied to objects corresponding to the hard anchor point set. The threshold of the hysteresis band is adaptively adjusted according to the device resource status. A hysteresis band consisting of upper and lower thresholds for switching is set for soft anchor point objects, and a minimum hold duration is set to prevent frequent changes in the proxy level when resource levels fluctuate critically. For example, for background particle objects, a hysteresis band based on the graphics processor utilization rate is set to a lower threshold of 75% and an upper threshold of 85%, with a minimum hold duration of 30 frames. When the utilization rate fluctuates within this range, the proxy level of the background particles is not switched. However, no hysteresis is applied to hard anchor point objects such as subtitles to ensure their timely protection. The threshold of the hysteresis band is adaptively adjusted according to the device resource status, such as historical frame delay. This anchor-aware non-uniform hysteresis suppresses strategy jitter in soft anchor objects while ensuring timely protection of hard anchor objects.

[0100] In some embodiments, preview deviation includes one or more of the following: temporal deviation, spatial deviation, boundary deviation, color deviation, audio-visual synchronization deviation, and cumulative deviation across multiple frames. Temporal deviation indicates the deviation of the object's presentation time relative to a reference; boundary deviation indicates the deviation of the subject's outline or mask edge relative to a reference; and cumulative deviation across multiple frames indicates the cumulative amount of deviation of the object over multiple frames. For example, in a reference scene, the timing deviation of the transition peak frame is considered a temporal deviation, and the cumulative deviation of the subtitles over multiple frames is considered a cumulative deviation across multiple frames. These multiple dimensions collectively characterize the deviation of the preview result relative to the derived reference expression, providing detailed criteria for triggering local correction.

[0101] In some embodiments, local correction includes one or more of the following: increasing the proxy playback level of the object corresponding to the preview deviation, increasing the priority of the object corresponding to the preview deviation, triggering keyframe recalculation, triggering short window precalculation, and performing anchor point correction. Appropriate correction actions are selected for different types of deviations. For example, anchor point correction is performed on transition objects with excessive time deviations to bring their timing back to the reference time, and the edge proxy level of main objects with excessive boundary deviations is increased to restore contour accuracy. By selecting the corresponding correction action for the deviation type, the correction only applies to key objects and does not uniformly adjust the entire frame.

[0102] In some embodiments, the derived benchmark representation is determined based on one or more of the following: benchmark results generated by the exported rendering pipeline, historical keyframe features, local region features, and a preset error model. The above describes methods using historical keyframe features or preset error models as prediction benchmarks. In other embodiments, when a higher precision benchmark is required, the exported rendering pipeline can generate benchmark results for the target object, or a benchmark can be constructed based on the local region features where the object is located. Different benchmark sources have their own suitability in terms of accuracy and computational cost; for example, preset error models are low-cost but have limited accuracy, while exported rendering pipelines offer high accuracy but are more expensive. The aforementioned benchmark sources can also support the monitoring of preview deviations, and therefore all belong to the implementation methods for determining the derived benchmark representation.

[0103] In some embodiments, the method of this application further includes recording the editing intent graph, anchor point constraint relationships, target playback strategy combination, and local correction actions used in this playback, generating preview description data. This preview description data is used to reuse the target playback strategy combination when the same video editing draft is reopened. During playback, the graph state, anchor point constraint information, the adopted strategy combination, and the executed correction actions are recorded as preview description data. For example, in a baseline scenario, the partitioning results of the hard anchor point set, the proxy level used for subtitles and transitions, and the anchor point correction performed on the transitions are recorded. When the user reopens the same draft, the target playback strategy combination in this preview description data is directly reused without needing to re-solve it from scratch. By recording and reusing preview description data, the reproducibility of playback decisions is improved while avoiding repeated solutions.

[0104] In some embodiments, when restoring a video editing draft across devices, preview description data is reused. During reuse, the anchor point constraints corresponding to the hard anchor point set remain unchanged, and only the proxy playback level of the objects corresponding to the soft anchor point set is scaled according to the device resource status of the target device, or the target playback strategy combination is re-determined based on the anchor point constraints corresponding to the hard anchor point set. When the computing power of the source device and the target device differs, directly reusing the strategy combination of the source device may still result in stuttering on the target device with lower computing power. For example, if the background particles on the source device use level 1 proxy, when restoring to the target device with lower computing power, the anchor point constraints of hard anchor point objects such as subtitles and transitions remain unchanged, and only the proxy level of soft anchor point objects such as background particles is scaled from level 1 to level 3; when the scaling is insufficient for adaptation, the target playback strategy combination is re-solved based on the unchanged hard anchor point constraints. This selective scaling ensures that key editing intentions remain consistent across different devices, while non-critical content adapts to device capabilities.

[0105] This application also provides an electronic device including a processor and a memory. The memory stores an application program, which, when executed by the processor, implements the aforementioned video editing preview method based on editing intent graphs and anchor point constraints. This application also provides a computer-readable storage medium storing an application program. When the application program is loaded and executed by a processor, it implements the aforementioned video editing preview method based on editing intent graphs and anchor point constraints. Those skilled in the art will understand that all or part of the above methods can be implemented by hardware related to application program instructions. This program can be stored in a computer-readable storage medium and, when executed, implements the aforementioned steps.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0107] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video editing preview method based on editing intent graph and anchor point constraints, characterized in that, include: S1. Obtain a video editing draft, which includes track information, object information, parameter information, and user editing behavior information within a preset time window; S2. Based on the video editing draft, generate an editing intent graph consisting of object nodes and relation edges; S3. Based on the edit intent map, determine the set of hard anchor points and the set of soft anchor points, and establish anchor point constraint relationships; S4. For at least a portion of the object nodes in the edit intent graph, generate object-level hierarchical proxy playback representations; S5. Obtain the device resource status, and under the premise of satisfying the resource constraints of the device resource status and the timing alignment constraints of the hard anchor point set, determine the target playback strategy combination that minimizes the preview deviation cost based on the anchor point constraint relationship and the hierarchical proxy playback expression. S6. Under a unified master clock, perform original playback or proxy playback on the objects in the current playback window according to the target playback strategy combination, perform timing alignment control on multi-track objects and output preview results; S7. Monitor the preview deviation of the preview result relative to the exported baseline expression. In response to the preview deviation exceeding a preset deviation threshold, perform local correction on the object corresponding to the preview deviation. The monitoring of the preview deviation is based on the prediction baseline of historical keyframe features or a preset error model. In response to the cumulative deviation of the object corresponding to the hard anchor point set exceeding a preset cumulative deviation threshold for multiple consecutive frames, trigger a short window real recalculation for the object corresponding to the hard anchor point set, and update the exported baseline expression with the result of the short window real recalculation.

2. The video editing and preview method according to claim 1, characterized in that, The set of hard anchor points includes objects or moments corresponding to at least one of the following: subtitle in point, subtitle out point, audio rhythm hit point, transition peak frame, speed change point, main body boundary key point, lip-sync point, and key control point of the user's current editing object; the set of soft anchor points includes objects or moments corresponding to at least one of the following: background blur area, decorative particle effect, non-critical background filter, and low-sensitivity object not directly related to the current editing target.

3. The video editing and preview method according to claim 2, characterized in that, In S4, the hierarchical proxy playback expression includes at least one of temporal proxy, geometric proxy, edge proxy, color proxy and special effects proxy, and generates a proxy that maintains key performance for the object corresponding to the hard anchor point set, and generates a low-level proxy for the object corresponding to the soft anchor point set.

4. The video editing and preview method according to claim 3, characterized in that, In step S7, in response to multiple objects simultaneously triggering the short window real recalculation or the local correction, the multiple objects are sorted according to the composite sorting key formed by the priority of the objects corresponding to the hard anchor point set and the preview deviation cost, and the local correction is executed serially according to the sorting result, with the resource reserve reserved for the current playback window by the target playback strategy combination as the correction quota; in response to the exhaustion of the correction quota, the objects that have not undergone the local correction are postponed to the next playback window, and the sorting weight of the objects that have not undergone the local correction is increased.

5. The video editing and preview method according to claim 4, characterized in that, The playback strategy for the objects corresponding to the soft anchor point set is switched, and the local correction is subject to a hysteresis band and a minimum hold time. The objects corresponding to the hard anchor point set are not subject to the hysteresis band, and the threshold of the hysteresis band is adaptively adjusted according to the device resource status.

6. The video editing and preview method according to claim 1, characterized in that, Also includes: Record the editing intent map, anchor point constraints, target playback strategy combination, and local correction actions used in this playback, and generate preview description data; the preview description data is used to reuse the target playback strategy combination when the same video editing draft is opened again.

7. The video editing and preview method according to claim 6, characterized in that, When restoring the video editing draft across devices, the preview description data is reused. During reuse, the anchor point constraint relationship corresponding to the set of hard anchor points remains unchanged. Only the proxy playback level of the object corresponding to the set of soft anchor points is scaled according to the device resource status of the target device, or the target playback strategy combination is re-determined based on the anchor point constraint relationship corresponding to the set of hard anchor points.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program that, when executed by the processor, implements the video editing and previewing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an application program that, when executed by a processor, implements the video editing and previewing method according to any one of claims 1 to 7.

Images