A video editing method, device and electronic equipment

Abstract

The application provides a video editing method and device and electronic equipment, and applies to the technical field of multimedia data processing. The application decomposes an editing task submitted by a user into multiple ordered subtasks, and adds a decoding task and an encoding task at the beginning and the end. The decoding task converts a video into an original frame sequence and passes the original frame sequence to the first subtask. The subtasks are iteratively executed in sequence from back to front. Frame data is directly passed between the subtasks through input and output buffers without additional copying. The encoding task encodes the processed frame sequence to generate a target video, so as to avoid the accumulation of quantization errors caused by repeated encoding and decoding, thereby saving hardware resources while reducing picture quality loss.

Description

Technical Field

[0001] This invention relates to the field of multimedia data processing technology, and in particular to a video editing method, apparatus, and electronic device. Background Technology

[0002] Currently, the diffusion model architecture centered on DiT (Diffusion Transformer) has been widely used in the video editing field, and mainstream solutions have verified its quality advantages.

[0003] However, existing technologies still follow a fragmented paradigm of "single region, single task, single round of inference": each new editing step involves the complete execution of the entire process of "upload → decode → inference → encoding → download." This leads to two major problems: First, when multiple overlapping regions in the same frame require sequential editing by multiple algorithms, each algorithm executes the complete process independently, unable to share encoding and decoding operations, wasting bandwidth and computing resources; second, when multiple scattered or overlapping segments in the same video are edited by different tasks, repeated encoding and decoding cause cascading compression distortion, and under H.264 / H.265 encoding, quantization errors are amplified step by step, making the image "blurrier with each cut."

[0004] Therefore, how to enable multiple video editing subtasks to share video encoding and decoding operations and reduce the loss of video editing quality has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a video editing method, apparatus, and electronic device that overcomes or at least partially solves the above problems, the technical solution of which is as follows:

[0006] A video editing method, comprising:

[0007] Obtain a video editing task submitted by a user, wherein the video editing task includes a video to be edited and at least one editing operation;

[0008] The editing operation is divided into multiple ordered subtasks, and decoding and encoding tasks are inserted at the beginning and end respectively to obtain a subtask queue.

[0009] The decoding task is executed to decode the video to be edited, and the resulting original video frame sequence is passed to the first subtask.

[0010] In the opposite direction to the order of the subtask queue, each subtask is scheduled to be executed iteratively for editing, and the frame buffers edited by each subtask are passed forward level by level until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence.

[0011] The encoding task is performed to encode the received final output frame sequence and output the target video.

[0012] Optionally, dividing the editing operation into multiple ordered subtasks and inserting decoding and encoding tasks at the beginning and end respectively to obtain a subtask queue includes:

[0013] When there are multiple editing operations in different regions on the same consecutive frame segment of the video to be edited, the editing operations are divided into multiple sub-tasks with a sequential order according to the spatial dependency between the regions;

[0014] When editing operations exist on different consecutive frame segments of the video to be edited, the editing operation corresponding to each segment is divided into an independent subtask;

[0015] Sort all the subtasks obtained after the division in reverse order of their dependencies to obtain ordered subtasks.

[0016] The decoding task is inserted before the first sorted subtask, and the encoding task is inserted after the last subtask, to generate a subtask queue.

[0017] Optionally, the step of sequentially scheduling each subtask for iterative editing in the opposite direction to the subtask queue order, and passing the edited frame buffers of each subtask forward level by level, until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence, includes:

[0018] The frame data in the input frame buffer of each subtask in the subtask queue is checked sequentially from back to front to see if it meets the execution conditions. Subtasks that meet the execution conditions are designated as the current scheduled subtasks: frame data is loaded from the input frame buffer of the context manager to which the current scheduled subtask belongs; frames belonging to the current editing area are edited and written to the output frame buffer; other frames are directly passed through to the output frame buffer; the output frame buffer of the next subtask is used as the input frame buffer of the previous subtask, and so on, until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence.

[0019] Optionally, the step of sequentially checking whether the frame data in the input frame buffer of each subtask in the subtask queue meets the execution conditions in reverse order includes:

[0020] Starting from the last subtask in the subtask queue, traverse each subtask sequentially in the direction of the first subtask;

[0021] For the currently traversed subtask, obtain the number of consecutive frames already cached in the input frame buffer corresponding to the subtask;

[0022] The number of consecutive frames is compared with the frame block length required for a single processing of the subtask. If the number of consecutive frames is greater than or equal to the frame block length, the execution condition is determined to be met.

[0023] If the number of consecutive frames is less than the frame block length, then continue detecting the previous subtask.

[0024] Optionally, the step of loading frame data from the input frame buffer of the context manager to which the currently scheduled subtask belongs, editing the frames belonging to the current editing region in the frame data and writing them to the output frame buffer, and directly passing other frames to the output frame buffer includes:

[0025] The task executor corresponding to the current scheduled subtask loads a frame block from the input frame buffer of its context manager. The length of the frame block is equal to the length of the frame block required for a single processing of the current scheduled subtask, and an overlapping frame is set between two adjacent loaded frame blocks.

[0026] An editing operation is performed on each frame in the frame block. For frames in the frame block that belong to the editing region, an edited frame is generated based on the editing parameters carried by the current scheduling subtask. For frames that do not belong to the editing region, the input frame is directly used as the output frame.

[0027] The edited frame block is written to the output frame buffer, and the processed frame data in the input frame buffer is released.

[0028] Optionally, the method further includes:

[0029] After the current scheduled subtask has completed processing all frames in its input frame buffer, the current scheduled subtask is marked as completed and removed from the detection order.

[0030] The encoding task is triggered when all subtasks are marked as completed and the decoding task has no more video frames to decode.

[0031] Optionally, the step of performing the decoding task, decoding the video to be edited, and passing the resulting original video frame sequence to the first subtask, includes:

[0032] The decoding task is executed by calling the video decoder to open the video file to be edited and recording the position of the currently decoded frame.

[0033] According to the frame block length required for the first subtask, the corresponding number of video frames are continuously decoded from the video to be edited to obtain the original video frame sequence.

[0034] The original video frame sequence is written into the input frame buffer of the context manager of the first subtask;

[0035] After the first subtask completes the processing of a frame block, the decoding task continues to decode the next frame block starting from the already decoded frame position, until all frames of the video to be edited are decoded.

[0036] Optionally, performing the encoding task, encoding the received final output frame sequence, and outputting the target video includes:

[0037] Obtain the encoding parameters of the video to be edited;

[0038] The final output frame sequence is encoded according to the encoding parameters, the encoded video stream is written to a file, and the target video is output.

[0039] A video editing device includes: a video editing task acquisition unit, a subtask queue acquisition unit, a decoding task execution unit, a task scheduling unit, and an encoding task execution unit;

[0040] The video editing task obtaining unit is used to obtain the video editing task submitted by the user, wherein the video editing task includes the video to be edited and at least one editing operation;

[0041] The subtask queue obtaining unit is used to divide the editing operation into multiple ordered subtasks, and insert decoding tasks and encoding tasks at the beginning and end respectively to obtain a subtask queue.

[0042] The decoding task execution unit is used to execute the decoding task, decode the video to be edited, and pass the resulting original video frame sequence to the first subtask;

[0043] The task scheduling unit is used to schedule each subtask to be edited iteratively in the opposite direction to the order of the subtask queue, and pass the edited frame buffer of each subtask forward level by level until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence.

[0044] The encoding task execution unit is used to execute the encoding task, encode the received final output frame sequence, and output the target video.

[0045] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the video editing method.

[0046] By employing the above technical solution, the present invention provides a video editing method, apparatus, and electronic device that obtains a video editing task submitted by a user, wherein the video editing task includes a video to be edited and at least one editing operation; divides the editing operation into multiple ordered sub-tasks, and inserts decoding and encoding tasks at the beginning and end respectively to obtain a sub-task queue; executes the decoding task to decode the video to be edited, and passes the resulting original video frame sequence to the first sub-task; schedules each sub-task to iteratively execute editing in the opposite direction to the order of the sub-task queue, and passes the frame buffers edited by each sub-task forward level by level until the output frame buffer of the first sub-task is passed as the final output frame sequence to the encoding task; executes the encoding task to encode the received final output frame sequence and outputs the target video. The present invention, by dividing multiple editing operations into ordered sub-tasks and performing decoding and encoding only once at the beginning and end, and sharing intermediate frame data between sub-tasks through zero-copy transmission of input and output frame buffers, avoids the accumulation of quantization errors caused by repeated encoding and decoding, thereby reducing video editing quality loss while reducing hardware resource consumption.

[0047] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0048] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0049] Figure 1 A logic block diagram of the video editing reasoning process provided in an embodiment of the present invention is shown;

[0050] Figure 2 A flowchart illustrating one embodiment of the video editing method provided by this invention is shown.

[0051] Figure 3 The diagram illustrates a specific implementation of step S210 in the video editing method provided in this embodiment of the invention.

[0052] Figure 4 A logic block diagram of the scheduling pipeline provided in an embodiment of the present invention is shown;

[0053] Figure 5 A schematic diagram of the structure of the video editing device provided in an embodiment of the present invention is shown;

[0054] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0055] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0056] Currently, while advanced diffusion models such as DiT (Diffusion Transformer) are widely used in the video editing field, significant limitations remain in task processing paradigms. Existing technologies generally follow a fragmented model of "single region, single task, single round of inference," meaning that each new edit requires an independent execution of the entire process of "upload—decode—inference—encode—download." This model leads to two prominent problems: First, when processing multi-region editing within a single frame, multiple algorithms need to edit different regions of the same frame sequentially. However, existing solutions encapsulate each algorithm as an independent API call, requiring repeated encoding / decoding and file transfer for each call, resulting in wasted computational and bandwidth resources and image quality loss due to multiple compressions. Second, when dealing with multi-scene video editing, different editing tasks often target scattered and potentially overlapping segments within the same video, with dependencies between tasks. Existing technologies repeatedly execute encoding / decoding and file transfer in different tasks, not only wasting resources but also causing cascading distortion due to repeated compression. Especially under H.264 / H.265 encoding, quantization errors accumulate step by step, leading to a continuous decline in image quality.

[0057] Based on this, this embodiment of the invention provides a video editing method. First, the user-submitted editing task is decomposed into multiple ordered subtasks, with decoding and encoding tasks added at the beginning and end, respectively. The decoding task converts the video into a raw frame sequence and passes it to the first subtask. Subsequently, the input buffers of each subtask are checked from back to front. When the frame data meets the conditions, the subtask is scheduled to perform editing: frames in a specified area are edited and output, while the remaining frames are directly passed through. Frame data is directly passed between subtasks through input and output buffers, without the need for additional copying. Finally, the encoding task encodes the processed frame sequence to generate the target video. It is evident that this method avoids the accumulation of quantization errors caused by repeated encoding and decoding, thereby saving hardware resources while reducing image quality loss.

[0058] refer to Figure 1The logical block diagram of the video editing inference process shown in this embodiment of the invention can be implemented collaboratively by the following core modules: a video decoder, used to stream decode the video to be edited and record the position of the currently decoded frame to ensure that a specified number of frames can be decoded at a specified time; a video encoder, used to receive sequentially input video frames and encode them into the final video according to a specific video format; a context manager, used to manage the input frame buffer and output frame buffer of each subtask, where each subtask corresponds to one context manager, the frames in the buffer are identified by absolute frame sequence numbers, the output frame buffer of the next subtask is used as the input frame buffer of the previous subtask, and frames in non-editing areas are directly passed through with zero copy; a task executor, used to manage the runtime context of a single subtask, perform editing operations on the segmented video frames, and set overlapping frames between adjacent segments to ensure consistency; and a task scheduler, used to divide the user-submitted editing operations into multiple ordered subtasks and control the execution order of encoding / decoding and task executors.

[0059] like Figure 2 The diagram shows a flowchart of one embodiment of the video editing method provided by this invention. The method may include:

[0060] S200: Obtain a video editing task submitted by the user, wherein the video editing task includes the video to be edited and at least one editing operation.

[0061] In this context, a video editing task refers to a complete editing request submitted by the user, including the video to be edited and at least one editing operation performed on that video. This invention supports the internal splitting of a single task submission into multiple ordered subtasks for pipelined processing.

[0062] The video to be edited refers to the original video file provided by the user, which serves as the input source for the video editing task. After decoding, the video yields a sequence of original video frames. All editing operations are performed based on this original frame sequence or its derived frame sequences, and the absolute frame number of each frame remains unchanged after any number of edits.

[0063] Editing operations refer to specific modifications to video content specified by the user, such as subtitle erasure, lip-sync modification, target replacement, and style transfer. Each editing operation can correspond to a continuous segment or one or more regions of the video and may be accompanied by modal parameters such as text prompts, masks, and audio.

[0064] Specifically, embodiments of the present invention can receive video files uploaded by users through a client interface as videos to be edited, along with a set of editing instructions. Each editing instruction includes at least the target video segment (start and end frame numbers), a spatial description of the editing area (such as a mask image or polygon vertex coordinates), an identifier of the editing algorithm used, and other modal data required by the algorithm (such as text prompts, reference images, and audio tracks). The above information is assembled into a task object, the submission time and task identifier are recorded, and the object is entered into the subsequent processing queue.

[0065] S210. Divide the editing operation into multiple ordered subtasks, and insert decoding tasks and encoding tasks at the beginning and end respectively to obtain a subtask queue.

[0066] In this context, ordered subtasks refer to independent editing units generated by the task scheduler after dividing one or more editing operations according to spatial dependencies or segment positions, with a clear sequential execution order. Each subtask is only responsible for editing a single region on a specific continuous segment of the video. The output of the subsequent subtask serves as the input of the preceding subtask, and subtasks with smaller numbers are processed first.

[0067] The decoding task is a special task inserted at the head of the subtask queue by the task scheduler. It is responsible for calling the video decoder to stream the video to be edited and passing the resulting raw video frame sequence to the context manager of the first subtask. The decoding task also records the position of the currently decoded frame, allowing for continued decoding of subsequent frame blocks as needed.

[0068] The encoding task refers to a special task inserted at the end of the subtask queue by the task scheduler. It is responsible for calling the video encoder to receive the final output frame sequence and encoding it into the target video file according to a specific video format (such as H.264 / H.265).

[0069] The subtask queue refers to an ordered sequence generated by the task scheduler, consisting of: "decoding task + ordered subtasks 1 to N + encoding task". This queue defines the logical execution order of each subtask, and the scheduler dynamically schedules the execution of each subtask according to its priority from back to front.

[0070] Specifically, in this embodiment of the invention, based on the position of the frame segment to which each editing operation is performed and the spatial dependency between the editing areas, the editing operations that overlap or depend on each other can be divided into multiple sub-tasks with a sequential order. Independent editing operations performed on different consecutive frame segments can be divided into independent sub-tasks. All sub-tasks can be sorted in reverse order of execution dependency, and decoding and encoding tasks can be inserted at the beginning and end of the sorted sequence to obtain a sub-task queue.

[0071] S220. Execute the decoding task, decode the video to be edited, and pass the resulting original video frame sequence to the first subtask.

[0072] The original video frame sequence refers to the continuous frame data output by the video decoder after streaming decoding of the video to be edited. Each frame has an absolute frame number relative to the original video (which does not change due to subsequent editing). This sequence serves as the initial input for the first subtask.

[0073] Specifically, in this embodiment of the invention, a decoding task can be performed to decode the original video frame sequence from the video to be edited block by block in a streaming manner according to the frame block length required by the first subtask. After each decoding, the current frame block is written into the input frame buffer of the first subtask and the decoding progress is recorded. After the first subtask completes the processing of the current frame block, the next frame block is decoded until the entire video decoding is completed.

[0074] S230. In the opposite direction to the order of the subtask queue, schedule each subtask to perform editing iteratively, and pass the edited frame buffer of each subtask forward level by level until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence.

[0075] The final output frame sequence refers to the complete frame sequence assembled from the output frame buffer of the first subtask after all ordered subtasks have been processed sequentially. Each frame in the final output frame sequence has been processed by the required editing operations and has not undergone any intermediate encoding or compression, thus avoiding the accumulation of quantization errors.

[0076] Specifically, in this embodiment of the invention, the input frame buffer of each subtask can be checked sequentially from the tail to the head of the subtask queue to see if it has a sufficient number of frame data. If it does, the corresponding subtask is scheduled to perform a round of editing. After the editing is completed, the resulting output frame buffer is used as the input frame buffer of the previous subtask, and so on, level by level.

[0077] S240. Execute the encoding task, encode the received final output frame sequence, and output the target video.

[0078] The target video refers to the playable video file generated after the encoding task encodes the final output frame sequence. Optionally, the original encoding parameters of the video to be edited (such as format, bitrate, frame rate, and GOP size) can be reused during encoding to preserve image quality to the greatest extent possible.

[0079] Specifically, embodiments of the present invention can perform encoding tasks, encode the final output frame sequence according to the original encoding configuration of the video to be edited, and output it as the target video.

[0080] This invention provides a video editing method, comprising: obtaining a user-submitted video editing task, wherein the video editing task includes a video to be edited and at least one editing operation; dividing the editing operation into multiple ordered subtasks, and inserting decoding and encoding tasks at the beginning and end respectively to obtain a subtask queue; executing the decoding task to decode the video to be edited, and passing the resulting original video frame sequence to the first subtask; sequentially scheduling each subtask to iteratively execute editing in the opposite direction to the order of the subtask queue, and passing the edited frame buffers of each subtask forward level by level until the output frame buffer of the first subtask is passed as the final output frame sequence to the encoding task; executing the encoding task to encode the received final output frame sequence and output the target video. This invention, by dividing multiple editing operations into ordered subtasks and performing decoding and encoding only once at the beginning and end, and sharing intermediate frame data between subtasks through zero-copy transmission of input and output frame buffers, avoids the accumulation of quantization errors caused by repeated encoding and decoding, thereby reducing video editing quality loss while reducing hardware resource consumption.

[0081] Optional, based on Figure 2 The method shown is as follows: Figure 3 The diagram shows a specific implementation of step S210 in the video editing method provided in this embodiment of the invention. Step S210 may specifically include:

[0082] S300: When there are multiple editing operations in different regions on the same consecutive frame segment of the video to be edited, the editing operations are divided into multiple sub-tasks with a sequential order according to the spatial dependency between the regions.

[0083] Specifically, embodiments of the present invention can identify the frame segment range in all editing operations through a task scheduler. If the frame segments of multiple editing operations completely overlap or intersect, and these operations act on different spatial regions on the same frame image (e.g., the subtitle region and the lip-sync region), the scheduler determines the spatial dependency based on the physical logic of the editing algorithm or the order specified by the user. For example, subtitle erasure should be performed before background repair to avoid the repaired area being covered by the subtitle; or target replacement should be completed before color correction. Based on this dependency, the scheduler splits these editing operations into two or more sequentially executed subtasks and records the frame segment, editing region mask, algorithm parameters, and dependency index corresponding to each subtask. Each subtask is assigned an independent task executor, and subsequent processing is performed serially according to the dependency order.

[0084] S310. When there are editing operations on different consecutive frame segments of the video to be edited, divide the editing operation corresponding to each segment into an independent subtask.

[0085] Specifically, in this embodiment of the invention, the task scheduler can traverse all editing operations and group them according to the start and end positions of the frame segments they affect. If the frame segments covered by two editing operations have no overlap (i.e., the end frame number of one segment is less than the start frame number of another segment), they are determined to belong to different consecutive frame segments. The scheduler divides one or more editing operations attached to each segment (if there are multiple areas to edit within the same segment, they are processed first according to S300) into an independent subtask. There is no data overlap between the independent subtasks at the frame level, so there is no frame-level interdependence, and they can be processed independently. However, considering the pipeline scheduling of the entire editing process, these subtasks are still assigned an initial sorting number according to the order of the segments in the video.

[0086] S320. Sort all the subtasks obtained after the division in reverse order of their dependencies to obtain ordered subtasks.

[0087] Specifically, in this embodiment of the invention, a task scheduler can collect all generated subtasks and analyze their execution dependencies: for subtasks with sequential dependencies generated in step S300, the later-executed subtask depends on the output of the earlier-executed subtask; for independent subtasks of different segments generated in step S310, since frame segments do not overlap, theoretically there is no absolute dependency. However, in order to achieve... Figure 4 The scheduling pipeline shown is a back-to-foreign process. The task scheduler sorts all subtasks in reverse order of their dependencies; that is, the last subtask executed in the dependency chain is at the front of the queue, and the first subtask executed is at the back. For independent subtasks without dependencies, they are arranged in reverse order of the original video frame order (e.g., if segment B is after segment A, then subtask B is before subtask A). This results in an ordered queue: subtask 1 (end of the dependency chain or subsequent segment), subtask 2, ..., subtask n (beginning of the dependency chain or preceding segment). This sorting ensures that subsequent schedulers, when checking in reverse order, can correctly drive the data pipeline.

[0088] S330. Insert a decoding task before the first sorted subtask and an encoding task after the last subtask to generate a subtask queue.

[0089] Specifically, in this embodiment of the invention, after obtaining the sorted list of subtasks, a new task queue object is created through the task scheduler. First, the decoding task is inserted as the first element of the queue. The decoding task encapsulates the logic for calling the video decoder, including functions such as opening the video file to be edited, recording the decoding position, and decoding frame blocks as needed. Then, subtasks 1 to n are added to the queue in sequence. Finally, the encoding task is inserted at the end of the queue. The encoding task encapsulates the logic for calling the video encoder, including functions such as obtaining the original video encoding parameters, compressing the final frame sequence, and outputting the target video file. The final generated subtask queue is [decoding task, subtask 1, subtask 2, ..., subtask n, encoding task]. This queue serves as the input to the subsequent scheduler and runs throughout the entire video editing process.

[0090] In this embodiment of the invention, subtasks are divided according to the spatial dependency and segment independence of editing operations, and then the decoding and encoding tasks are inserted in reverse order. This allows the subtasks at the end of the dependency chain to be scheduled and processed first. Thus, on the basis of sharing a single encoding and decoding, it ensures that complex editing tasks of multiple regions and multiple segments can be executed in a pipeline without conflict. This avoids the loss of image quality caused by repeated encoding and decoding, and ensures the correctness of the editing logic.

[0091] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S230 may specifically include:

[0092] The system sequentially checks whether the frame data in the input frame buffer of each subtask in the subtask queue meets the execution conditions, starting from the end and working backwards. Subtasks that meet the execution conditions are designated as the current scheduled subtasks. The system loads frame data from the input frame buffer of the context manager to which the current scheduled subtask belongs, edits the frames in the frame data that belong to the current editing area, and writes them to the output frame buffer. Other frames are directly passed through to the output frame buffer.

[0093] The input frame buffer refers to a storage area maintained by the context manager, used to store consecutive frame data to be processed by the current subtask. Each subtask has an independent input frame buffer, and the frames in the buffer are identified by absolute frame numbers within a consecutive interval from a to b. The output frame buffer of the next subtask serves as the input frame buffer of the previous subtask, and zero-copy transfer is achieved through pointer assignment.

[0094] The execution condition refers to the criteria used to determine whether a subtask can be scheduled by the scheduler for one iteration. Specifically, it is defined as follows: the number of consecutive frames buffered in the input frame buffer of the subtask is greater than or equal to the frame block length required for a single processing iteration of the subtask. The subtask always satisfies the execution condition when the last frame of the video has entered the input buffer, until the last frame has been processed.

[0095] Frame data refers to the video frame content and its associated absolute frame number information stored in the input or output frame buffer. Each subtask loads frame data from its input frame buffer, edits it, and writes the resulting new frame data to the output frame buffer.

[0096] The current editing region refers to the video space range that the currently scheduled subtask is responsible for editing, corresponding to one or more overlapping regions within a specific continuous segment of the original video. Only frames belonging to this region need to execute the actual editing algorithm; other frames are directly passed through.

[0097] The output frame buffer refers to a storage area maintained by the context manager, used to store the output frame data after the current subtask has been processed. For frames belonging to the current editing area, the edited frame is written; for frames not belonging to the current editing area, the input frame is directly passed through. This output frame buffer is then used as the input frame buffer for the previous subtask.

[0098] Specifically, in this embodiment of the invention, the number of consecutive frames cached in the input frame buffer of each subtask can be checked sequentially from the tail to the head of the subtask queue to see if it reaches the frame block length required for a single processing of the subtask. If it does, the execution condition is determined to be met; otherwise, the previous subtask is checked. A frame block of a set length is read from the input frame buffer of the currently scheduled subtask that meets the execution condition. Each frame within the frame block that is located in the current editing area is edited according to the editing parameters of the subtask. Frames that are not in the editing area are directly passed through. The processed frame block is then written to the output frame buffer.

[0099] The output frame buffer of the next subtask is used as the input frame buffer of the previous subtask, and so on, until the output frame buffer of the first subtask is used as the final output frame sequence and passed to the encoding task.

[0100] Specifically, in the subtask queue, the output frame buffer of the next subtask (the one with the larger number) is directly used as the input frame buffer of the previous subtask (the one with the smaller number). Zero-copy passing is achieved through pointer assignment, eliminating the need to copy frame data. For example, when subtask i (i decrements from n to 2) completes one iteration and updates its output frame buffer, the scheduler assigns the address of that output frame buffer to the input frame buffer pointer of subtask i-1. Subtask i-1 reads frame data from this buffer during execution. For frames not belonging to the editing area of ​​subtask i, subtask i-1 directly passes them through during its own iteration, thus avoiding additional copying. This recursive progression continues until subtask 1 has processed all frames, at which point its output frame buffer contains the complete frame sequence edited by all subtasks. The scheduler then assembles this frame sequence into the final output frame sequence and passes the pointer to the encoding task.

[0101] In an optional embodiment of the present invention, the frame data in the input frame buffer of each subtask in the subtask queue is checked sequentially from back to front to see if it meets the execution conditions. Specifically, this may include:

[0102] Starting from the last subtask in the subtask queue, traverse each subtask sequentially in the direction of the first subtask.

[0103] Specifically, in this embodiment of the invention, the task scheduler can point the pointer to the last subtask (i.e., subtask n) in the subtask queue at the beginning of each round of detection, and then move towards subtask 1 in a decrementing index manner. Each scheduling iteration restarts the traversal from the tail of the queue, rather than continuing from the position of the last interruption, to ensure that the higher-numbered subtasks receive the highest priority for inspection.

[0104] For the currently traversed subtask, obtain the number of consecutive frames already cached in the input frame buffer corresponding to the subtask.

[0105] Specifically, in this embodiment of the invention, for the currently pointed-to subtask, the task scheduler accesses its bound context manager to read the status information of the input frame buffer. The input frame buffer stores consecutive frame blocks output after processing by the previous subtask (i.e., the subtask with the larger number), each frame carrying an original absolute frame sequence number. The context manager maintains the start and end sequence numbers of the consecutive frames already stored in the buffer, as well as the total number of frames. The scheduler directly obtains the number of currently buffered consecutive frames (e.g., from frame a to frame b, a total of b-a+1 frames) by calling the interface provided by the context manager. If the buffer is empty or has discontinuous gaps (e.g., missing several intermediate frames), the number of consecutive frames is calculated based on the longest consecutive prefix or independent consecutive segments. According to the zero-copy delivery mechanism, the output frame buffer of the next subtask always guarantees consecutive frame blocks, so discontinuities usually do not occur.

[0106] The number of consecutive frames is compared with the frame block length required for a single subtask processing. If the number of consecutive frames is greater than or equal to the frame block length, the execution condition is deemed met.

[0107] Specifically, each subtask provided in this embodiment of the invention determines the frame block length required for a single processing session (denoted as ) during initialization. This length is determined by the maximum number of input frames of the editing algorithm used by the subtask (such as the DiT model) or by user configuration, for example, a continuous sequence of 8 or 16 frames. The scheduler buffers the number of consecutive frames in the input frame buffer of the currently traversed subtask (denoted as ). ) and that length Compare. If If the current subtask meets the execution condition, the scheduler immediately stops traversing and sets this subtask as the current scheduled subtask, preparing for one iteration. If If the condition is not met, the process continues traversing the previous subtask. In a special case, if the last frame of the video is already in the input frame buffer (i.e., the buffer contains the last frame and there are no more subsequent frames), then regardless of... Has it been achieved? All conditions are determined to be met to ensure that the tail frame can be processed.

[0108] If the number of consecutive frames is less than the frame block length, then continue to detect the previous subtask.

[0109] Specifically, in this embodiment of the invention, after determining that the currently traversed subtask does not meet the execution conditions, the task scheduler moves the pointer one position towards the head of the subtask queue, i.e., from subtask i to subtask i-1. Then, the process of obtaining the number of consecutive frames and comparing them with the frame block length is repeated. If the execution conditions are still not met even after moving to subtask 1, the scheduler proceeds to the next step: attempting to load a frame block of a certain length from the original input video and provide it to the decoding task, thereby filling the input frame buffer of subtask 1. After that, a new round of traversal starts again from the last subtask. In addition, when a subtask is skipped due to insufficient input frame buffer, the scheduler does not immediately load a new frame, but first checks all the earlier subtasks to ensure that the scheduling order always prioritizes the subsequent subtasks.

[0110] This invention, through a process of detecting whether the number of consecutive frames in the input frame buffer of each subtask meets the frame block length from back to front, and only scheduling subtasks that meet the conditions to perform one round of iteration, allows the needs of the subsequent subtasks to drive the decoding and execution of the preceding subtasks step by step. This achieves frame-level pipelined processing while ensuring the editing logic order, avoiding any idle time for any subtask waiting for the entire video to be processed, and maximizing the low latency advantage brought by sharing a single encoding and decoding.

[0111] In an optional embodiment provided by this invention, frame data is loaded from the input frame buffer of the context manager to which the current scheduled subtask belongs, frames belonging to the current editing region are edited and written to the output frame buffer, and other frames are directly passed through to the output frame buffer. Specifically, this may include:

[0112] The task executor corresponding to the current scheduled subtask loads a frame block from the input frame buffer of its context manager. The length of the frame block is equal to the length of the frame block required for a single processing of the current scheduled subtask, and an overlap frame is set between two adjacent loaded frame blocks.

[0113] Specifically, the current scheduled subtask has an independent task executor. In this embodiment of the invention, the task executor first reads a frame block from the input frame buffer of its bound context manager. The length L of the frame block is equal to the length of the frame block required for a single processing run of the subtask (determined by the maximum number of input frames used in the editing algorithm, such as 8 or 16 frames). To ensure temporal consistency between blocks during long video editing (e.g., the continuity requirements of optical flow, target tracking, etc.), the task executor sets overlapping frames between adjacent loaded frame blocks. The number of overlapping frames O (e.g., 2 or 4 frames) is predefined by the editing algorithm, meaning the last O frames of the current frame block are the same as the first O frames of the next loaded frame block. If this is the first load, no prior overlap is needed; if the remaining number of frames in the input buffer is less than L and the end frame of the video has not been reached, subsequent decoding is required; if the end frame has been reached, the actual number of remaining frames is loaded. After loading, the task executor records the starting absolute frame number, ending frame number, and overlapping region boundary of this frame block.

[0114] Editing operations are performed on each frame in the frame block. For frames that belong to the editing area, the edited frame is generated based on the editing parameters carried by the current scheduled subtask. For frames that do not belong to the editing area, the input frame is directly used as the output frame.

[0115] Specifically, in this embodiment of the invention, the task executor can traverse each frame in the loaded frame block, while the absolute sequence number of the frame remains unchanged. For the current frame, it is first determined whether the frame belongs to the continuous frame segment range handled by the current subtask, and whether the pixel position of the frame falls within the editing area (e.g., determined by mask or geometric description). If it meets the requirement of belonging to the editing area, the corresponding DiT diffusion model or video editing algorithm is called according to the editing parameters carried by the subtask to generate the edited frame data. During the editing process, for overlapping frame areas, the algorithm can utilize the information of the overlapping frames to enhance temporal consistency (e.g., by interpolating or smoothing constraints using the features of the overlapping frames). If the current frame does not belong to the editing area (including not being within the target segment or outside the mask), the task executor directly uses the input frame as the output frame without performing any modifications. The pass-through operation in the non-editing area is implemented based on reference passing, eliminating the need to copy pixel data, thereby reducing memory and computational overhead.

[0116] Write the edited frame block to the output frame buffer and release the processed frame data from the input frame buffer.

[0117] In this embodiment of the invention, the task executor writes the processed entire frame block (including edited frames and pass-through frames) into the output frame buffer of its respective context manager in the original frame order. Each frame in the output frame buffer retains its absolute frame sequence number, and the writing order is guaranteed to be consecutive. After writing is complete, the task executor notifies the context manager to release the frame data in the input frame buffer that has been overwritten by this processing. The specific release strategy is as follows: since there are overlapping frames between frame blocks, the next load needs to retain the 0 frame at the end of the current frame block as the front overlap of the next block. Therefore, only the part before the end overlap frame (i.e., LO frames) in the current frame block is released from the input frame buffer. The context manager releases the data by moving the buffer pointer or updating the start flag, avoiding data accumulation that could lead to memory bloat. After the release is complete, the task executor enters a waiting state, waiting for the task scheduler to allocate an execution signal next time, and then continues loading subsequent frame blocks from the overlapping frames already retained in the input frame buffer.

[0118] This invention enables long video editing to be completed efficiently with limited memory by loading data in units of frame blocks, setting overlapping frames to ensure timing consistency, executing algorithms frame by frame in the editing area instead of directly passing through the editing area, and releasing the processed frame data in the input buffer in a timely manner after processing. At the same time, it uses overlapping frames to eliminate image quality jumps at block boundaries, thereby further reducing memory usage and improving the processing stability of multi-subtask pipelines on the basis of sharing a single encoding and decoding.

[0119] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, the method may further include:

[0120] After the current scheduled subtask has finished processing all frames in its input frame buffer, the current scheduled subtask is marked as completed and removed from the detection order.

[0121] Specifically, the task scheduler provided in this embodiment maintains a status flag (e.g., "not started," "running," and "completed") for each subtask. After processing all frame data in its input frame buffer (i.e., all required frames have been edited or passed through, and there are no remaining frames to be processed in the input buffer), the task executor of the currently scheduled subtask returns a completion signal to the scheduler. Upon receiving this signal, the scheduler updates the subtask's status flag to "completed" and removes it from the subsequent detection traversal order. The removal can be achieved by the task scheduler maintaining an active subtask list, initially containing subtasks 1 to n; when a subtask is completed, the corresponding item is removed from the list; subsequent detections from back to front only traverse the remaining subtasks in the list. If a subtask is marked as complete and removed after task partitioning because its editing area is empty or it does not require processing any frames (e.g., a user-defined no-operation), it will no longer be assigned an execution signal or its input frame buffer by the task scheduler, thus avoiding invalid scheduling.

[0122] The encoding task is triggered when all subtasks are marked as completed and the decoding task has no more video frames to decode.

[0123] Specifically, in each scheduling cycle (e.g., after each subtask iteration), the task scheduler checks two conditions: first, whether the status flags of all ordered subtasks (subtask 1 to subtask n) are all "completed"; second, whether the decoding special task has processed all frames of the video to be edited (i.e., the decoded frame position has reached the end of the video file, and the decoding task has no frame blocks to be output). When both conditions are met, it means that all frames of the original video have been decoded, all editing operations have been completed, and the final output frame sequence has been completely stored in the output frame buffer of the first subtask. At this time, the task scheduler inserts the encoding special task into the tail of the scheduling queue and immediately triggers the execution of the encoding task. After the encoding task starts, the final output frame sequence is read from the output frame buffer of the first subtask according to the aforementioned steps, encoded, and the target video is output. If only all subtasks are completed but the decoding task still has undecoded frames (e.g., decoding is paused due to pipeline blockage), the task scheduler will not trigger the encoding task, but will continue to try to load new frame blocks from the original video and schedule the first subtask until the decoding task ends naturally.

[0124] This invention avoids invalid traversal by promptly removing completed subtasks from the detection sequence and precisely triggering the encoding task when all subtasks are completed and the decoding task has no new frames to decode. This ensures that the editing pipeline terminates naturally with minimal scheduling overhead, thereby guaranteeing that the final output frame sequence enters the encoding stage completely and without errors.

[0125] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S220 may specifically include:

[0126] Perform the decoding task, call the video decoder to open the video file to be edited, and record the position of the currently decoded frame.

[0127] Specifically, in this embodiment of the invention, the decoding task can be activated through a task scheduler, calling the video decoder interface and passing in the file path of the video to be edited. The video decoder opens the file, reads the file header information (such as encoding format, resolution, frame rate, bitrate, and total number of frames), and initializes its internal decoding state. Simultaneously, the video decoder maintains a pointer to the current decoded frame position (e.g., decoded to frame 0, meaning no frames have been decoded yet). This pointer is continuously updated during the decoding process to ensure that decoding of subsequent frames can resume from the point of interruption at any time until the video ends. If the video file is corrupted or the format is unsupported, the decoding task returns an error and terminates the entire editing process.

[0128] Based on the frame block length required for the first subtask, the corresponding number of video frames are continuously decoded from the video to be edited to obtain the original video frame sequence.

[0129] Specifically, in this embodiment of the invention, the context manager of the first subtask (i.e., subtask 1) is queried by executing the decoding task to obtain the frame block length L (e.g., 8 or 16 frames) required for a single processing of the subtask. If the number of remaining undecoded frames in the video is less than L and the end of the file has not been reached, the actual number of remaining frames is decoded. Starting from the position of the currently decoded frame, the video decoder continuously decodes L frames (or the number of remaining frames). Each decoded frame is saved as raw pixel data (e.g., RGB or YUV format), and each frame is labeled with its absolute frame number in the original video (e.g., frame 0, frame 1, ..., frame L-1). After decoding is completed, the set of decoded consecutive frames is temporarily stored as a raw video frame sequence in a temporary buffer inside the decoding task.

[0130] Write the original video frame sequence into the input frame buffer of the context manager of the first subtask.

[0131] Specifically, in this embodiment of the invention, after obtaining the original video frame sequence through the decoding task, the write interface of the context manager bound to the first subtask is called to write the frame sequence into the input frame buffer of subtask 1. The context manager appends the frame data to the end of the buffer according to the absolute frame sequence number and updates the start sequence number, end sequence number, and consecutive frame count of the buffer. If the input frame buffer of subtask 1 already contains some frame data (e.g., a frame block has been written before), the newly written frame sequence should immediately follow the existing frames to ensure the continuity of frame sequence numbers in the buffer. After writing is completed, the decoding task clears the temporary buffer and records the end frame position of this decoding as the new "currently decoded frame position".

[0132] After the first subtask completes the processing of a frame block, the decoding task continues to decode the next frame block starting from the already decoded frame position, until all frames of the video to be edited have been decoded.

[0133] Specifically, the decoding task enters a waiting state, listening for the completion signal of subtask 1. Each time the task executor of subtask 1 completes a round of frame block iteration, the task scheduler notifies the decoding task. The decoding task then checks whether the currently decoded frame position has reached the end of the video file: if not, it repeats the steps of "decoding according to frame block length → writing to the input frame buffer," writing the decoded next frame block to the input frame buffer of subtask 1 and updating the position of the decoded frame; if the end has been reached and all frames have been decoded and written, the decoding task marks itself as complete and no longer performs subsequent decoding operations. Once all frames of the video to be edited have been decoded and all subtasks have been processed, the decoding task exits. Throughout this process, the decoding task and subtask 1 form a producer-consumer pipeline, outputting only one frame block at a time, avoiding loading the entire video into memory at once and reducing memory usage.

[0134] This invention, through its embodiment, decodes each frame block according to the length required for the first subtask, records the decoding position, and blocks and waits for the subtask to finish processing before decoding the next block. This forms a frame block-level pipeline for the decoding and editing process, which avoids loading the entire video into memory at once, thus reducing memory usage, and ensures that the decoding speed matches the subtask processing speed. This minimizes decoding latency and memory peaks while sharing a single decoding chain.

[0135] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S240 may specifically include:

[0136] Obtain the encoding parameters of the video to be edited.

[0137] Specifically, in this embodiment of the invention, after activating the encoding task through the task scheduler, the original encoding parameters of the video to be edited are first obtained from the decoding task or task management module. The original encoding parameters may include the video encoding format (e.g., H.264, H.265), resolution (e.g., 1920×1080), frame rate (e.g., 25fps), bitrate mode and target bitrate (e.g., constant bitrate or variable bitrate, specific value such as 4Mbps), GOP (Group of Pictures) size, color space and chroma sampling format (e.g., YUV420), and encoding profile and level (e.g., High Profile, Level 4.0). If the original encoding parameters cannot be obtained (e.g., the original video is not provided), the system's preset default parameters (e.g., H.264, bitrate 2Mbps) can be used. The encoding task saves these parameters in its internal context for subsequent encoder initialization configuration.

[0138] The final output frame sequence is encoded according to the encoding parameters, and the encoded video stream is written to a file to output the target video.

[0139] Specifically, in this embodiment of the invention, the encoding task can call the video encoder and use the encoding parameters obtained in the previous step for initial configuration (such as setting the encoding format, resolution, frame rate, bitrate, GOP size, etc.). The encoder opens the output file (the target video file path is specified by the user or generated by the system) and prepares to write data. Then, the encoding task receives the final output frame sequence passed from the output frame buffer of the first subtask. Each frame in this sequence retains its original absolute frame number and pixel data format (such as RGB). Before passing it to the encoder, the encoding task may need to perform color space conversion (e.g., RGB to YUV) to match the encoder's input requirements. The encoder compresses and encodes the final output frame sequence frame by frame (or frame group by frame) sequentially, generating NAL units or encoded data packets conforming to standards such as H.264 / H.265. During the encoding process, the encoding task listens to the data stream output by the encoder and appends it to the output file each time a complete data block (such as the encoded data of a frame) is generated. When all frames of the entire final output frame sequence have been encoded, the encoding task calls the relevant interface of the encoder (such as flush) to refresh the remaining data in the encoding buffer, closes the output file, and releases the encoder resources. The final output is the target video file, which is the final result after editing all subtasks. Because the entire editing process only involves encoding once, the accumulation of quantization errors caused by multiple encodings is avoided, and the loss of image quality is minimized.

[0140] This invention obtains the original encoding parameters of the video to be edited and reuses these parameters to encode the final output frame sequence in one go. This ensures that the output video maintains the same encoding configuration as the original video while completely avoiding the accumulation of quantization errors introduced by multiple rounds of encoding and decoding. Thus, the edited image quality is preserved to the greatest extent while ensuring compatibility.

[0141] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0142] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0143] Corresponding to the above method embodiments, this invention also provides a video editing device, the structure of which is as follows: Figure 5 As shown, it may include: a video editing task acquisition unit 10, a subtask queue acquisition unit 20, a decoding task execution unit 30, a task scheduling unit 40, and an encoding task execution unit 50.

[0144] The video editing task acquisition unit 10 is used to acquire video editing tasks submitted by users, wherein the video editing task includes the video to be edited and at least one editing operation.

[0145] Subtask queue acquisition unit 20 is used to divide the editing operation into multiple ordered subtasks, and insert decoding tasks and encoding tasks at the beginning and end respectively to obtain the subtask queue.

[0146] The decoding task execution unit 30 is used to perform decoding tasks, decode the video to be edited, and pass the resulting original video frame sequence to the first subtask.

[0147] The task scheduling unit 40 is used to schedule each subtask to perform editing iteratively in the opposite direction to the order of the subtask queue, and pass the edited frame buffer of each subtask forward level by level until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence.

[0148] The encoding task execution unit 50 is used to execute encoding tasks, encode the received final output frame sequence, and output the target video.

[0149] Optionally, the subtask queue acquisition unit 20 can be used to divide editing operations into multiple subtasks with a sequential order according to the spatial dependency between regions when there are multiple editing operations in different regions on the same continuous frame segment of the video to be edited; when there are editing operations on different continuous frame segments of the video to be edited, divide the editing operation corresponding to each segment into an independent subtask; sort all the subtasks obtained after division in reverse order of dependency to obtain each ordered subtask; insert a decoding task before the first sorted subtask and an encoding task after the last subtask to generate a subtask queue.

[0150] Optionally, the task scheduling unit 40 can be used to sequentially check whether the frame data in the input frame buffer of each subtask in the subtask queue meets the execution conditions in a backward order. For subtasks that meet the execution conditions, the task scheduling unit 40 is used as the current scheduled subtask. The task scheduling unit loads frame data from the input frame buffer of the context manager to which the current scheduled subtask belongs, edits the frames in the frame data that belong to the current editing area, and writes them to the output frame buffer. Other frames are directly passed through to the output frame buffer. The task scheduling unit loads the output frame buffer of the next subtask as the input frame buffer of the previous subtask, and passes the data level by level until the output frame buffer of the first subtask is used to pass the final output frame sequence to the encoding task.

[0151] Optionally, the task scheduling unit 40 can be further configured to traverse each subtask sequentially from the last subtask in the subtask queue toward the first subtask; for the currently traversed subtask, obtain the number of consecutive frames already cached in the input frame buffer corresponding to the subtask; compare the number of consecutive frames with the frame block length required for a single processing of the subtask; if the number of consecutive frames is greater than or equal to the frame block length, then the execution condition is satisfied; if the number of consecutive frames is less than the frame block length, then continue to detect the previous subtask.

[0152] Optionally, the task scheduling unit 40 can further be used to load a frame block from the input frame buffer of its context manager using the task executor corresponding to the current scheduled subtask. The length of the frame block is equal to the length of the frame block required for a single processing by the current scheduled subtask, and an overlapping frame is set between two adjacent loaded frame blocks. An editing operation is performed on each frame in the frame block. For frames in the frame block that belong to the editing area, an edited frame is generated according to the editing parameters carried by the current scheduled subtask. For frames that do not belong to the editing area, the input frame is directly used as the output frame. The edited frame block is written to the output frame buffer, and the processed frame data in the input frame buffer is released.

[0153] Optionally, the video editing device may also include an encoding task triggering unit.

[0154] The encoding task triggering unit is used to mark the current scheduled subtask as completed and remove it from the detection order after the current scheduled subtask has completed processing all frames in its input frame buffer; and to trigger the encoding task execution unit 50 to execute the encoding task when all subtasks have been marked as completed and the decoding task has no video frames to decode.

[0155] Optionally, the decoding task execution unit 30 can be used to execute decoding tasks, call the video decoder to open the video file to be edited, and record the position of the currently decoded frame; according to the frame block length required by the first subtask, continuously decode the corresponding number of video frames from the video to be edited to obtain the original video frame sequence; write the original video frame sequence into the input frame buffer of the context manager of the first subtask; after the first subtask completes the processing of a frame block, the decoding task continues to decode the next frame block from the already decoded frame position until all frames of the video to be edited are decoded.

[0156] Optionally, the encoding task execution unit 50 can be used to obtain the encoding parameters of the video to be edited; encode the final output frame sequence according to the encoding parameters; write the encoded video stream to a file; and output the target video.

[0157] This invention provides a video editing apparatus for: obtaining a user-submitted video editing task, wherein the video editing task includes a video to be edited and at least one editing operation; dividing the editing operation into multiple ordered subtasks, inserting decoding and encoding tasks at the beginning and end respectively, to obtain a subtask queue; executing the decoding task to decode the video to be edited, and passing the resulting original video frame sequence to the first subtask; sequentially scheduling each subtask to iteratively execute editing in the opposite direction to the order of the subtask queue, and passing the edited frame buffers of each subtask forward level by level until the output frame buffer of the first subtask is passed as the final output frame sequence to the encoding task; and executing the encoding task to encode the received final output frame sequence and output the target video. This invention, by dividing multiple editing operations into ordered subtasks and performing decoding and encoding only once at the beginning and end, and sharing intermediate frame data between subtasks through zero-copy transmission of input and output frame buffers, avoids the accumulation of quantization errors caused by repeated encoding and decoding, thereby reducing video editing quality loss while reducing hardware resource consumption.

[0158] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0159] The video editing device includes a processor and a memory. The video editing task acquisition unit 10, the sub-task queue acquisition unit 20, the decoding task execution unit 30, the task scheduling unit 40, and the encoding task execution unit 50 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0160] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and by adjusting kernel parameters, the accumulation of quantization errors caused by repeated encoding and decoding can be avoided, thus saving hardware resources and reducing image quality loss.

[0161] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the video editing method.

[0162] This invention provides a processor for running a program, wherein the program executes the video editing method during runtime.

[0163] like Figure 6 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned video editing method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0164] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program that initializes video editing method steps.

[0165] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0167] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0168] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0170] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0171] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0172] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0173] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. A video editing method, characterized in that, include: Obtain a video editing task submitted by a user, wherein the video editing task includes a video to be edited and at least one editing operation; The editing operation is divided into multiple ordered subtasks, and decoding and encoding tasks are inserted at the beginning and end respectively to obtain a subtask queue. The decoding task is executed to decode the video to be edited, and the resulting original video frame sequence is passed to the first subtask. In the opposite direction to the order of the subtask queue, each subtask is scheduled to be executed iteratively for editing, and the frame buffers edited by each subtask are passed forward level by level until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence. The encoding task is performed to encode the received final output frame sequence and output the target video.

2. The method according to claim 1, characterized in that, The step of dividing the editing operation into multiple ordered subtasks, and inserting decoding and encoding tasks at the beginning and end respectively, to obtain a subtask queue, includes: When there are multiple editing operations in different regions on the same consecutive frame segment of the video to be edited, the editing operations are divided into multiple sub-tasks with a sequential order according to the spatial dependency between the regions; When editing operations exist on different consecutive frame segments of the video to be edited, the editing operation corresponding to each segment is divided into an independent subtask; Sort all the subtasks obtained after the division in reverse order of their dependencies to obtain ordered subtasks. The decoding task is inserted before the first sorted subtask, and the encoding task is inserted after the last subtask, to generate a subtask queue.

3. The method according to claim 1, characterized in that, The step of scheduling each subtask to be edited iteratively in the opposite direction to the order of the subtask queue, and passing the edited frame buffers of each subtask forward level by level, until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence, includes: The frame data in the input frame buffer of each subtask in the subtask queue is checked sequentially from back to front to see if it meets the execution conditions. Subtasks that meet the execution conditions are designated as the current scheduled subtasks: frame data is loaded from the input frame buffer of the context manager to which the current scheduled subtask belongs; frames belonging to the current editing area are edited and written to the output frame buffer; other frames are directly passed through to the output frame buffer; the output frame buffer of the next subtask is used as the input frame buffer of the previous subtask, and so on, until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence.

4. The method according to claim 3, characterized in that, The step of sequentially checking whether the frame data in the input frame buffer of each subtask in the subtask queue meets the execution conditions in reverse order includes: Starting from the last subtask in the subtask queue, traverse each subtask sequentially in the direction of the first subtask; For the currently traversed subtask, obtain the number of consecutive frames already cached in the input frame buffer corresponding to the subtask; The number of consecutive frames is compared with the frame block length required for a single processing of the subtask. If the number of consecutive frames is greater than or equal to the frame block length, the execution condition is determined to be met. If the number of consecutive frames is less than the frame block length, then continue detecting the previous subtask.

5. The method according to claim 3, characterized in that, The steps of loading frame data from the input frame buffer of the context manager to which the currently scheduled subtask belongs, editing the frames belonging to the current editing region within the frame data and writing them to the output frame buffer, and directly passing other frames to the output frame buffer include: The task executor corresponding to the current scheduled subtask loads a frame block from the input frame buffer of its context manager. The length of the frame block is equal to the length of the frame block required for a single processing of the current scheduled subtask, and an overlapping frame is set between two adjacent loaded frame blocks. An editing operation is performed on each frame in the frame block. For frames in the frame block that belong to the editing region, an edited frame is generated based on the editing parameters carried by the current scheduling subtask. For frames that do not belong to the editing region, the input frame is directly used as the output frame. The edited frame block is written to the output frame buffer, and the processed frame data in the input frame buffer is released.

6. The method according to claim 3, characterized in that, Also includes: After the current scheduled subtask has completed processing all frames in its input frame buffer, the current scheduled subtask is marked as completed and removed from the detection order. The encoding task is triggered when all subtasks are marked as completed and the decoding task has no more video frames to decode.

7. The method according to claim 1, characterized in that, The step of executing the decoding task, decoding the video to be edited, and passing the resulting original video frame sequence to the first subtask includes: The decoding task is executed by calling the video decoder to open the video file to be edited and recording the position of the currently decoded frame. According to the frame block length required for the first subtask, the corresponding number of video frames are continuously decoded from the video to be edited to obtain the original video frame sequence. The original video frame sequence is written into the input frame buffer of the context manager of the first subtask; After the first subtask completes the processing of a frame block, the decoding task continues to decode the next frame block starting from the already decoded frame position, until all frames of the video to be edited are decoded.

8. The method according to claim 1, characterized in that, The execution of the encoding task, which encodes the received final output frame sequence and outputs the target video, includes: Obtain the encoding parameters of the video to be edited; The final output frame sequence is encoded according to the encoding parameters, the encoded video stream is written to a file, and the target video is output.

9. A video editing device, characterized in that, include: The system includes a video editing task acquisition unit, a subtask queue acquisition unit, a decoding task execution unit, a task scheduling unit, and an encoding task execution unit. The video editing task obtaining unit is used to obtain the video editing task submitted by the user, wherein the video editing task includes the video to be edited and at least one editing operation; The subtask queue obtaining unit is used to divide the editing operation into multiple ordered subtasks, and insert decoding tasks and encoding tasks at the beginning and end respectively to obtain a subtask queue. The decoding task execution unit is used to execute the decoding task, decode the video to be edited, and pass the resulting original video frame sequence to the first subtask; The task scheduling unit is used to schedule each subtask to be edited iteratively in the opposite direction to the order of the subtask queue, and pass the edited frame buffer of each subtask forward level by level until the output frame buffer of the first subtask is passed to the encoding task as the final output frame sequence. The encoding task execution unit is used to execute the encoding task, encode the received final output frame sequence, and output the target video.

10. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the video editing method as described in any one of claims 1 to 8.

Images