Video processing system, video processing method, and program product
By generating video frame index information for parallel decoding and utilizing a multi-channel heterogeneous decoding engine and concurrent task architecture, the problems of discontinuous access and I/O delay in video frame loading are solved, achieving efficient video processing and computing resource utilization.
Patent Information
- Application Number
- CN202510906027.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-01
- Publication Date
- 2025-10-03
AI Technical Summary
In existing video frame loading technology, non-continuous jump access between frames cannot be achieved, resulting in the data stream being unable to meet the randomness requirements of the algorithm and there being significant I/O delays, which seriously restricts video processing efficiency and performance improvements.
By generating index information of video frames, utilizing multi-channel heterogeneous decoding engines to process video frames in parallel, building a task concurrency architecture, asynchronously preloading video frames, realizing cross-task data structure conversion and sharing, and optimizing the decoding and consumption process of video frames.
It improves decoding efficiency, reduces processing jams, enhances computing resource utilization and the overall performance of the video processing system, and ensures the flexibility and efficiency of the video processing system.
Smart Images
Figure CN120751173A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video processing technology, and in particular to a video processing system, a video processing method and a program product. Background Art
[0002] Currently, video frame loading technology typically uses a single video sequential read method. However, this method has significant limitations in data loading. For example, it cannot implement non-continuous jump access between frames, resulting in the data stream being unable to meet the randomness requirements of the algorithm. In addition, significant I / O latency will occur, seriously restricting video processing efficiency and performance improvements. Summary of the Invention
[0003] In view of this, embodiments of the present application provide a video processing system, a video processing method, and a program product.
[0004] In the first aspect, an embodiment of the present application provides a video processing system, including: a parsing layer, configured to parse target video data and generate index information of the target video data, the index information including position information of video frames in the target video data in the target video data; a decoding layer, configured to locate the target video frame from the target video data based on the index information, and schedule the target video frame to a target decoding engine for decoding.
[0005] In combination with the first aspect, in certain implementations of the first aspect, the number of target video data is multiple, and the decoding layer includes multiple heterogeneous decoding engines; the decoding layer is also configured to determine the target decoding engine corresponding to the target video data from the multiple heterogeneous decoding engines based on the metadata of each target video data, so as to schedule the target video frame of the target video data to the target decoding engine for decoding, and the metadata includes the video data format and / or resolution.
[0006] In combination with the first aspect, in certain implementations of the first aspect, the decoding layer is further configured to determine the video frame decoding task and the video frame consumption task corresponding to the target video frame, and construct a task concurrency architecture between the video frame decoding task and the video frame consumption task so as to execute the video frame decoding task and the video frame consumption task in parallel.
[0007] In combination with the first aspect, in certain implementations of the first aspect, the task concurrency architecture is constructed based on multiple CUDA Streams, and the video frame decoding task and the video frame consumption task are each executed in a different CUDA Stream.
[0008] In combination with the first aspect, in certain implementations of the first aspect, the video processing system further includes: a data cache layer, configured to divide other modal data other than video data into multiple data blocks based on the target data granularity, cache the multiple data blocks and generate index information for each of the multiple data blocks, so that when executing the video frame consumption task corresponding to the target video frame, the target data block can be located from the multiple data blocks based on the index information of each of the multiple data blocks, so as to coordinate the loading of the target video frame and the target data block.
[0009] In combination with the first aspect, in certain implementations of the first aspect, the video processing system also includes: a dynamic loading layer, configured to asynchronously preload new target video frames that need to be consumed by the video frame consumption task when the video frame consumption task corresponding to the target video frame processes the target video frame.
[0010] In combination with the first aspect, in certain implementations of the first aspect, the video processing system further includes: an adaptation layer configured to convert the target video frame into a target data structure required by the video frame consumption task corresponding to the target video frame; and / or, based on a cross-task data description protocol, enable the converted target video frame to be shared between different video frame consumption tasks.
[0011] In combination with the first aspect, in some implementations of the first aspect, the video frame consumption task includes a model training task.
[0012] In combination with the first aspect, in certain implementations of the first aspect, the index information also includes the timestamp of the video frame in the target video data, and / or the frame type of the video frame in the target video data; wherein the target video data is encoded using a fixed frame rate, and / or the video frames in the target video data only include key frames and forward prediction frames, the key frame represents an independently encoded frame containing complete image information, and the forward prediction frame represents an encoded frame obtained by motion compensation prediction based on the temporally preceding video frame.
[0013] In the second aspect, an embodiment of the present application provides a video processing method, including: parsing the target video data, generating index information of the target video data, the index information including the position information of the video frame in the target video data; based on the index information, locating the target video frame from the target video data, and scheduling the target video frame to the target decoding engine for decoding.
[0014] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the video processing method of the second aspect.
[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; and the processor is configured to execute the video processing method of the second aspect.
[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements the video processing method of the second aspect.
[0017] In this application, on the one hand, the generation of index information provides clear guidance for the decoding process, avoiding the decoding layer from blindly searching for the target video frame frame by frame in the entire target video data, thereby greatly improving decoding efficiency and shortening the time required for decoding; on the other hand, the precise positioning of the target video frame enables the video processing system to respond more quickly to the processing requirements for specific video frames. Then, based on the index information, the decoding layer locates the target video frame from the target video data and schedules the target video frame to the target decoding engine for decoding, which can avoid processing jams caused by insufficient decoding capabilities and effectively improve the utilization of computing resources and decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.
[0019] Figure 1 FIG2 is a schematic diagram of the structure of a video processing system provided in one embodiment of the present application.
[0020] Figure 2 Shown is a structural diagram of a video processing system provided by another embodiment of the present application.
[0021] Figure 3 FIG2 is a schematic diagram of the structure of a video processing system provided in another embodiment of the present application.
[0022] Figure 4 FIG2 is a schematic diagram of the structure of a video processing system provided in another embodiment of the present application.
[0023] Figure 5 FIG2 is a schematic diagram of the structure of a video processing system provided in another embodiment of the present application.
[0024] Figure 6 FIG2 is a flow chart of a video processing method provided in an embodiment of the present application.
[0025] Figure 7 FIG2 is a flow chart of a video processing method provided in another embodiment of the present application.
[0026] Figure 8 Shown is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0028] Figure 1 FIG. 1 is a schematic diagram of the structure of a video processing system provided by an embodiment of the present application. Figure 1 As shown, the video processing system 10 includes a parsing layer 110 and a decoding layer 120 .
[0029] The parsing layer 110 is configured to parse the target video data and generate index information of the target video data; the decoding layer 120 is configured to locate the target video frame from the target video data based on the index information and schedule the target video frame to the target decoding engine for decoding.
[0030] Specifically, index information is indicative information generated by the parsing layer 110 after parsing the target video data. In this embodiment, the index information includes position information of video frames in the target video data. Exemplarily, the position information may be the physical storage location of each video frame of the target video data in the video file, such as a byte offset.
[0031] In one implementation, the encapsulation format of the target video data is parsed to separate components such as the video stream, audio stream, and metadata; each video frame in the video stream is traversed, and the starting byte offset of each video frame of the target video data in the target video data is recorded, thereby generating index information of the target video data.
[0032] In some embodiments, the index information also includes the timestamp of the video frame in the target video data, and / or the frame type of the video frame in the target video data; wherein the target video data is encoded using a fixed frame rate, and / or the video frames in the target video data only include key frames (Intra Frame, I frame) and forward predicted frames (Predictive Frame, P frame).
[0033] Specifically, the timestamp is used to identify the position of the video frame on the time axis. Optionally, the timestamp includes a presentation time stamp (PTS) and a decoding time stamp (DTS).
[0034] In this embodiment, the target video data is encoded using a fixed frame rate, ensuring a consistent number of video frames per second and uniform frame intervals, thereby maintaining consistency in the temporal dimension. Based on this, the frame position can be directly calculated from the timestamp, eliminating the need for frame-by-frame scanning. This facilitates subsequent processing by the parsing layer 110 and the decoding layer 120, improving video processing efficiency.
[0035] Furthermore, a key frame represents an independently coded frame containing complete image information, and a forward prediction frame represents a coded frame obtained by motion-compensated prediction based on the preceding video frame in time sequence. Optionally, a key frame has a larger data volume and serves as the starting point for random access to the video (e.g., locating an I-frame when fast-forwarding / fast-rewinding). A forward prediction frame is motion-compensated prediction coded based on the preceding frame (usually an I-frame or P-frame), storing only difference data (e.g., motion vectors, residuals, etc.), and the data volume is generally smaller than that of an I-frame.
[0036] In some embodiments, the index information can be saved as a separate file (e.g., .idx, json) or embedded in the target video data. Furthermore, in some embodiments, different computing nodes can use the index information to read different frames of the same target video data (e.g., node A reads frame 100, node B reads frame 200), avoiding duplicate loading and supporting applications in parallel processing and distributed training scenarios.
[0037] The decoding layer 120 receives the index information sent by the parsing layer 110 and, based on the index information, determines the physical storage location of the target video frame in the target video data. The decoding layer 120 decodes the video data of the target video frame based on the target decoding engine to output the decoded video data of the target video frame. Alternatively, the decoding layer 120 may only inform the target decoding engine of the storage location of the target video frame, which then directly jumps to the target video frame and begins decoding.
[0038] In the embodiments, the generation of index information provides clear guidance for the decoding process, preventing the decoding layer from blindly searching for the target video frame frame by frame within the entire target video data, thereby significantly improving decoding efficiency and shortening the decoding time. Furthermore, the precise location of the target video frame enables the video processing system to more quickly respond to processing requirements for specific video frames. The decoding layer then locates the target video frame within the target video data based on the index information and dispatches it to the target decoding engine for decoding. This avoids processing delays caused by insufficient decoding capabilities and effectively improves computing resource utilization and decoding efficiency.
[0039] Considering that a single central processing unit (CPU) or graphics processing unit (GPU) is prone to computing bottlenecks when decoding multiple channels of video data, in order to process complex video data, the present application simultaneously runs multiple heterogeneous decoding engines in the video processing system 10 to improve video processing efficiency and meet high-throughput computing requirements. Furthermore, the decoding layer 120 is further configured to determine the target decoding engine corresponding to the target video data from the multiple heterogeneous decoding engines based on the metadata of each target video data, so as to dispatch the target video frame of the target video data to the target decoding engine for decoding.
[0040] It is understandable that the multi-channel heterogeneous engine can process multi-channel target video data at the same time, thereby improving the overall performance of the video processing system through heterogeneous computing and parallel processing, solving the bottlenecks of computing power, power consumption, and compatibility of a single decoding engine, and improving decoding efficiency. Optionally, the multi-channel heterogeneous engine can be divided into hardware type heterogeneity and software stack heterogeneity. Among them, hardware type heterogeneity includes computing units of different architectures, such as CPU, GPU, edge device chips, etc. Different engines of software stack heterogeneity support different decoding libraries and protocols, such as GPU engine, CPU engine and neural network engine.
[0041] Meta-information is a key parameter that describes the properties of the target video data, including the video data format and / or resolution. The video data format includes the encoding format and the encapsulation format. Meta-information can also include at least one of the following: frame rate, bit rate, and color space.
[0042] In some embodiments, the decoding layer 120 is configured with a decoding engine scheduling strategy. Specifically, a correspondence between multiple decoding engines and multiple video data is pre-set, and a target decoding engine is scheduled based on this correspondence. For example, the correspondence can be a correspondence between multiple decoding engines and multiple video data encoding formats. The decoding layer 120 determines the target decoding engine based on the data encoding format of the target video frame and this correspondence, and schedules the target video frame to the target decoding engine for decoding based on the target decoding engine to obtain the original video data corresponding to the target video frame.
[0043] Specifically, for each target video data, the decoding layer 120 extracts the metadata of each target video data, selects a target decoding engine from multiple heterogeneous decoding engines based on the encoding format and resolution contained in the metadata, and transmits the target video frame of the target video data to the target decoding engine for decoding. This has the advantage of allowing different target video frames to be processed in parallel by different decoding engines, improving video processing efficiency.
[0044] It is understandable that in order to avoid increased latency caused by overloading a single decoding engine, when multiple decoding engines work in parallel, the load of each decoding engine can be monitored in real time. When it is detected that one or more decoding engines are heavily loaded, other decoding engines with smaller loads can be called to parse the target video frames to balance the load of each decoding engine.
[0045] In this embodiment, first, multiple heterogeneous decoding engines can process multiple target video data in parallel, avoiding the bottleneck problem of a single decoding engine in multi-tasking processing and significantly improving decoding efficiency. Secondly, a selection mechanism based on metadata (such as video data format and / or resolution) enables video data with different characteristics to be allocated to the most suitable decoding engine, thereby improving resource utilization and reducing waste. In addition, this design enhances the decoding compatibility of the video processing system, enabling flexible adaptation to a variety of video formats and resolutions, ensuring the stability and reliability of the decoding process.
[0046] To accommodate high-performance computing scenarios, a producer-consumer model can be constructed in the decoding layer 120 to decouple the decoding and consumption processes, enabling parallel execution of the two and fully utilizing computing resources. Specifically, in this embodiment, the decoding layer 120 is further configured to determine the video frame decoding task and video frame consumption task corresponding to the target video frame, and to construct a task concurrency architecture between the video frame decoding task and the video frame consumption task so that the video frame decoding task and the video frame consumption task can be executed in parallel.
[0047] Specifically, the video frame decoding and consumption tasks form a producer-consumer model, where the producer is responsible for generating data (i.e., the video frame decoding task), and the consumer is responsible for processing the data (i.e., the video frame consumption task). For example, they can execute specific business logic based on the decoded video frames.
[0048] The concurrent task architecture between the video frame decoding task and the video frame consumption task refers to a parallel working mechanism designed to achieve efficient processing of target video frames in the video processing system 10. Traditional video processing methods generally perform decoding and consumption tasks sequentially, that is, first complete the decoding of all video frames, and then perform video frame consumption processing. However, this approach will result in high latency and low efficiency when processing a large number of video frames.
[0049] In this embodiment, the task concurrency architecture breaks the sequential restrictions of video frame decoding tasks and video frame consumption tasks, allowing decoding tasks and consumption tasks to proceed simultaneously. In other words, when the decoding layer begins decoding the target video frame, once some video frames have completed decoding, these decoded frames can be immediately scheduled to the consumption task for processing or display, without having to wait for all video frames to complete decoding. This parallel execution method can fully utilize the system's computing resources, reduce the total time of task execution, and improve the overall efficiency and smoothness of video processing.
[0050] In some embodiments, the video frame consumption task includes a model training task. That is, in the process of processing video frames, the decoded video frame data can be used for model training. Exemplarily, the model training task includes a model training task for an embodied robot. For example, the decoded video frame can be used as data input to train the visual perception model of the embodied robot, so that it can more accurately identify objects, scenes and dynamic changes in the environment; it can also be used to train the navigation model to learn the optimal path planning strategy based on the spatial information in the video frame; it can also be used to train the operation skill model, allowing the robot to learn complex operation skills by observing video frames of human actions. It provides key data support for the learning and intelligence improvement of embodied robots, expands the application value of video data in the field of robotics, and realizes deep integration from data decoding to intelligent model construction.
[0051] In some embodiments of the present application, a task concurrency architecture is constructed based on multiple Compute Unified Device Architecture Streams (CUDA Streams), and video frame decoding tasks and video frame consumption tasks are each executed in a different CUDA Stream.
[0052] Specifically, CUDA is a parallel computing architecture that applies the powerful computing capabilities of GPUs to non-graphics tasks. A CUDA stream is a concept in CUDA that is an execution sequence. Commands within it are executed in the order they are submitted, but commands in different streams can be executed in parallel.
[0053] In one implementation, multiple CUDA streams are created, each corresponding to a task execution sequence. The video frame decoding task is then assigned to a CUDA stream for execution. After a portion of the decoding task is completed, the decoded video frames are placed into a shared buffer. Simultaneously, the video frame consumption task is assigned to another CUDA stream for execution. The consumer task reads the decoded video frames from the shared buffer and performs corresponding processing, such as model training.
[0054] Since tasks in different CUDA Streams can be executed in parallel, this embodiment achieves parallel processing of tasks by assigning video frame decoding tasks and consumption tasks to different CUDA Streams for execution, thereby effectively reducing the total time for task execution and improving the processing efficiency of the video processing system. In addition, the CUDA Stream-based architecture can better utilize GPU resources, avoid idle resources, and improve the performance of the video processing system. At the same time, through the shared buffer mechanism, the decoded video frames can be passed to the consumption task for processing in a timely manner, ensuring the smooth connection of the decoding and consumption processes and reducing the delay of data transmission.
[0055] Figure 2 FIG. 1 is a schematic diagram of the structure of a video processing system provided by another embodiment of the present application. Figure 2 As shown, the video processing system also includes a data cache layer 130, which is configured to divide other modal data other than video data into multiple data blocks based on the target data granularity, cache the multiple data blocks and generate index information for each of the multiple data blocks, so that when executing the video frame consumption task corresponding to the target video frame, the target data block can be located from the multiple data blocks based on the index information of the multiple data blocks, so as to coordinate the loading of the target video frame and the target data block.
[0056] Data granularity is a measure of the level of detail and generalization in data. It directly impacts data storage, query efficiency, and the depth and breadth of analysis. Optionally, target data intensity can be determined based on storage and computing resources. When resources are abundant, fine-grained data can be used; when resources are limited, granularity needs to be weighed to optimize costs.
[0057] Other modal data includes text, audio, and other data. Correspondingly, dividing other modal data other than video data into multiple data blocks based on the target data granularity can be dividing text data into multiple data blocks based on paragraphs / sentences, and / or dividing audio data into multiple data blocks based on time windows (e.g., 1-second segments).
[0058] Optionally, the index information of each of the multiple data blocks includes the location information of the data block in its corresponding modal data, timestamp, etc. Exemplarily, data block partitioning strategies include hash partitioning, range partitioning, list partitioning, etc. Among them, hash partitioning is suitable for scenarios that require load balancing; range partitioning is based on the value range of a certain field of data and is suitable for time series data; list partitioning is suitable for data with clear classification attributes. It is also possible to combine multiple strategies, such as first partitioning by time range and then using hash partitioning for the data within each range.
[0059] In this embodiment, the video processing system includes a data caching layer that partitions modal data other than video data based on the target data granularity, forming multiple data blocks. These data blocks are cached, and corresponding index information is generated for each data block. When a video frame consumption task related to a target video frame needs to be executed, the system can use this index information to quickly locate the specific target data block from the multiple data blocks. This allows for coordinated loading of the target video frame and target data blocks, thereby improving video processing efficiency and data management coordination.
[0060] Video frame consumption tasks typically have timing dependencies and require sequential processing. Therefore, if a video consumption task completes processing the current frame and then requests the next frame's data, data loading may be delayed. Therefore, to avoid interruptions or freezes in video consumption tasks, subsequent video frames can be preloaded to improve the smoothness and efficiency of the video processing system.
[0061] Figure 3 FIG. 1 is a schematic diagram of the structure of a video processing system provided by another embodiment of the present application. Figure 3 As shown, the video processing system 10 further includes a dynamic loading layer 140 configured to asynchronously preload new target video frames that the video frame consumption task needs to consume when the video frame consumption task corresponding to the target video frame processes the target video frame.
[0062] Specifically, asynchronous preloading means loading the data required for subsequent tasks in advance in the background without affecting the execution of the current task, so that the data is available when needed, reducing waiting time.
[0063] In one implementation, a video frame consumption task is processing the current target video frame. The dynamic loading layer 140 asynchronously preloads the next target video frame or frames while the consumption task is processing the current frame. The preloaded frames are immediately available after the consumption task completes processing, reducing waiting time and improving processing efficiency.
[0064] For example, suppose the video frame consumption task is processing the target video frame at the 10th second. Simultaneously, the dynamic loading layer 140 asynchronously preloads the target video frames at the 11th and 12th seconds. When the consumption task completes processing the 10th second, the target video frame at the 11th second has already been loaded and can be immediately processed and displayed. This effectively reduces video playback lag and improves the user experience.
[0065] Considering that in some scenarios, the preloading speed is related to the preloading window size setting, the preloading window size can also be set to improve the processing speed of the video frame consumption task. Optionally, for scenarios with high real-time requirements but limited memory, a smaller window can be set; for high-latency data links, a larger window can be set to fully utilize the bandwidth for early loading. Optionally, adaptive adjustment of the preloading window size can also be set. For example, if the processing speed of the video frame consumption task is greater than the preloading speed, the preloading window can be expanded.
[0066] In this embodiment, while the consumer task is processing the current frame, the next few frames are preloaded, which can reduce lag and improve smoothness. In addition, asynchronous preloading runs in the background without affecting the current task, making full use of idle resources.
[0067] In some scenarios, the data structures of video frames corresponding to different video frame consumption tasks may be different. Therefore, in order to solve the problems of data compatibility and cross-task data sharing, the data structure of video frames can be converted to ensure efficient flow of video frames between different tasks and avoid repeated decoding or conversion.
[0068] Figure 4 FIG. 1 is a schematic diagram of the structure of a video processing system provided by another embodiment of the present application. Figure 4 As shown, the video processing system also includes: an adaptation layer 150, configured to convert the target video frame into a target data structure required by the video frame consumption task corresponding to the target video frame; and / or, based on a cross-task data description protocol, enable the converted target video frame to be shared between different video frame consumption tasks.
[0069] Optionally, the conversion includes color space conversion and / or data type conversion. Color space conversion includes converting the target video frame from YUV format to RGB format, or converting from BGR format to RGB format. Data type conversion includes converting the uint type of the target video frame to float type, or converting integer pixel values to fixed-point representation, etc.
[0070] The main purpose of the cross-task data description protocol is to standardize how different video frame consumption tasks interact in terms of data representation, transmission, and parsing. For example, the cross-task data description protocol includes the DLPack protocol, which defines a common data structure that enables efficient and accurate data exchange between different video frame consumption tasks.
[0071] In this embodiment, the adaptation layer converts target video frames into corresponding target data structures based on the requirements of different video frame consumption tasks. This ensures that the target video frames are provided in the most suitable format for various consumption tasks, enhancing the versatility and flexibility of the video processing system. Furthermore, this conversion reduces data preprocessing time before task execution, improves processing efficiency, and enhances the overall performance of the video processing system.
[0072] In this embodiment, the cross-task data description protocol defines a set of unified data formats and interface specifications, so that the converted video frames can be consistently identified, accessed and utilized in different consumption tasks. Video frame consumption tasks may be diverse, such as target recognition, feature extraction, or training tasks of deep learning models. After the target video frame is converted into a data format that conforms to the protocol, different types of consumption tasks can directly share and use these video frame data without additional data conversion steps, thereby improving the reusability of data and the collaborative efficiency between tasks. This sharing mechanism reduces the overhead of data conversion and transmission, reduces the resource consumption of the video processing system, and at the same time speeds up the execution speed of different video frame consumption tasks, thereby improving the efficiency and fluency of the entire video processing system.
[0073] The following combination Figure 5 and Figure 6 Detailed description of the specific process of the video processing system in the scenario of reinforcement learning and multimodal model training.
[0074] Figure 5 FIG. 1 is a structural diagram of a video processing system 10 provided in another embodiment of the present application. Figure 5 As shown, the video processing system 10 includes: a parsing layer 110 , a data cache layer 130 , a decoding layer 120 , a dynamic loading layer 140 and an adaptation layer 150 .
[0075] The parsing layer 110 is configured to parse the target video data and generate an index file containing a timestamp, video frame location information, and frame type (I / P). In the embodiment of the present application, the target video data is encoded using a fixed frame rate and only contains I / P frames and a timestamp field. The parsing layer generates an index file corresponding to the video frame. Exemplarily, the final target video data includes the original video file (mp4) and the corresponding index file (index).
[0076] The data cache layer 130 splits the modal data other than the target video data into multiple small blocks (Chunk1-Chunk4) and caches them to facilitate subsequent rapid retrieval of these data and improve video processing efficiency.
[0077] The decoding layer 120 includes multiple heterogeneous decoding engines (GPU0-GPU3), each of which contains Decoder1 and Decoder2, which are used to dynamically route the target video data according to metadata (resolution, format) to achieve batch decoding of video frames.
[0078] Dynamic loading layer 140 uses a buffering mechanism to asynchronously preload random video frames from the next batch while processing the video frames of the current training batch. Specifically, the core components of dynamic loading layer 140 include a video frame prefetcher and a cache file prefetcher, which are responsible for preloading video frames and caching them to improve processing efficiency.
[0079] The adaptation layer 150 converts the decoded video frames from the luminance-chrominance color space (Luminance (Y), Chrominance (U, V)) YUV format into a red, green, and blue color mode (Red Green Blue, RGB) tensor, and at the same time realizes data sharing with other mainstream training frameworks in the upper layer through the Deep Learning Data Structure (DLPack) standard protocol. Specifically, the adaptation layer 150 converts the decoded video frames into a format that can be directly used by the training framework, such as a tensor. Then, the decoded video frames are adapted to multiple frameworks for analysis or reasoning by the open source machine learning framework TensorFlow, the open source Python machine learning library (PyTorch), the scientific computing acceleration framework (Just AftereXecution, JAX), and the efficient and scalable deep learning framework (Mixed Network, MXNet). For example, video classification is implemented based on PyTorch, and target detection is implemented based on TensorFlow, opening up the link from video processing to model training.
[0080] Figure 6 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present application. Figure 6 As shown in Figure 1, the training framework (such as PyTorch) is the core of the entire model training process. It is responsible for initiating data requests and executing model training tasks. Specifically, the PyTorch framework includes a component called Dataset, which is responsible for requesting data from the Prefetch process so that model training can be performed based on this data.
[0081] The Prefetch process is a component specifically used to prepare data in advance. Its main function is to preload the next batch of video frames while the training framework is processing the current batch of video frames, thereby effectively reducing the waiting time caused by data loading. Specifically, when the Prefetch process receives a data request from the Dataset component, it will start to obtain video data from the decoding pipeline. The decoding pipeline is responsible for reading the metadata information of the video data from high-performance storage. This metadata information can help the decoding pipeline understand the structure of the video data and the specific location of the target video frame.
[0082] After obtaining the video data's metadata, the decoding pipeline further analyzes this information and, based on the analysis results, distributes the target video data to the corresponding decoder for processing. Furthermore, the decoding pipeline uses a seek operation to locate the desired target video frame within the video file. It then parses the target video frame into YUV format and converts the YUV format to RGB, making the video data more suitable for subsequent model training.
[0083] In some cases, if the training framework requires text data, the Prefetch process will also read the corresponding text files from high-performance storage and return them to the Prefetch process. The Prefetch process then combines the target video frames and text files to form a complete training data batch and returns this combined data to the training framework. This allows the training framework to train the model based on the received data, effectively utilizing this data to improve model performance.
[0084] This example proposes a random video frame loading solution based on hybrid encoding and decoding technology and data format optimization. This solution can efficiently load and process video data and multimodal data, providing strong support for the training of embodied intelligence models. In this way, we can effectively solve the bottleneck problem encountered in the video data loading and decoding process, thereby significantly improving the efficiency and quality of model training.
[0085] Figure 7 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present application. Figure 7 As shown, the video processing method includes the following steps.
[0086] Step S710: parse the target video data to generate index information of the target video data.
[0087] Specifically, the index information is indicative information generated after parsing the target video data. In this embodiment, the index information includes position information of video frames in the target video data. Exemplarily, the position information may be the physical storage location of each video frame of the target video data in the video file, such as a byte offset.
[0088] In one implementation, the encapsulation format of the target video data is parsed to separate components such as the video stream, audio stream, and metadata; each video frame in the video stream is traversed, and the starting byte offset of each video frame of the target video data in the target video data is recorded, thereby generating index information of the target video data.
[0089] Step S720 : Locate the target video frame from the target video data based on the index information, and dispatch the target video frame to the target decoding engine for decoding.
[0090] Specifically, based on the index information, the physical storage location of the target video frame in the target video data is determined, and the target decoding engine decodes the video data of the target video frame to output the decoded video data of the target video frame. Alternatively, only the storage location of the target video frame is notified to the target decoding engine, and then the target decoding engine directly jumps to the target video frame to begin decoding.
[0091] In the embodiments, the generation of index information provides clear guidance for the decoding process, preventing the decoding layer from blindly searching for the target video frame frame by frame within the entire target video data, thereby significantly improving decoding efficiency and shortening the decoding time. Furthermore, the precise location of the target video frame enables the video processing system to more quickly respond to processing requirements for specific video frames. The decoding layer then locates the target video frame within the target video data based on the index information and dispatches it to the target decoding engine for decoding. This avoids processing delays caused by insufficient decoding capabilities and effectively improves computing resource utilization and decoding efficiency.
[0092] Below, reference Figure 8 To describe the electronic device according to the embodiment of the present application. Figure 8 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.
[0093] like Figure 8 As shown, the electronic device 80 includes one or more processors 801 and a memory 802 .
[0094] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 80 to perform desired functions.
[0095] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the video processing method of any embodiment of the present application and / or other desired functions. Various contents such as target video data, video frames in the target video data, index information, target video frames, etc. may also be stored in the computer-readable storage medium.
[0096] In one example, the electronic device 80 may further include an input device 803 and an output device 804 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0097] The input device 803 may include, for example, a keyboard, a mouse, and the like.
[0098] The output device 804 can output various information to the outside, including target video data, video frames in the target video data, index information, target video frames, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.
[0099] Of course, to simplify, Figure 8 Only some of the components related to the present application in the electronic device 80 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 80 may further include any other appropriate components according to specific application scenarios.
[0100] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the video processing method according to various embodiments of the present application described above in this specification.
[0101] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0102] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the video processing method according to various embodiments of the present application described above in this specification.
[0103] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0104] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.
[0105] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0106] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.
[0107] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0108] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
Claims
1. A video processing system, characterized in that: include: a parsing layer configured to parse the target video data and generate index information of the target video data, wherein the index information includes position information of video frames in the target video data in the target video data; The decoding layer is configured to locate a target video frame from the target video data based on the index information, and schedule the target video frame to a target decoding engine for decoding.
2. The video processing system according to claim 1, wherein: The number of the target video data is multiple, and the decoding layer includes a multi-channel heterogeneous decoding engine; The decoding layer is also configured to determine the target decoding engine corresponding to the target video data from the multi-channel heterogeneous decoding engines based on the metadata of each target video data, so as to schedule the target video frame of the target video data to the target decoding engine for decoding, the metadata including the video data format and / or resolution.
3. The video processing system according to claim 1, wherein: The decoding layer is also configured to determine the video frame decoding task and the video frame consumption task corresponding to the target video frame, and construct a task concurrency architecture between the video frame decoding task and the video frame consumption task so as to execute the video frame decoding task and the video frame consumption task in parallel.
4. The video processing system according to claim 3, wherein: The task concurrency architecture is constructed based on multiple CUDA Streams, and the video frame decoding task and the video frame consumption task are each executed in a different CUDA Stream.
5. The video processing system according to claim 1, wherein: Also includes: The data cache layer is configured to divide other modal data other than video data into multiple data blocks based on the target data granularity, cache the multiple data blocks and generate index information for each of the multiple data blocks, so that when executing the video frame consumption task corresponding to the target video frame, the target data block is located from the multiple data blocks based on the index information for each of the multiple data blocks, so as to coordinate the loading of the target video frame and the target data block.
6. The video processing system according to claim 1, wherein: Also includes: The dynamic loading layer is configured to asynchronously preload a new target video frame that the video frame consumption task needs to consume when the video frame consumption task corresponding to the target video frame processes the target video frame.
7. The video processing system according to claim 1, wherein: Also includes: The adaptation layer is configured to convert the target video frame into a target data structure required by the video frame consumption task corresponding to the target video frame; and / or, based on a cross-task data description protocol, enable the converted target video frame to be shared between different video frame consumption tasks.
8. The video processing system according to any one of claims 3 to 7, characterized in that: The video frame consumption task includes a model training task.
9. The video processing system according to any one of claims 1 to 4, characterized in that: The index information further includes a timestamp of a video frame in the target video data, and / or a frame type of a video frame in the target video data; The target video data is encoded at a fixed frame rate, and / or the video frames in the target video data only include key frames and forward prediction frames, the key frames represent independent encoded frames containing complete image information, and the forward prediction frames represent encoded frames obtained by motion compensation prediction based on the temporally preceding video frames.
10. A video processing method, characterized in that: include: Parsing target video data to generate index information of the target video data, wherein the index information includes position information of video frames in the target video data in the target video data; Based on the index information, a target video frame is located from the target video data, and the target video frame is dispatched to a target decoding engine for decoding.
11. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the video processing method according to claim 10.
12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the video processing method according to claim 10.
13. A computer program product, characterized in that The computer program product comprises instructions, which, when executed on an electronic device, enable the electronic device to implement the video processing method according to claim 10 .