A high-performance distributed combined real-time processing method for multiple videos

Abstract

The present invention relates to a high-performance distributed combined multi-channel video real-time processing method, belonging to the technical field of video processing. The method includes the following steps: A. Construct a video processing pipeline, which includes a video decoding module, a video pre-processing module, a video model inference module, and a video post-processing module; B. Start multiple processes / threads: Each processing module starts in a multi-process manner, and each process starts multiple threads; the number of processes / threads started by each module is jointly determined according to the needs of the business scenario, the model performance, and the limitations of software and hardware resources; C. Construct a shared queue between upstream and downstream modules and set a data access strategy. In the present invention, the video decoding, pre-processing, model inference, and post-processing modules are all separated, which is simple to use. The specially designed data access mechanism between adjacent modules can significantly improve the multi-channel video processing efficiency.

Description

Technical Field

[0001] The present invention relates to a high-performance distributed combined multi-channel video real-time processing method, belonging to the technical field of video processing. Background Art

[0002] In many scenarios such as intelligent security and autonomous driving, it is a basic and very important task to perform multi-dimensional analysis of video for voice, text, face, object, and scene, and automatically extract some specific events occurring therein or specific behaviors of monitoring targets. In actual applications, the real-time performance of video processing and analysis and the number of concurrent processing channels are two very key indicators. NVIDIA provides a set of data flow analysis tools, DeepStream, which supports GPU hardware acceleration. Developers do not need to design an end-to-end solution, but only need to focus on building the core deep learning network for video analysis. However, the DeepStream tool also has some deficiencies. Since each module is tightly bound, it is extremely cumbersome to dynamically delete / add / replace plugins in the pipeline or modify the module functions. Summary of the Invention

[0003] The purpose of the present invention is to overcome the problems existing in the prior art and provide a high-performance distributed combined multi-channel video real-time processing method. The video decoding, pre-processing, model inference, and post-processing modules are all separated, which is easy to use. The specially designed data access mechanism between adjacent modules can significantly improve the multi-channel video processing efficiency.

[0004] To solve the above problems, a high-performance distributed combined multi-channel video real-time processing method of the present invention includes the following steps:

[0005] A. Construct a video processing pipeline, where the video processing pipeline includes a video decoding module, a video pre-processing module, a video model inference module, and a video post-processing module;

[0006] B. Start multiple processes / threads: Each processing module starts in a multi-process manner, and each process starts multiple threads; the number of processes / threads started by each module is jointly determined according to business scenario requirements, model performance, and software and hardware resource limitations;

[0007] C. Construct a shared queue between upstream and downstream modules and set a data access strategy.

[0008] Further, the starting of multiple processes / threads includes: setting the number of video source channels as N0; the number of decoding module processes as N1, and the number of threads started by the i-th decoding process as M 1i ; the number of pre-processing module processes as N2, and the number of threads started by the i-th pre-processing process as M 2i; The number of processes of the model inference module is N3, and the number of threads started by the i-th model inference process is M 3i ; The number of processes of the post-processing module is N4, and the number of threads started by the i-th post-processing process is M 4i .

[0009] Further, step C specifically includes the following steps:

[0010] C1. Video source data reading: Only considering the case where each video can be decoded by only one process / thread and one process / thread can decode multiple videos, then

[0011] Let If it satisfies where P ∈ [0, N1 - 1], then distribute the k-th video to the thread of the (P + 1)-th decoding process;

[0012] C2. Data access between upstream and downstream module processes:

[0013] C21. Setting the number of shared queues:

[0014] Denote the number of queues between module j and module j + 1 as q j，j+1 , then it is required that q j，j+1 ≥ max{N j , N j+1}, to avoid multiple processes operating on the same queue simultaneously, thus generating system switching overhead;

[0015] C22. Designing the data access strategy;

[0016] C23. Batch inference of the model.

[0017] Further, C22 specifically includes the following steps: 1) q j，j+1 = N j Case:

[0018] The upstream module processes are bound to the shared queues one by one, and the data processed by each upstream module process is stored in its own dedicated shared queue. The downstream module processes use the pull method to poll and read data from the shared queues;

[0019] 2) q j，j+1 = N j+1 Case:

[0020] The upstream module processing processes use the push method to store the processed result data into the shared queues, so one upstream module processing process corresponds to multiple shared queues;

[0021] 3) q j，j+1 > max{N j, N j+1} Case:

[0022] The number of processes in the upstream and downstream modules is less than the number of shared queues. In this case, a high-performance distributed combination method is used for data access.

[0023] Furthermore, C23 specifically includes the following operation steps:

[0024] 1) Create a temporary list and set the batch inference data size batch_size;

[0025] 2) Read data from the input queue in a non-waiting manner. If successful, save the data to the temporary list and proceed to step 3); otherwise, jump to step 4);

[0026] 3) Determine whether the length of the list is equal to batch_size. If so, proceed to step 4); otherwise, repeat step 2);

[0027] 4) Perform model batch inference on the data in the temporary list;

[0028] 5) Empty the data in the temporary list and repeat step 2).

[0029] Furthermore, the tools used by the decoding module include FFmpeg and VideoProcessingFramework.

[0030] The beneficial effects of the present invention are: 1) The processing modules are decoupled and easy to operate. In this method, the video decoding, pre-processing, model processing, and post-processing modules involved are independent of each other. Therefore, operations such as adding, deleting, and modifying modules are easy to implement.

[0031] 2) High processing efficiency. Each processing module starts in a multi-process manner, and each process can start multiple threads. The specially designed high-performance distributed combination data access method between the upstream and downstream modules maximizes the multi-channel video processing efficiency.

[0032] 3) Strong scalability. This method does not limit the implementation language, and users can choose according to their needs, such as the Python language with a low entry threshold, or the efficient C language, or a combination of multiple languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the multi-channel video real-time processing method with high-performance distributed combination of the present invention;

[0034] Figure 2 is a flow chart of video decoding with multi-process / thread enabled in the present invention;

[0035] Figure 3It is the access data flow chart of the multi - process pull mode in the present invention;

[0036] Figure 4 It is the access data flow chart of the multi - process push mode in the present invention;

[0037] Figure 5 It is the access data flow chart of the multi - process combination mode in the present invention;

[0038] Figure 6 It is the model batch inference flow chart in the present invention. Detailed implementation manners

[0039] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0040] As Figure 1 shown, the high - performance distributed combined multi - channel video real - time processing method of the present invention includes the following steps:

[0041] A. Construct a video processing pipeline, and the video processing pipeline includes a video decoding module, a video pre - processing module, a video model inference module, and a video post - processing module; the tools used by the decoding module include FFmpeg and VideoProcessingFramework.

[0042] The video pre - processing module mainly performs some transformation operations on video frames according to business and model inference needs, such as image grayscale conversion, pixel value standardization, and image scaling. The main deep - learning model is embedded in the video model inference module, and it analyzes video content according to business scenario needs, such as object detection, pedestrian tracking, pose estimation, and action recognition. The video post - processing module mainly further analyzes and processes the model inference results, such as NMS operation in object detection and business logic judgment.

[0043] The present invention can call the same basic module multiple times according to business needs. For example, two model inference modules are continuously called, and the result of the previous video model inference module is used as the input of the next video model inference module.

[0044] B. Start multi - processes / threads: Each processing module starts in a multi - process mode, and each process starts multiple threads; the number of processes / threads started by each module is jointly determined according to business scenario needs, model performance, and software and hardware resource limitations;

[0045] The start of multi - processes / threads includes: Let the number of video source channels be N0; the number of decoding processes be N1, and the number of threads started by the i - th decoding process be M 1i; The number of processes of the pre - processing module is N2, and the number of threads started by the i - th pre - processing process is M 2i ; The number of processes of the model inference module is N3, and the number of threads started by the i - th model inference process is M 3i ; The number of processes of the post - processing module is N4, and the number of threads started by the i - th post - processing process is M 4i 。

[0046] C. Construct a shared queue between upstream and downstream modules and set data access and storage policies;

[0047] Step C specifically includes the following steps:

[0048] C1. Video source data reading: Only consider the situation where each video can be decoded by only one process / thread, and one process / thread can decode multiple videos. Then

[0049] Let If it satisfies where P ∈ [0, N1 - 1], then distribute the k - th video to the th thread of the (P + 1) - th decoding process; For example, for 8 video sources, when 2 decoding processes are started and each process starts 2 threads, the video stream data reading method is as Figure 2 shown.

[0050] C2. Data access and storage between upstream and downstream module processes:

[0051] C21. Setting the number of shared queues:

[0052] Record the number of queues between module j and module j + 1 as q j，j+1 , then it is required that q j，j+1 ≥ max{N j , N j+1}, to avoid multiple processes operating on the same queue simultaneously, thus generating system switching overhead, where N j represents the maximum number of processes that module j can concurrently process at the same time, N j+1 represents the maximum number of processes that module j + 1 can concurrently process at the same time, max{N j , N j+1} represents the maximum number of processes of the module with stronger concurrent processing ability between module j and module j + 1;

[0053] C22. Design of data access and storage policies; When accessing and storing data, the modulo - remainder rule is adopted.

[0054] C22 specifically includes the following steps: 1) q j，j+1 = N j Case:

[0055] The upstream module processes are bound to the shared queues one by one. The data processed by each upstream module process is stored in its own dedicated shared queue, and the downstream module processes use the pull method to poll and read data from the shared queues.

[0056] Suppose there are 4 upstream module processes, 2 downstream module processes, and 4 shared queues. Then the access method is as Figure 3 shown;

[0057] Adopting the non-waiting mode can further improve the data reading efficiency. For example, the implementation in the Python language is as follows:

[0058] while True:

[0059] try:

[0060] input_data = queue.get_nowait()

[0061] except:

[0062] pass.

[0063] 2) q j,j+1 = N j+1 Case:

[0064] The upstream module processing processes use the push method to store the processed result data in the shared queue. Therefore, one upstream module processing process corresponds to multiple shared queues. Suppose there are 2 upstream module processes, 4 downstream module processes, and 4 shared queues. The data access method is as Figure 4 shown.

[0065] 3) q j,j+1 > max{N j , N j+1} Case:

[0066] The number of upstream and downstream module processes is less than the number of shared queues. At this time, a high-performance distributed combination method is used for data access. Suppose there are 2 upstream module processes, 2 downstream module processes, and 4 shared queues. Then the data access method is as Figure 5 shown.

[0067] C23. Model batch inference; Adopting the batch inference mechanism to further accelerate the model inference speed and reduce the possible data backlog in the input queue, as Figure 6 shown.

[0068] C23 specifically includes the following operation steps:

[0069] 1) Establish a temporary list and set the batch inference data size batch_size;

[0070] 2) Read data from the input queue in a non-waiting manner. If successful, save the data to a temporary list and proceed to step 3); otherwise, jump to step 4).

[0071] 3) Determine whether the length of the list is equal to batch_size. If so, proceed to step 4); otherwise, repeat step 2).

[0072] 4) Perform batch inference on the data in the temporary list.

[0073] 5) Clear the data in the temporary list and repeat step 2).

[0074] The present invention constructs each processing module by adopting a multi-process method and ingeniously designs the data access strategy between adjacent module processes. While ensuring that the video stream is not disordered, it improves the video processing speed as much as possible to meet the needs of real-time applications. In addition, the implementation of this method is not limited to a specific language and has strong scalability. The number of processes enabled for each module and the number of threads enabled for each process can be arbitrarily configured according to needs.

[0075] The processing method of the present invention decouples modules such as video decoding, pre-processing, model inference, and post-processing, and focuses on designing the data access method between different modules. By editing the functions of the corresponding modules, video content analysis such as object detection and tracking, pose estimation, and action recognition can be performed.

[0076] Inspired by the ideal embodiments of the present invention described above, through the above description, relevant staff can make various changes and modifications without departing from the technical idea of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A high-performance distributed combined multi-channel video real-time processing method, characterized in that, It includes the following steps: A. Construct a video processing pipeline, which includes a video decoding module, a video pre-processing module, a video model inference module, and a video post-processing module; B. Start multi-processes / threads: Each processing module starts in a multi-process manner, and each process starts multiple threads; The number of processes / threads started by each module is jointly determined according to the needs of the business scenario, the model performance, and the limitations of software and hardware resources; C. Construct a shared queue between upstream and downstream modules and set the data access and storage strategy; The multi-process / thread startup includes: setting the number of video source channels as N0; the number of decoding processes as N1, and the number of threads started by the i-th decoding process as M 1i ; Step C specifically includes the following steps: C1. Video source data reading: Only consider the case where each video can be decoded by only one process / thread, and one process / thread can decode multiple videos. Then Let If it satisfies where P ∈ [0, N1 - 1], then the k-th video is distributed to the thread of the (P + 1)-th decoding process; C2. Data access and storage between upstream and downstream module processes: C21. Setting the number of shared queues: Let the number of queues between module j and module j + 1 be q j,j+1 , then it is required that q j,j+1 ≥ max{N j , N j+1}, to avoid multiple processes operating on a queue simultaneously, thereby incurring system switching overhead. Among them, N j represents the maximum number of processes that module j can concurrently handle at the same time, and N j+1 represents the maximum number of processes that module j + 1 can concurrently handle at the same time. max{N j , N j+1} represents the maximum number of processes of the module with stronger concurrent processing ability among module j and module j + 1; C22. Design of the data access and storage strategy; C23. Batch inference of the model; C22 specifically includes the following steps: 1) q j,j+1 = N j Case: The upstream module processes are bound to the shared queues one by one. The data processed by each upstream module process is stored in its own dedicated shared queue, and the downstream module processes poll and read data from the shared queues in a pulling manner; 2)q j,j+1 = N j+1 Case: The upstream module processing processes store the processed result data in the shared queues in a pushing manner. Therefore, one upstream module processing process corresponds to multiple shared queues; 3)q j,j+1 > max{N j , N j+1}} case: The number of upstream and downstream module processes is less than the number of shared queues. In this case, a high-performance distributed combination method is used for data access and storage.

2. The high-performance distributed combined multi-channel video real-time processing method according to claim 1, wherein: C23 specifically includes the following operation steps: 1) Establish a temporary list and set the batch inference data size batch_size; 2) Read data from the input queue in a non-waiting manner. If successful, save the data to the temporary list and proceed to step 3), otherwise jump to step 4); 3) Determine whether the length of the list is equal to batch_size. If so, proceed to step 4), otherwise repeat step 2); 4) Perform batch inference of the model on the data in the temporary list; 5) Empty the data in the temporary list and repeat step 2).

3. The high-performance distributed combined multi-channel video real-time processing method according to claim 1, characterized in that: The tools used by the decoding module include FFmpeg and VideoProcessingFramework.

Images