Single video stream multi-algorithm binding method and system based on decoding frame multiplexing

Abstract

The invention provides a single video stream multi-algorithm binding method and system based on decoding frame multiplexing, and the method comprises the steps: carrying out the pulling of a video stream through a video processing node, obtaining video data, carrying out the decoding processing and color space conversion processing of the video data, and outputting a processed image frame; all reasoning nodes connected with the video processing nodes are bound with different types of algorithms respectively, so that the output processing image frames are copied to the corresponding reasoning nodes respectively, the processing image frames are subjected to pre-processing reasoning processing through the reasoning nodes, a reasoning result is obtained, and the reasoning result is subjected to restoration processing; reasoning results output by all reasoning nodes are sent to a kafka message queue through a post-processing node, and corresponding reasoning results are extracted from the kafka message queue and are forwarded to a server side, so that a video stream is only decoded once, and image frames obtained by decoding can be deeply copied according to the number of subsequent algorithms; and meanwhile, algorithm reasoning is carried out on the image frames independently by utilizing a plurality of algorithms, redundant calculation consumption of decoding is converted into data copy consumption of the decoded frames, multiple times of decoding of the same video frame is avoided, calculation redundancy is effectively reduced, and resource consumption is saved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a single video stream multi-algorithm binding method and system based on decoding frame multiplexing. Background Art

[0002] The current video processing workflow mainly includes opening the video stream through OpenCV's VideoCapture, reading data from the video stream frame by frame, decoding it into image frames, loading the pre-trained model, and pre-processing each frame of the image, such as converting the resolution and / or color space according to the input format of the model; finally, drawing boxes, markers and other visual information on the image frame according to the inference results of the model. However, the above method has the following problems: First, when a video stream needs to be processed by multiple algorithms, a complete decoding process needs to be performed for each algorithm. Since decoding is a computationally intensive operation, this will lead to a large amount of redundant calculations and increase the decoding calculation overhead; second, the redundant decoding process will consume a lot of computing resources such as CPU and host memory bandwidth. If GPU decoding is used, it will also consume GPU computing resources and device memory resources, making it impossible to access too many video streams. Summary of the invention

[0003] The object of the present invention is to provide a single video stream multi-algorithm binding method and system based on decoded frame multiplexing, which obtains video data by pulling a video stream through a video processing node, and performs decoding processing and color space conversion processing on the video data, so as to output processed image frames; bind different types of algorithms to all inference nodes connected to the video processing node, so as to copy the output processed image frames to corresponding inference nodes respectively, perform pre-processing inference processing on the processed image frames through the inference nodes, obtain inference results and restore the inference results; send the inference results output by all inference nodes to a kafka message queue through a post-processing node, extract the corresponding inference results from the kafka message queue and forward them to a server, so that the video stream is decoded only once, and the decoded image frames can be deeply copied according to the number of subsequent algorithms, and at the same time, multiple algorithms are used to independently perform algorithmic inference on the image frames, and the redundant calculation consumption of decoding is converted into data copy consumption of decoded frames, so as to avoid multiple decoding of the same video frame, effectively reduce calculation redundancy and save resource consumption, facilitate allocation of resources to other tasks, and improve resource utilization efficiency.

[0004] The present invention is achieved through the following technical solutions:

[0005] A single video stream multi-algorithm binding method based on decoded frame multiplexing includes:

[0006] Pull a video stream through a video processing node to obtain corresponding video data; decode and convert the video data into a color space to obtain a corresponding processed image frame;

[0007] Binding different types of algorithms to all the inference nodes connected to the video processing node, and then copying the processed image frames output by the video processing node to the corresponding inference nodes; performing pre-processing inference processing on the processed image frames through the inference nodes to obtain corresponding inference results, and restoring the inference results;

[0008] The inference results output by all inference nodes are sent to the Kafka message queue through the post-processing node. Based on the image data acquisition requirements of the server connected to the post-processing node, the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0009] Optionally, a video stream is pulled through a video processing node to obtain corresponding video data; the video data is decoded and color space converted to obtain corresponding processed image frames, including:

[0010] The video source connected to the video processing node is monitored to obtain the video stream URL of the video source; based on the video stream URL, a single-channel video stream pulling process is performed on the video source; the video data obtained by the video stream pulling process is identified to obtain corresponding video code format information and image color feature information;

[0011] Based on the video code format information, the video data is decoded to obtain a corresponding decoded image frame; and based on the image color feature information, the decoded image frame is converted into a color space to obtain a corresponding processed image frame.

[0012] Optionally, a video stream is pulled through a video processing node to obtain corresponding video data, including:

[0013] Step S1, using the following formula (1), according to the processing status of the video processing node, the video stream pulling efficiency is obtained:

[0014]

[0015] In the above formula (1), E v Indicates the video stream pulling efficiency; D b Indicates bandwidth, that is, the maximum amount of data transmitted by the video stream, in bits per second; T r Indicates the transmission time of the video stream in seconds; L s Indicates the transmission delay of the video stream in seconds; Np Indicates the number of processing nodes, that is, the number of nodes that process video streams simultaneously; C p Indicates the processing delay of each processing node in seconds, that is, the time required for the node to process video data;

[0016] Step S2, using the following formula (2), according to the video stream pulling efficiency, obtain the video data quality score:

[0017]

[0018] In the above formula (2), Q v Represents the video data quality score; R i represents the i-th video quality index; W i Representation and video quality index R i The weight coefficient of M1 is the total amount of video data, that is, the data size of the entire video stream; V represents the jth video quality loss factor; , Denotes the video loss factor F , The weight coefficient of m represents the number of video quality indicators; n represents the number of video quality loss factors;

[0019] Step S3, using the following formula (3), according to the video stream pulling efficiency and the video data quality score, obtain the control factor of dynamic control and optimization, so as to pull the video stream to obtain the corresponding video data.

[0020]

[0021] In the above formula (3), C . represents a control factor, which is used to adjust the video stream pulling strategy, and the pulling strategy includes adjusting the pulling bandwidth and / or adjusting the number of processing nodes; β represents the priority factor of the video stream, which is between 0 and 1 and is the processing priority of different video streams; α represents the adjustment coefficient; T B Represents the preset video stream pulling efficiency threshold; e represents a natural constant.

[0022] Optionally, all the inference nodes connected to the video processing node are respectively bound to different types of algorithms, and then the processed image frames output by the video processing node are respectively copied to the corresponding inference nodes; the processed image frames are pre-processed by the inference nodes to obtain corresponding inference results, and the inference results are restored, including:

[0023] Obtain computing power resource information of all inference nodes connected to the video processing node, and determine the algorithms matched by all inference nodes based on the computing power resource information, so as to bind and load different types of algorithms to all inference nodes respectively; obtain working status information of each inference node after binding and loading the algorithm, and screen available inference nodes capable of performing image frame inference based on the working status information;

[0024] Based on the number of all available reasoning nodes, the processed image frames output by the video processing node are copied in corresponding numbers and transmitted to the corresponding available reasoning nodes respectively; the resolution of the copied image frames is first proportionally scaled by the available reasoning nodes, and then the copied image frames are subjected to corresponding algorithm reasoning processing to obtain reasoning results, and the reasoning results are proportionally restored.

[0025] Optionally, the inference results output by all inference nodes are sent to a Kafka message queue by a post-processing node, and based on the image data acquisition requirements of a server connected to the post-processing node, corresponding inference results are extracted from the Kafka message queue and forwarded to the server, including:

[0026] The post-processing node identifies the image content attributes of the inference results output by all inference nodes to obtain the image content attribute information of all inference results, and based on the image content attribute information, sends all the inference results to the Kafka message queue in sequence;

[0027] The image content attribute information of all inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0028] A single video stream multi-algorithm binding system based on decoded frame multiplexing, including:

[0029] The video processing node is used to pull a video stream to obtain corresponding video data; decode and convert the video data into a color space to obtain corresponding processed image frames;

[0030] A plurality of inference nodes, each of which is connected to the video processing node and is respectively bound to different types of algorithms, and the processed image frames output by the video processing node are copied to the corresponding inference nodes respectively; the inference nodes perform pre-processed inference processing on the processed image frames to obtain corresponding inference results, and restore the inference results;

[0031] The post-processing node is used to send the inference results output by all inference nodes to the Kafka message queue, and based on the image data acquisition requirements of the server connected to the post-processing node, extract the corresponding inference results from the Kafka message queue and forward them to the server.

[0032] Optionally, the video processing node is used to pull a video stream to obtain corresponding video data; and decode and color space convert the video data to obtain corresponding processed image frames, including:

[0033] Monitor the video source connected to it to obtain the video stream URL of the video source; based on the video stream URL, perform a single-channel video stream pulling process on the video source; identify the video data obtained by the video stream pulling process to obtain corresponding video code format information and image color feature information;

[0034] Based on the video code format information, the video data is decoded to obtain a corresponding decoded image frame; and based on the image color feature information, the decoded image frame is converted into a color space to obtain a corresponding processed image frame.

[0035] Optionally, several inference nodes are respectively bound to different types of algorithms, and the processed image frames output by the video processing node are respectively copied to corresponding inference nodes; the inference nodes perform pre-processed inference processing on the processed image frames to obtain corresponding inference results, and restore the inference results, including:

[0036] Obtain the computing power resource information of all inference nodes, and determine the algorithms matched by all inference nodes based on the computing power resource information, so as to bind and load different types of algorithms to all inference nodes respectively; obtain the working status information of each inference node after binding and loading the algorithm, and screen the available inference nodes that can perform image frame inference based on the working status information;

[0037] Based on the number of all available reasoning nodes, the processed image frames output by the video processing node are copied in corresponding numbers and transmitted to the corresponding available reasoning nodes respectively; the resolution of the copied image frames is first proportionally scaled by the available reasoning nodes, and then the copied image frames are subjected to corresponding algorithm reasoning processing to obtain reasoning results, and the reasoning results are proportionally restored.

[0038] Optionally, the post-processing node is used to send the inference results output by all inference nodes to a Kafka message queue, and based on the image data acquisition requirements of a server connected to the post-processing node, extract the corresponding inference results from the Kafka message queue and forward them to the server, including:

[0039] The post-processing node identifies the image content attributes of the inference results output by all inference nodes to obtain the image content attribute information of all inference results, and based on the image content attribute information, sends all the inference results to the Kafka message queue in sequence;

[0040] The image content attribute information of all inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The single video stream multi-algorithm binding method and system based on decoded frame multiplexing provided in the present application obtain video data by pulling a video stream through a video processing node, and decode and convert the video data into a color space, so as to output a processed image frame; bind different types of algorithms to all inference nodes connected to the video processing node, so as to copy the output processed image frames to the corresponding inference nodes, perform inference processing on the processed image frames through the inference nodes to obtain the inference results and restore the inference results; send the inference results output by all inference nodes to the Kafka message queue through the post-processing node, extract the corresponding inference results from the Kafka message queue and forward them to the server, so that the video stream is decoded only once, and the decoded image frames can be deeply copied according to the number of subsequent algorithms, and multiple algorithms are used to independently perform algorithmic inference on the image frames, and the redundant computing consumption of decoding is converted into data copy consumption of decoded frames, so as to avoid multiple decoding of the same video frame, effectively reduce computing redundancy and save resource consumption, facilitate the allocation of resources to other tasks, and improve resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0044] Figure 1 A schematic flow chart of a single video stream multi-algorithm binding method based on decoding frame multiplexing provided by the present invention.

[0045] Figure 2 A structural schematic diagram of a single video stream multi-algorithm binding system based on decoding frame multiplexing provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] See also Figure 1 As shown, a single video stream multi-algorithm binding method based on decoding frame multiplexing is provided in one embodiment of the present application. The single video stream multi-algorithm binding method based on decoding frame multiplexing includes:

[0050] Pull a video stream through a video processing node to obtain corresponding video data; decode and convert the video data into a color space to obtain a corresponding processed image frame;

[0051] Bind different types of algorithms to all the inference nodes connected to the video processing node, and then copy the processed image frames output by the video processing node to the corresponding inference nodes; perform pre-processed inference processing on the processed image frames through the inference nodes to obtain corresponding inference results, and restore the inference results;

[0052] The inference results output by all inference nodes are sent to the Kafka message queue through the post-processing node. Based on the image data acquisition requirements of the server connected to the post-processing node, the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0053] The beneficial effects of the above embodiments are as follows: the single video stream multi-algorithm binding method based on decoded frame multiplexing obtains video data by pulling a video stream through a video processing node, and performs decoding processing and color space conversion processing on the video data, so as to output processed image frames; different types of algorithms are respectively bound to all inference nodes connected to the video processing node, so as to copy the output processed image frames to the corresponding inference nodes respectively, and perform inference processing on the processed image frames through the inference nodes to obtain inference results and restore the inference results; the inference results output by all inference nodes are sent to the Kafka message queue through the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server, so that the video stream is decoded only once, and the decoded image frames can be deeply copied according to the number of subsequent algorithms, and multiple algorithms are used to independently perform algorithmic inference on the image frames, and the redundant computing consumption of decoding is converted into data copy consumption of decoded frames, so as to avoid multiple decoding of the same video frame, effectively reduce computing redundancy and save resource consumption, facilitate the allocation of resources to other tasks, and improve resource utilization efficiency.

[0054] In another embodiment, a video stream is pulled through a video processing node to obtain corresponding video data; the video data is decoded and color space converted to obtain corresponding processed image frames, including:

[0055] The video source connected to the video processing node is monitored to obtain the video stream URL of the video source; based on the video stream URL, a single video stream is pulled for the video source; the video data obtained by the video stream pulling process is identified to obtain the corresponding video code format information and image color feature information;

[0056] Based on the video code format information, the video data is decoded to obtain a corresponding decoded image frame; and based on the image color feature information, the decoded image frame is converted into a color space to obtain a corresponding processed image frame.

[0057] The beneficial effects of the above embodiments are as follows: the video processing node is a node unit for first-hand decoding of the video stream, which is connected to the video source and is used to directly obtain the corresponding video stream data from the video source. In order to ensure the accuracy of the acquisition of the video stream data, the RF source is monitored by the video processing node to obtain the video stream URL of the video source, so that the source of the video stream can be accurately calibrated, and then the video stream URL is used as a reference to perform a single-channel video stream pull process on the video source, thereby improving the efficiency of video stream pull. In addition, the video data obtained by the video stream pull process is identified to obtain the video code format information and image color feature information of the video data, providing a reliable basis for subsequent decoding and color space conversion. Generally speaking, the decoded image frame obtained by the decoding process is usually a YUV image or NV12 image, and the format of the image frame input by the algorithm model is usually RGB or BGR. For this reason, the NV12 or YUV420P image frame is converted into an RGB or BGR image frame through color space conversion processing, so that the image frame output by the video processing node can be directly input into the inference node for processing in the future, and there is no need to convert the color format of the image frame before the algorithm reasoning, thereby improving the efficiency of the algorithm reasoning.

[0058] In another embodiment, a video stream is pulled by a video processing node to obtain corresponding video data, including:

[0059] Step S1, using the following formula (1), according to the processing status of the video processing node, the video stream pulling efficiency is obtained:

[0060]

[0061] In the above formula (1), E v Indicates the video stream pulling efficiency; D b Indicates bandwidth, that is, the maximum amount of data transmitted by the video stream, in bits per second; T r Indicates the transmission time of the video stream in seconds; L s Indicates the transmission delay of the video stream in seconds; N p Indicates the number of processing nodes, that is, the number of nodes that process video streams simultaneously; C p Indicates the processing delay of each processing node in seconds, that is, the time required for the node to process video data;

[0062] Step S2, using the following formula (2), according to the video stream pulling efficiency, obtain the video data quality score:

[0063]

[0064] In the above formula (2), Q v Represents the video data quality score; Ri represents the i-th video quality index, which includes clarity and fluency, etc.; W i Representation and video quality index R i M1 represents the total amount of video data, that is, the data size of the entire video stream; V represents the jth video quality loss factor, which includes noise and frame loss, etc. , Denotes the video loss factor F , The weight coefficient of m represents the number of video quality indicators; n represents the number of video quality loss factors;

[0065] Step S3, using the following formula (3), according to the video stream pulling efficiency and the video data quality score, obtain the control factor of dynamic control and optimization, so as to pull the video stream to obtain the corresponding video data.

[0066]

[0067] In the above formula (3), C . represents a control factor, which is used to adjust the video stream pulling strategy, which includes adjusting the pulling bandwidth and / or adjusting the number of processing nodes; β represents the priority factor of the video stream, which is between 0 and 1 and is the processing priority of different video streams; α represents the adjustment coefficient; T B represents the preset video stream pulling efficiency threshold; e represents a natural constant;

[0068] Through the above-mentioned control factors, the video stream pulling strategy can be optimized in real time. When the control factor is greater than or equal to the preset control factor threshold, it indicates that the quality and pulling efficiency of the video stream are good, and the current pulling strategy can continue to be maintained; when the control factor is less than the preset control factor threshold, it is necessary to adjust the pulling strategy, such as adjusting (for example, increasing or decreasing) the pulling bandwidth and / or adjusting (for example, increasing or decreasing) the number of processing nodes to improve the overall video stream processing efficiency.

[0069] The beneficial effects of the above embodiment are as follows: by using the above formula (1), according to the processing status of the video processing node, the video stream pulling efficiency is obtained; by calculating the relationship between the bandwidth and delay of the video stream, the efficiency of video stream pulling is evaluated; the influence of multiple processing nodes is taken into account to ensure that the bandwidth can be maximized in practical applications; then by using the above formula (2), according to the video stream pulling efficiency, the video data quality score is obtained; multiple indicators of video quality are comprehensively considered and adjusted by the pulling efficiency; by subtracting the influence of the loss factor, the quality score of the video data is finally obtained, so that the video processing can be optimized under the premise of ensuring quality; finally, by using the above formula (3), according to the video stream pulling efficiency and the video data quality score, the control factor of dynamic control and optimization is obtained, so that the video stream is pulled to obtain the corresponding video data, so that the overall performance of video stream processing can be effectively improved by adjusting the control factor to meet the needs of practical applications.

[0070] In another embodiment, different types of algorithms are respectively bound to all inference nodes connected to the video processing node, and then the processed image frames output by the video processing node are respectively copied to the corresponding inference nodes; the processed image frames are pre-processed by the inference nodes to obtain corresponding inference results, and the inference results are restored, including:

[0071] Obtain the computing power resource information of all the inference nodes connected to the video processing node, and determine the algorithms matched by all the inference nodes based on the computing power resource information, so as to bind and load different types of algorithms to all the inference nodes respectively; obtain the working status information of each inference node after binding and loading the algorithm, and based on the working status information, filter the available inference nodes that can perform image frame inference;

[0072] Based on the number of all available reasoning nodes, the processed image frames output by the video processing node are copied in corresponding numbers and transmitted to the corresponding available reasoning nodes respectively; the resolution of the copied image frames is first scaled proportionally through the available reasoning nodes, and then the copied image frames are subjected to corresponding algorithm reasoning processing to obtain reasoning results, and the reasoning results are restored proportionally.

[0073] The beneficial effect of the above embodiment is that the inference node acts as the operation carrier of different algorithm models, and its own computing power resources directly affect the operation feasibility of the algorithm model within it. The computing power resource information of all inference nodes is obtained, and the computing power resource information is analyzed to determine the algorithm matched by each inference node, so that all inference nodes are bound and loaded with different types of algorithms respectively, ensuring that each inference node can obtain the bound matching algorithm. In addition, the working status information of each inference node after binding and loading the algorithm is obtained, wherein the working status information may be but is not limited to the corresponding operation speed of the inference node during the operation of the loaded algorithm. When the operation speed is greater than or equal to the preset speed threshold, the corresponding inference node is determined as an available inference node that can perform image frame inference, otherwise, the corresponding inference node is not determined as an available inference node that can perform image frame inference, so that the selected available inference nodes can be ensured to perform image frame inference efficiently. Based on the number of all available inference nodes, the processed image frames output by the video processing node are copied in a corresponding number and transmitted to the corresponding available inference nodes respectively. The available inference nodes first scale the resolution of the copied image frames in proportion, and then perform the corresponding algorithm inference processing on the copied image frames to obtain the inference results, and then perform proportional restoration processing on the inference results to obtain accurate image frame inference results. Specifically, the inference node mainly performs inference on the frame images decoded by the video processing node and converted into color space, such as using the inference node bound to the YOLOv5s and YOLOv8n algorithms to perform inference to obtain target detection results, using the inference node bound to the YOLov7-W6 algorithm to perform inference to obtain human posture recognition results (for fall detection), and using the inference node bound to the Dlib Landmark algorithm to perform inference to obtain face posture recognition results. The result of the inference is some coordinate information, such as the coordinate information related to the detection box of the detected target for YOLOv5. Since the image resolution of the input frame may be different from the input resolution required by the model, different models may also have different requirements for the resolution of the image frame. For example, the input image resolution of the YOLOv5s model is 640x640, while the input image resolution of the YOLOv7-w6 model is 720*720, and the resolution of the decoded frame is 1920x1080. Therefore, each inference module needs to perform specific preprocessing work, scale the resolution of the decoded frame to 720x720, and fill the scaled "blank" area with grayscale, as shown in the figure below. After decoding, specific inference nodes need to be post-processed. For example, YOLOv5 will filter out the inference structure according to the set confidence and NMS value, and then restore the coordinates of the inference result in proportion.

[0074] In another embodiment, the inference results output by all inference nodes are sent to the Kafka message queue by the post-processing node, and based on the image data acquisition requirements of the server connected to the post-processing node, the corresponding inference results are extracted from the Kafka message queue and forwarded to the server, including:

[0075] The post-processing node identifies the image content attributes of the inference results output by all inference nodes, obtains the image content attribute information of all inference results, and sends all inference results to the Kafka message queue in sequence based on the image content attribute information;

[0076] The image content attribute information of all inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0077] The beneficial effect of the above embodiment is that the image content attribute recognition is performed on the inference results output by all inference nodes through the post-processing node, and the image content attribute information of each of the inference results is obtained, wherein the image content attribute information may be but is not limited to the content type information on the image frame screen corresponding to the inference result, and then all the inference results are sent to the kafka message queue in sequence based on the image content attribute information, so that all the inference results are accurately distinguished and arranged in the kafka message queue. Then the image content attribute information of all the inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and according to the matching results, the corresponding inference results are extracted from the kafka message queue and forwarded to the server, ensuring that other servers can accurately obtain the actually required inference results for further processing.

[0078] See also Figure 2 As shown, a single video stream multi-algorithm binding system based on decoding frame multiplexing is provided in one embodiment of the present application. The single video stream multi-algorithm binding system based on decoding frame multiplexing includes:

[0079] The video processing node is used to pull a video stream to obtain corresponding video data; decode and convert the video data into a color space to obtain corresponding processed image frames;

[0080] A plurality of inference nodes are connected to the video processing node and are respectively bound to different types of algorithms, and the processed image frames output by the video processing node are respectively copied to the corresponding inference nodes; the inference nodes perform pre-processed inference processing on the processed image frames to obtain corresponding inference results, and restore the inference results;

[0081] The post-processing node is used to send the inference results output by all inference nodes to the Kafka message queue. Based on the image data acquisition requirements of the server connected to the post-processing node, the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0082] The beneficial effects of the above embodiments are as follows: the single video stream multi-algorithm binding system based on decoded frame multiplexing obtains video data by pulling a video stream through a video processing node, and performs decoding processing and color space conversion processing on the video data, so as to output processed image frames; different types of algorithms are respectively bound to all inference nodes connected to the video processing node, so as to copy the output processed image frames to the corresponding inference nodes respectively, and perform pre-processing inference processing on the processed image frames through the inference nodes to obtain inference results and restore the inference results; the inference results output by all inference nodes are sent to the Kafka message queue through the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server, so that the video stream is decoded only once, and the decoded image frames can be deeply copied according to the number of subsequent algorithms, and multiple algorithms are used to independently perform algorithmic inference on the image frames, and the redundant computing consumption of decoding is converted into data copy consumption of decoded frames, so as to avoid multiple decoding of the same video frame, effectively reduce computing redundancy and save resource consumption, facilitate the allocation of resources to other tasks, and improve resource utilization efficiency.

[0083] In another embodiment, the video processing node is used to pull a video stream to obtain corresponding video data; the video data is decoded and color space converted to obtain corresponding processed image frames, including:

[0084] Monitor the video source connected to it to obtain the video stream URL of the video source; based on the video stream URL, perform a single-channel video stream pulling process on the video source; identify the video data obtained by the video stream pulling process to obtain the corresponding video code format information and image color feature information;

[0085] Based on the video code format information, the video data is decoded to obtain a corresponding decoded image frame; and based on the image color feature information, the decoded image frame is converted into a color space to obtain a corresponding processed image frame.

[0086] The beneficial effects of the above embodiments are as follows: the video processing node is a node unit for first-hand decoding of the video stream, which is connected to the video source and is used to directly obtain the corresponding video stream data from the video source. In order to ensure the accuracy of the acquisition of the video stream data, the RF source is monitored by the video processing node to obtain the video stream URL of the video source, so that the source of the video stream can be accurately calibrated, and then the video stream URL is used as a reference to perform a single-channel video stream pull process on the video source, thereby improving the efficiency of video stream pull. In addition, the video data obtained by the video stream pull process is identified to obtain the video code format information and image color feature information of the video data, providing a reliable basis for subsequent decoding and color space conversion. Generally speaking, the decoded image frame obtained by the decoding process is usually a YUV image or NV12 image, and the format of the image frame input by the algorithm model is usually RGB or BGR. For this reason, the NV12 or YUV420P image frame is converted into an RGB or BGR image frame through color space conversion processing, so that the image frame output by the video processing node can be directly input into the inference node for processing in the future, and there is no need to convert the color format of the image frame before the algorithm reasoning, thereby improving the efficiency of the algorithm reasoning.

[0087] In another embodiment, a plurality of inference nodes are respectively bound to different types of algorithms, and the processed image frames output by the video processing node are respectively copied to the corresponding inference nodes; the inference nodes perform pre-processed inference processing on the processed image frames to obtain corresponding inference results, and restore the inference results, including:

[0088] Obtain the computing power resource information of all inference nodes, and based on the computing power resource information, determine the algorithms that match all inference nodes, so as to bind and load different types of algorithms to all inference nodes; obtain the working status information of each inference node after binding and loading the algorithm, and based on the working status information, filter the available inference nodes that can perform image frame inference;

[0089] Based on the number of all available reasoning nodes, the processed image frames output by the video processing node are copied in corresponding numbers and transmitted to the corresponding available reasoning nodes respectively; the resolution of the copied image frames is first scaled proportionally through the available reasoning nodes, and then the copied image frames are subjected to corresponding algorithm reasoning processing to obtain reasoning results, and the reasoning results are restored proportionally.

[0090] The beneficial effect of the above embodiment is that the inference node acts as the operation carrier of different algorithm models, and its own computing power resources directly affect the operation feasibility of the algorithm model within it. The computing power resource information of all inference nodes is obtained, and the computing power resource information is analyzed to determine the algorithm matched by each inference node, so that all inference nodes are bound and loaded with different types of algorithms respectively, ensuring that each inference node can obtain the bound matching algorithm. In addition, the working status information of each inference node after binding and loading the algorithm is obtained, wherein the working status information may be but is not limited to the corresponding operation speed of the inference node during the operation of the loaded algorithm. When the operation speed is greater than or equal to the preset speed threshold, the corresponding inference node is determined as an available inference node that can perform image frame inference, otherwise, the corresponding inference node is not determined as an available inference node that can perform image frame inference, so that the selected available inference nodes can be ensured to perform image frame inference efficiently. Based on the number of all available inference nodes, the processed image frames output by the video processing node are copied in a corresponding number and transmitted to the corresponding available inference nodes respectively. The available inference nodes first scale the resolution of the copied image frames in proportion, and then perform the corresponding algorithm inference processing on the copied image frames to obtain the inference results, and then perform proportional restoration processing on the inference results to obtain accurate image frame inference results. Specifically, the inference node mainly performs inference on the frame images decoded by the video processing node and converted into color space, such as using the inference node bound to the YOLOv5s and YOLOv8n algorithms to perform inference to obtain target detection results, using the inference node bound to the YOLov7-W6 algorithm to perform inference to obtain human posture recognition results (for fall detection), and using the inference node bound to the Dlib Landmark algorithm to perform inference to obtain face posture recognition results. The result of the inference is some coordinate information, such as the coordinate information related to the detection box of the detected target for YOLOv5. Since the image resolution of the input frame may be different from the input resolution required by the model, different models may also have different requirements for the resolution of the image frame. For example, the input image resolution of the YOLOv5s model is 640x640, while the input image resolution of the YOLOv7-w6 model is 720*720, and the resolution of the decoded frame is 1920x1080. Therefore, each inference module needs to perform specific preprocessing work, scale the resolution of the decoded frame to 720x720, and fill the scaled "blank" area with grayscale, as shown in the figure below. After decoding, specific inference nodes need to be post-processed. For example, YOLOv5 will filter out the inference structure according to the set confidence and NMS value, and then restore the coordinates of the inference result in proportion.

[0091] In another embodiment, the post-processing node is used to send the inference results output by all inference nodes to the Kafka message queue, and based on the image data acquisition requirements of the server connected to the post-processing node, extract the corresponding inference results from the Kafka message queue and forward them to the server, including:

[0092] The post-processing node identifies the image content attributes of the inference results output by all inference nodes, obtains the image content attribute information of all inference results, and sends all inference results to the Kafka message queue in sequence based on the image content attribute information;

[0093] The image content attribute information of all inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

[0094] The beneficial effect of the above embodiment is that the image content attribute recognition is performed on the inference results output by all inference nodes through the post-processing node, and the image content attribute information of each of the inference results is obtained, wherein the image content attribute information may be but is not limited to the content type information on the image frame screen corresponding to the inference result, and then all the inference results are sent to the kafka message queue in sequence based on the image content attribute information, so that all the inference results are accurately distinguished and arranged in the kafka message queue. Then the image content attribute information of all the inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and according to the matching results, the corresponding inference results are extracted from the kafka message queue and forwarded to the server, ensuring that other servers can accurately obtain the actually required inference results for further processing.

[0095] In general, the single video stream multi-algorithm binding method and system based on decoded frame multiplexing obtains video data by pulling a video stream through a video processing node, and performs decoding processing and color space conversion processing on the video data to output processed image frames; binds different types of algorithms to all inference nodes connected to the video processing node, so as to copy the output processed image frames to the corresponding inference nodes respectively, and performs pre-processing inference processing on the processed image frames through the inference nodes to obtain inference results and restore the inference results; sends the inference results output by all inference nodes to the Kafka message queue through the post-processing node, and extracts the corresponding inference results from the Kafka message queue and forwards them to the server, so that the video stream is decoded only once, and the decoded image frames can be deeply copied according to the number of subsequent algorithms, and multiple algorithms are used to independently perform algorithmic inference on the image frames, and the redundant computing consumption of decoding is converted into data copy consumption of decoded frames, so as to avoid multiple decoding of the same video frame, effectively reduce computing redundancy and save resource consumption, facilitate the allocation of resources to other tasks, and improve resource utilization efficiency.

[0096] The above is only a specific implementation of the present invention, and any other improvements made based on the concept of the present invention are considered to be within the protection scope of the present invention.

Claims

1. A single video stream multi-algorithm binding method based on decoding frame multiplexing, characterized in that: include: Pull one video stream through the video processing node to obtain the corresponding video data; Decoding and color space conversion are performed on the video data to obtain corresponding processed image frames; Binding different types of algorithms to all the inference nodes connected to the video processing node, and then copying the processed image frames output by the video processing node to the corresponding inference nodes; Performing a pre-processing inference process on the processed image frame through the inference node to obtain a corresponding inference result, and performing a restoration process on the inference result; The inference results output by all inference nodes are sent to the Kafka message queue through the post-processing node. Based on the image data acquisition requirements of the server connected to the post-processing node, the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

2. The single video stream multi-algorithm binding method based on decoding frame multiplexing according to claim 1, characterized in that: Pull one video stream through the video processing node to obtain the corresponding video data; The video data is decoded and converted into a color space to obtain a corresponding processed image frame, including: The video source connected to the video processing node is monitored to obtain the video stream URL of the video source; based on the video stream URL, a single-channel video stream pulling process is performed on the video source; the video data obtained by the video stream pulling process is identified to obtain corresponding video code format information and image color feature information; Based on the video code format information, the video data is decoded to obtain a corresponding decoded image frame; and based on the image color feature information, the decoded image frame is converted into a color space to obtain a corresponding processed image frame.

3. The single video stream multi-algorithm binding method based on decoding frame multiplexing according to claim 1, characterized in that: Pull a video stream through the video processing node to obtain the corresponding video data, including: Step S1, using the following formula (1), according to the processing status of the video processing node, the video stream pulling efficiency is obtained: In the above formula (1), E v Indicates the video stream pulling efficiency; D b Indicates bandwidth, that is, the maximum amount of data transmitted by the video stream, in bits per second; T r Indicates the transmission time of the video stream in seconds; L s Indicates the transmission delay of the video stream in seconds; N p Indicates the number of processing nodes, that is, the number of nodes that process video streams simultaneously; C p Indicates the processing delay of each processing node in seconds, that is, the time required for the node to process video data; Step S2, using the following formula (2), according to the video stream pulling efficiency, obtain the video data quality score: In the above formula (2), Q v Represents the video data quality score; R i represents the i-th video quality index; W i Representation and video quality index R i The weight coefficient of M1 is the total amount of video data, that is, the data size of the entire video stream; V represents the jth video quality loss factor; , Denotes the video loss factor F , The weight coefficient of m represents the number of video quality indicators; n represents the number of video quality loss factors; Step S3, using the following formula (3), according to the video stream pulling efficiency and the video data quality score, obtain the control factor of dynamic control and optimization, so as to pull the video stream to obtain the corresponding video data. In the above formula (3), C . represents a control factor, which is used to adjust the video stream pulling strategy, and the pulling strategy includes adjusting the pulling bandwidth and / or adjusting the number of processing nodes; β represents the priority factor of the video stream, which is between 0 and 1 and is the processing priority of different video streams; α represents the adjustment coefficient; T B Represents the preset video stream pulling efficiency threshold; e represents a natural constant.

4. The single video stream multi-algorithm binding method based on decoding frame multiplexing according to claim 1, characterized in that: Binding different types of algorithms to all the inference nodes connected to the video processing node, and then copying the processed image frames output by the video processing node to the corresponding inference nodes; The inference node performs a pre-processing inference process on the processed image frame to obtain a corresponding inference result, and restores the inference result, including: Obtain computing power resource information of all inference nodes connected to the video processing node, and determine the algorithms matched by all inference nodes based on the computing power resource information, so as to bind and load different types of algorithms to all inference nodes respectively; obtain working status information of each inference node after binding and loading the algorithm, and screen available inference nodes capable of performing image frame inference based on the working status information; Based on the number of all available reasoning nodes, the processed image frames output by the video processing node are copied in corresponding numbers and transmitted to the corresponding available reasoning nodes respectively; the resolution of the copied image frames is first proportionally scaled by the available reasoning nodes, and then the copied image frames are subjected to corresponding algorithm reasoning processing to obtain reasoning results, and the reasoning results are proportionally restored.

5. The single video stream multi-algorithm binding method based on decoding frame multiplexing according to claim 1, characterized in that: The inference results output by all inference nodes are sent to the Kafka message queue by the post-processing node, and based on the image data acquisition requirements of the server connected to the post-processing node, the corresponding inference results are extracted from the Kafka message queue and forwarded to the server, including: The post-processing node identifies the image content attributes of the inference results output by all inference nodes to obtain the image content attribute information of all inference results, and based on the image content attribute information, sends all the inference results to the Kafka message queue in sequence; The image content attribute information of all inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

6. Single video stream multi-algorithm binding system based on decoded frame multiplexing, characterized in that: include: The video processing node is used to pull a video stream to obtain corresponding video data; Decoding and color space conversion are performed on the video data to obtain corresponding processed image frames; A plurality of inference nodes, each of which is connected to the video processing node and is respectively bound to different types of algorithms, and copies the processed image frames output by the video processing node to the corresponding inference nodes; Performing a pre-processing inference process on the processed image frame through the inference node to obtain a corresponding inference result, and performing a restoration process on the inference result; The post-processing node is used to send the inference results output by all inference nodes to the Kafka message queue, and based on the image data acquisition requirements of the server connected to the post-processing node, extract the corresponding inference results from the Kafka message queue and forward them to the server.

7. The single video stream multi-algorithm binding system based on decoding frame multiplexing as claimed in claim 6, characterized in that: The video processing node is used to pull a video stream to obtain corresponding video data; The video data is decoded and converted into a color space to obtain a corresponding processed image frame, including: Monitor the video source connected to it to obtain the video stream URL of the video source; based on the video stream URL, perform a single-channel video stream pulling process on the video source; identify the video data obtained by the video stream pulling process to obtain corresponding video code format information and image color feature information; Based on the video code format information, the video data is decoded to obtain a corresponding decoded image frame; and based on the image color feature information, the decoded image frame is converted into a color space to obtain a corresponding processed image frame.

8. The single video stream multi-algorithm binding system based on decoding frame multiplexing as claimed in claim 6, characterized in that: Several inference nodes are respectively bound to different types of algorithms, and the processed image frames output by the video processing nodes are respectively copied to corresponding inference nodes; The inference node performs a pre-processing inference process on the processed image frame to obtain a corresponding inference result, and restores the inference result, including: Obtain the computing power resource information of all inference nodes, and determine the algorithms matched by all inference nodes based on the computing power resource information, so as to bind and load different types of algorithms to all inference nodes respectively; obtain the working status information of each inference node after binding and loading the algorithm, and screen the available inference nodes that can perform image frame inference based on the working status information; Based on the number of all available reasoning nodes, the processed image frames output by the video processing node are copied in corresponding numbers and transmitted to the corresponding available reasoning nodes respectively; the resolution of the copied image frames is first proportionally scaled by the available reasoning nodes, and then the copied image frames are subjected to corresponding algorithm reasoning processing to obtain reasoning results, and the reasoning results are proportionally restored.

9. The single video stream multi-algorithm binding system based on decoding frame multiplexing as claimed in claim 6, characterized in that: The post-processing node is used to send the inference results output by all inference nodes to the Kafka message queue, and based on the image data acquisition requirements of the server connected to the post-processing node, extract the corresponding inference results from the Kafka message queue and forward them to the server, including: The post-processing node identifies the image content attributes of the inference results output by all inference nodes to obtain the image content attribute information of all inference results, and based on the image content attribute information, sends all the inference results to the Kafka message queue in sequence; The image content attribute information of all inference results is matched with the image data acquisition requirements of the server connected to the post-processing node, and the corresponding inference results are extracted from the Kafka message queue and forwarded to the server.

Images