Dynamic workflow configuration method and system based on chain-of-responsibility design mode

Abstract

The invention provides a workflow dynamic configuration method and system based on a chain-of-responsibility design mode, and the method comprises the steps: carrying out the pulling of a video stream through a video processing node, creating a decoder and a frame format converter, decoding the video stream to obtain decoded image frame data, and converting the decoded image frame data to obtain an image format conversion frame; circularly reading the image format conversion frames through the reasoning node, pushing the image format conversion frames into a reasoning queue, and performing reasoning operation on the image format conversion frames in the reasoning queue based on a producer-consumer model; a post-processing node receives frame data output by an inference node and forwards the frame data to a corresponding server side, a video processing workflow is decomposed into independent video processing nodes, inference nodes and post-processing nodes, each node is responsible for a specific task and can perform stable and reliable interaction, and when an algorithm model needs to be adapted again, the video processing workflow is divided into the video processing nodes, the inference nodes and the post-processing nodes. Only algorithm model expansion needs to be carried out on the reasoning node, recoding is not needed to adapt to the whole workflow, and the operation complexity of the workflow is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a workflow dynamic configuration method and system based on a responsibility chain design pattern. Background Art

[0002] The processing process of the current video processing workflow mainly includes opening the video stream through OpenCV's VideoCapture, reading data from the video stream frame by frame, decoding it into image frames, loading the pre-trained model, and pre-processing each frame of the image, such as converting the resolution and / or color space according to the input format of the model; finally, drawing boxes, markers and other visual information on the image frame according to the inference results of the model. However, the above method has the following problems: First, different pre-trained models have different requirements for the format and resolution of the input data, that is, there is a diversity in the model input requirements, so that whenever a new model is introduced, additional pre-processing steps are required for the video frame to meet the input requirements of the new model; second, the model also requires input in a specific color space (such as RGB or BGR color space), which requires adding color space conversion steps to the workflow, and these conversions need to be implemented each time a new model is added, thereby increasing the complexity of the workflow; third, the workflow is based on static configuration, so that different models need to be personalized when deployed. Encoding, when adding or replacing a model, the entire workflow needs to be reconfigured and tested, which is not only time-consuming but also error-prone. Summary of the invention

[0003] The purpose of the present invention is to provide a workflow dynamic configuration method and system based on a responsibility chain design pattern, which pulls the video stream through a video processing node and creates a decoder and a frame format converter, thereby decoding the video stream to obtain decoded image frame data, and then converting it to obtain an image format conversion frame; the image format conversion frame is cyclically read through an inference node and pushed into an inference queue, and then the image format conversion frame in the inference queue is inferred based on a producer-consumer model; the frame data output by the inference node is received through a post-processing node and forwarded to a corresponding server, which decomposes the video processing workflow into independent video processing nodes, inference nodes and post-processing nodes, each node is responsible for a specific task, and can interact stably and reliably. When the algorithm model needs to be re-adapted, it is only necessary to expand the algorithm model of the inference node, without the need to re-encode and adapt the entire workflow, thereby reducing the operational complexity of the workflow, shortening the calculation time of the workflow and improving the data processing accuracy of the workflow.

[0004] The present invention is achieved through the following technical solutions:

[0005] The workflow dynamic configuration method based on the responsibility chain design pattern includes:

[0006] Pull the video stream through the video processing node and obtain the corresponding video stream information, create a corresponding decoder and frame format converter based on the video stream information; decode the video stream based on the decoder to obtain decoded image frame data; and then perform frame format conversion processing on the decoded image frame data based on the frame format converter to obtain an image format conversion frame;

[0007] The image format conversion frames are read through the inference node loop, and the read image format conversion frames are pushed into the inference queue; based on the producer-consumer model, the image format conversion frames in the inference queue are inferred and operated, thereby generating corresponding inference operation results;

[0008] The inference operation result is identified by a post-processing node, thereby receiving the frame data output by the inference node, and forwarding the received frame data to a corresponding server.

[0009] Optionally, the video stream is pulled through a video processing node and corresponding video stream information is obtained; based on the video stream information, a corresponding decoder and a frame format converter are created; based on the decoder, the video stream is decoded to obtain decoded image frame data; and based on the frame format converter, the decoded image frame data is subjected to frame format conversion processing to obtain an image format conversion frame, including:

[0010] Based on the input RTSP video stream URL, open the depacketizer of the video processing node; perform signaling negotiation based on the C++ API of FFmpeg through the depacketizer, so as to pull the stream from the RTSP video stream, and obtain the video code format and image resolution of the RTSP video stream as the video stream information;

[0011] Based on the video code format and the image resolution, respectively create a corresponding decoder and a frame format converter, and start hardware acceleration for the decoder and the frame format converter;

[0012] Based on the decoder, the RTSP packets are read cyclically. When all the RTSP packets in the decoder context can be decoded to obtain a complete image frame, all the RTSP packets are decoded and the corresponding decoded image frame data are output; then based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame.

[0013] Optionally, the video stream is pulled through a video processing node and corresponding video stream information is obtained, and based on the video stream information, a corresponding decoder and frame format converter are created, including:

[0014] Step S1, using the following formula (1), the video processing node pulls the video stream and obtains the corresponding video stream information.

[0015]

[0016] In the above formula (1), S v represents the comprehensive score of the video stream information; T represents the total duration of the video stream; F(t) represents the frame quality score obtained at time t; R(t) represents the bit rate at time t; Q(t) represents the resolution score at time t; L(t) represents the brightness score at time t; α, β, γ, δ represent weight coefficients respectively; θ represents the nonlinear adjustment factor;

[0017] Step S2, using the following formula (2), based on the video stream information, create a corresponding decoder:

[0018]

[0019] In the above formula (2), D represents the performance score of the decoder; C represents the encoding format parameters of the video stream; E represents the network delay score; μ1, μ2, μ3 represent weight coefficients respectively; η1, η2, η3 represent nonlinear influence factors respectively; represents the overall adjustment factor;

[0020] Step S3, using the following formula (3), based on the video stream information, create a corresponding frame format converter,

[0021] F t =(λ×S v ×D+γ×Ξ p ) k (3)

[0022] In the above formula (3), F t represents the control signal after the frame format converter converts the external control signal into a frame format; λ represents the adjustment factor; γ represents the weight of the control environment factor; Ξ represents the external control signal; p represents the influencing factor of the external control signal; k represents the overall adjustment factor, which is used to control the strength of the final output signal.

[0023] Optionally, the image format conversion frames are read cyclically through the inference node, and the read image format conversion frames are pushed into the inference queue; and the image format conversion frames in the inference queue are inferred based on the producer-consumer model to generate corresponding inference results, including:

[0024] The producer-consumer model of the inference node cyclically reads the image format conversion frame output by the video processing node, pushes the image format conversion frame into the inference queue at the back end of the producer-consumer model, and uses the conditional variable wake-up algorithm to perform inference operation on the image format conversion frame in the inference queue, thereby generating corresponding inference operation results; and then returns the inference operation results to the front end of the producer-consumer model.

[0025] Optionally, identifying the inference operation result by a post-processing node, thereby receiving the frame data output by the inference node, and forwarding the received frame data to a corresponding server, includes:

[0026] Identify the inference operation result through a post-processing node to determine whether the inference operation result is complete; if the inference operation result is complete, receive the frame data output by the inference node; otherwise, do not receive the frame data output by the inference node;

[0027] Perform image content recognition on the frame data output by the inference node to obtain corresponding image content attribute information, and forward the received frame data to the corresponding server based on the image content attribute information.

[0028] The workflow dynamic configuration system based on the responsibility chain design pattern includes:

[0029] The video processing node is used to pull the video stream and obtain the corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information; based on the decoder, decode the video stream to obtain decoded image frame data; and then based on the frame format converter, perform frame format conversion processing on the decoded image frame data to obtain an image format conversion frame;

[0030] An inference node, used for cyclically reading image format conversion frames, and pushing the read image format conversion frames into an inference queue; performing inference operations on the image format conversion frames in the inference queue based on a producer-consumer model, thereby generating corresponding inference operation results;

[0031] The post-processing node is used to identify the inference operation result, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server.

[0032] Optionally, the video processing node is used to pull the video stream and obtain corresponding video stream information, create a corresponding decoder and frame format converter based on the video stream information; decode the video stream based on the decoder to obtain decoded image frame data; and then perform frame format conversion processing on the decoded image frame data based on the frame format converter to obtain an image format conversion frame, including:

[0033] Based on the input RTSP video stream URL, open the depacketizer of the video processing node; perform signaling negotiation based on the C++ API of FFmpeg through the depacketizer, so as to pull the stream from the RTSP video stream, and obtain the video code format and image resolution of the RTSP video stream as the video stream information;

[0034] Based on the video code format and the image resolution, respectively create a corresponding decoder and a frame format converter, and start hardware acceleration for the decoder and the frame format converter;

[0035] Based on the decoder, the RTSP packets are read cyclically. When all the RTSP packets in the decoder context can be decoded to obtain a complete image frame, all the RTSP packets are decoded and the corresponding decoded image frame data are output; then based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame.

[0036] Optionally, the inference node is used to cyclically read the image format conversion frames, and push the read image format conversion frames into the inference queue; perform inference operations on the image format conversion frames in the inference queue based on a producer-consumer model, thereby generating corresponding inference operation results, including:

[0037] The producer-consumer model of the inference node cyclically reads the image format conversion frame output by the video processing node, pushes the image format conversion frame into the inference queue at the back end of the producer-consumer model, and uses the conditional variable wake-up algorithm to perform inference operation on the image format conversion frame in the inference queue, thereby generating corresponding inference operation results; and then returns the inference operation results to the front end of the producer-consumer model.

[0038] Optionally, the post-processing node is used to identify the inference operation result, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server, including:

[0039] Identify the inference operation result through a post-processing node to determine whether the inference operation result is complete; if the inference operation result is complete, receive the frame data output by the inference node; otherwise, do not receive the frame data output by the inference node;

[0040] Perform image content recognition on the frame data output by the inference node to obtain corresponding image content attribute information, and forward the received frame data to the corresponding server based on the image content attribute information.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The workflow dynamic configuration method and system based on the responsibility chain design pattern provided in the present application pull the video stream through the video processing node and create a decoder and a frame format converter, so as to obtain decoded image frame data from the video stream by decoding, and then convert it to obtain image format conversion frame; the image format conversion frame is read cyclically by the inference node and pushed into the inference queue, and then the image format conversion frame in the inference queue is inferred based on the producer-consumer model; the frame data output by the inference node is received by the post-processing node and forwarded to the corresponding server, which decomposes the video processing workflow into independent video processing nodes, inference nodes and post-processing nodes, each node is responsible for a specific task and can interact stably and reliably. When the algorithm model needs to be re-adapted, it is only necessary to expand the algorithm model of the inference node, without the need to re-encode and adapt the entire workflow, thereby reducing the operational complexity of the workflow, shortening the calculation time of the workflow and improving the data processing accuracy of the workflow. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0044] Figure 1 A schematic flow chart of a workflow dynamic configuration method based on the responsibility chain design pattern provided by the present invention.

[0045] Figure 2 This is a schematic diagram of the structure of a workflow dynamic configuration system based on the responsibility chain design pattern provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] See also Figure 1 As shown, an embodiment of the present application provides a workflow dynamic configuration method based on the responsibility chain design pattern. The workflow dynamic configuration method based on the responsibility chain design pattern includes:

[0050] Pull the video stream through the video processing node and obtain the corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information; decode the video stream based on the decoder to obtain decoded image frame data; and then perform frame format conversion processing on the decoded image frame data based on the frame format converter to obtain an image format conversion frame;

[0051] The image format conversion frames are read through the inference node loop, and the read image format conversion frames are pushed into the inference queue; the image format conversion frames in the inference queue are inferred based on the producer-consumer model, thereby generating corresponding inference results;

[0052] The inference operation result is identified by a post-processing node, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server.

[0053] The beneficial effects of the above embodiments are as follows: the workflow dynamic configuration method based on the responsibility chain design pattern pulls the video stream through the video processing node and creates a decoder and a frame format converter, thereby decoding the video stream to obtain decoded image frame data, and then converting it to obtain image format conversion frames; the image format conversion frames are cyclically read through the inference node and pushed into the inference queue, and then the image format conversion frames in the inference queue are inferred based on the producer-consumer model; the frame data output by the inference node is received through the post-processing node and forwarded to the corresponding server, which decomposes the video processing workflow into independent video processing nodes, inference nodes and post-processing nodes, each node is responsible for a specific task and can interact stably and reliably. When the algorithm model needs to be re-adapted, it is only necessary to expand the algorithm model of the inference node, without the need to re-encode and adapt the entire workflow, thereby reducing the operational complexity of the workflow, shortening the calculation time of the workflow and improving the data processing accuracy of the workflow.

[0054] In another embodiment, a video stream is pulled through a video processing node to obtain corresponding video stream information, and a corresponding decoder and frame format converter are created based on the video stream information; based on the decoder, the video stream is decoded to obtain decoded image frame data; and based on the frame format converter, the decoded image frame data is subjected to frame format conversion processing to obtain an image format conversion frame, including:

[0055] Based on the input RTSP video stream URL, open the depacketizer of the video processing node; perform signaling negotiation based on the FFmpeg C++ API through the depacketizer to pull the stream from the RTSP video stream and obtain the video code format and image resolution of the RTSP video stream as the video stream information;

[0056] Based on the video code format and the image resolution, respectively create a corresponding decoder and a frame format converter, and start hardware acceleration for the decoder and the frame format converter;

[0057] Based on the decoder, the RTSP packets are read cyclically. When all the RTSP packets in the decoder context can be decoded to obtain a complete image frame, all the RTSP packets are decoded and the corresponding decoded image frame data are output; then based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame.

[0058] The beneficial effect of the above embodiment is that when the video end generates video data, the communication interface between the video processing node and the video end is based on the URL of the input RTSP (Real-time Streaming Protocol) video stream, the unpacker of the video processing node is opened, and the RTSP video stream is pulled and collected through the unpacker, that is, signaling negotiation is performed based on the C++API of FFmpeg, and when the signaling negotiation is successful, the RTSP video stream is started to be pulled, and in the pulling procedure, the pulled video stream is identified and analyzed to obtain the video code format and image resolution of the RTSP video stream, so as to accurately characterize the data code format and image resolution of the RTSP video stream, and provide a reliable basis for the subsequent construction of a decoder and a frame format converter. Specifically, based on the video code format and the image resolution, corresponding decoders and frame format converters are created respectively, for example, the decoder can be but not limited to an H264 decoder, and hardware acceleration can also be started for the decoder and the frame format converter, so that the working speed and stability of the decoder and the frame format converter can be improved. In addition, based on the decoder, RTSP packets are read cyclically, so that multiple RTSP packets can be continuously obtained, each of which corresponds to a part of image data. At the same time, context image data integration processing is performed on all RTSP packets currently read by the decoder to determine whether all RTSP packets currently read can be decoded to obtain a complete image frame. Only when a complete image frame can be decoded, all RTSP packets are decoded and the corresponding decoded image frame data are output. This ensures the integrity of the decoded image data and avoids the loss of decoded image data. Based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame, wherein the color space conversion processing can be converted from YUV (such as YUV420P) to RGB or BGR, so that the color space of the decoded image data can be flexibly converted to meet the subsequent reasoning requirements of different algorithm models.

[0059] In another embodiment, a video stream is pulled through a video processing node and corresponding video stream information is obtained, and based on the video stream information, a corresponding decoder and a frame format converter are created, including:

[0060] Step S1, using the following formula (1), the video processing node pulls the video stream and obtains the corresponding video stream information.

[0061]

[0062] In the above formula (1), S vrepresents the comprehensive score of the video stream information; T represents the total duration of the video stream; F(t) represents the frame quality score obtained at time t; R(t) represents the bit rate at time t; Q(t) represents the resolution score at time t; L(t) represents the brightness score at time t; α, β, γ, δ represent weight coefficients respectively; θ represents the nonlinear adjustment factor, which is used to characterize the sensitivity of the comprehensive score;

[0063] Step S2, using the following formula (2), based on the video stream information, create a corresponding decoder:

[0064]

[0065] In the above formula (2), D represents the performance score of the decoder; C represents the encoding format parameters of the video stream; E represents the network delay score; μ1, μ2, μ3 represent weight coefficients respectively; η1, η2, η3 represent nonlinear influence factors respectively; represents the overall adjustment factor, which is used to characterize the sensitivity of the performance score;

[0066] Step S3, using the following formula (3), based on the video stream information, create a corresponding frame format converter:

[0067] F t =(λ×S v ×D+γ×Ξ p ) k (3)

[0068] In the above formula (3), F t It represents the control signal after the frame format converter converts the external control signal into a frame format; λ represents the adjustment factor; γ represents the weight of the control environment factor; Ξ represents the external control signal; p represents the influencing factor of the external control signal; k represents the overall adjustment factor, which is used to control the strength of the final output signal; by adjusting λ, γ, and p, the frequency and quality of the frame format conversion can be controlled.

[0069] The beneficial effect of the above embodiment is that the above formula (1) is used to pull the video stream through the video processing node and obtain the corresponding video stream information. By introducing new parameters and weights, the quality of the video stream is comprehensively evaluated, making the scoring more detailed and adaptable; then the above formula (2) is used to create a corresponding decoder based on the video stream information, and by introducing exponential and nonlinear factors, the performance evaluation of the decoder is made more accurate and can adapt to different video streams and environmental conditions; finally, the above formula (3) is used to create a corresponding frame format converter based on the video stream information, so that the conversion of the frame format can be flexibly adjusted by complicating the control signal. The above process together constitutes a complete processing chain from video stream information extraction to decoder creation and then to frame format control, which can effectively adapt to various video processing scenarios.

[0070] In another embodiment, the image format conversion frames are read cyclically by the inference node, and the read image format conversion frames are pushed into the inference queue; the image format conversion frames in the inference queue are inferred based on the producer-consumer model, thereby generating corresponding inference results, including:

[0071] The producer-consumer model of the inference node cyclically reads the image format conversion frame output by the video processing node, pushes the image format conversion frame into the inference queue at the back end of the producer-consumer model, and uses the conditional variable wake-up algorithm to perform inference operations on the image format conversion frames in the inference queue to generate corresponding inference operation results; and then returns the inference operation results to the front end of the producer-consumer model.

[0072] The beneficial effect of the above embodiment is that the inference node, as a node for implementing algorithmic inference on video data, mainly includes a producer-consumer model and a corresponding inference algorithm model, wherein the producer-consumer model is used to obtain the image format conversion frame output by the video processing node, and the corresponding inference algorithm model is used to perform corresponding algorithmic inference operations on the obtained image format conversion frame, and the producer-consumer model mainly includes a front end and a back end, the front end is responsible for obtaining the inference frame and blocking to wait for the inference result, and the back end is responsible for inference and returns the inference operation result to the front end, that is, the back end can be used to load the corresponding inference algorithm model. Further, the workflow of the producer-consumer model mainly includes: the inference node cyclically reads the decoded frame, and passes the decoded frame to the back end through the front-end interface Commit of the producer-consumer model, and blocks and waits for the inference result (pro.get_future) through the future-promise mechanism of C++. Once the front-end part passes in the data, the back-end pushes the data into the inference queue m_InputQueue, and then uses the conditional variable to wake up the algorithm inference object to obtain the inference queue data and perform inference. After the inference is completed, the inference result is returned to the front-end, so that the acquisition of the image format conversion frame and the inference operation can be effectively separated. Later, only the inference algorithm model loaded in the back-end needs to be changed to adjust the adaptation algorithm, reducing the workload required for the change of the inference algorithm. In addition, the inference node workflow mainly includes: the main program Main passes the loading method of the algorithm model object to the producer-consumer model (CPMInstance), initializes a CPMInstance instance object pInstance, pInstance calls the Start method to start the working thread, and then loads the model according to the loading method of the algorithm model object; the main program Main passes pInstance to the inference node TrNode, and calls the Start method of TrNode to start the working thread, and then TrNode loops to read the data (decoded frame output by the video processing node) and pushes the data through the front-end part of pInstance (Commit interface), and then infers through the back-end part of pInstance (Forward interface) and returns the inference result.

[0073] In another embodiment, the reasoning operation result is identified by a post-processing node, thereby receiving the frame data output by the reasoning node, and forwarding the received frame data to a corresponding server, including:

[0074] The inference operation result is identified by the post-processing node to determine whether the inference operation result is complete; if the inference operation result is complete, the frame data output by the inference node is received; otherwise, the frame data output by the inference node is not received;

[0075] Image content recognition is performed on the frame data output by the inference node to obtain corresponding image content attribute information, and based on the image content attribute information, the received frame data is forwarded to the corresponding server.

[0076] The beneficial effect of the above embodiment is that the post-processing node can be used as a middleware node to receive the output of the reasoning node and forward it to other servers for use, which can ensure that the output of the reasoning node can be forwarded according to the data requirements of different servers, thereby improving the flexibility of data circulation. Specifically, the reasoning operation result is identified by the post-processing node to determine whether the reasoning operation result is complete, which can ensure the credibility of the frame data output by the reasoning node, and the image content of the frame data output by the reasoning node is identified to obtain the corresponding image content attribute information, so as to forward the received frame data to the corresponding server, ensuring that the server can obtain the corresponding frame data in a timely manner.

[0077] See also Figure 2 As shown, an embodiment of the present application provides a workflow dynamic configuration system based on the responsibility chain design pattern. The workflow dynamic configuration system based on the responsibility chain design pattern includes:

[0078] The video processing node is used to pull the video stream and obtain the corresponding video stream information, and based on the video stream information, create a corresponding decoder and frame format converter; based on the decoder, decode the video stream to obtain decoded image frame data; and then based on the frame format converter, perform frame format conversion processing on the decoded image frame data to obtain an image format conversion frame;

[0079] An inference node is used to cyclically read image format conversion frames and push the read image format conversion frames into an inference queue; perform inference operations on the image format conversion frames in the inference queue based on a producer-consumer model, thereby generating corresponding inference operation results;

[0080] The post-processing node is used to identify the inference operation result, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server.

[0081] The beneficial effects of the above embodiments are as follows: the workflow dynamic configuration system based on the responsibility chain design pattern pulls the video stream through the video processing node and creates a decoder and a frame format converter, thereby decoding the video stream to obtain decoded image frame data, and then converting it to obtain image format conversion frames; the image format conversion frames are cyclically read through the inference node and pushed into the inference queue, and then the image format conversion frames in the inference queue are inferred based on the producer-consumer model; the frame data output by the inference node is received through the post-processing node and forwarded to the corresponding server, which decomposes the video processing workflow into independent video processing nodes, inference nodes and post-processing nodes, each node is responsible for a specific task and can interact stably and reliably. When the algorithm model needs to be re-adapted, it is only necessary to expand the algorithm model of the inference node, without the need to re-encode and adapt the entire workflow, thereby reducing the operational complexity of the workflow, shortening the calculation time of the workflow and improving the data processing accuracy of the workflow.

[0082] In another embodiment, the video processing node is used to pull the video stream and obtain corresponding video stream information, create a corresponding decoder and frame format converter based on the video stream information; decode the video stream based on the decoder to obtain decoded image frame data; and then perform frame format conversion processing on the decoded image frame data based on the frame format converter to obtain an image format conversion frame, including:

[0083] Based on the input RTSP video stream URL, open the depacketizer of the video processing node; perform signaling negotiation based on the FFmpeg C++ API through the depacketizer to pull the stream from the RTSP video stream and obtain the video code format and image resolution of the RTSP video stream as the video stream information;

[0084] Based on the video code format and the image resolution, respectively create a corresponding decoder and a frame format converter, and start hardware acceleration for the decoder and the frame format converter;

[0085] Based on the decoder, the RTSP packets are read cyclically. When all the RTSP packets in the decoder context can be decoded to obtain a complete image frame, all the RTSP packets are decoded and the corresponding decoded image frame data are output; then based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame.

[0086] The beneficial effect of the above embodiment is that when the video end generates video data, the communication interface between the video processing node and the video end is based on the URL of the input RTSP (Real-time Streaming Protocol) video stream, the unpacker of the video processing node is opened, and the RTSP video stream is pulled and collected through the unpacker, that is, signaling negotiation is performed based on the C++API of FFmpeg, and when the signaling negotiation is successful, the RTSP video stream is started to be pulled, and in the pulling procedure, the pulled video stream is identified and analyzed to obtain the video code format and image resolution of the RTSP video stream, so as to accurately characterize the data code format and image resolution of the RTSP video stream, and provide a reliable basis for the subsequent construction of a decoder and a frame format converter. Specifically, based on the video code format and the image resolution, corresponding decoders and frame format converters are created respectively, for example, the decoder can be but not limited to an H264 decoder, and hardware acceleration can also be started for the decoder and the frame format converter, so that the working speed and stability of the decoder and the frame format converter can be improved. In addition, based on the decoder, RTSP packets are read cyclically, so that multiple RTSP packets can be continuously obtained, each of which corresponds to a part of image data. At the same time, context image data integration processing is performed on all RTSP packets currently read by the decoder to determine whether all RTSP packets currently read can be decoded to obtain a complete image frame. Only when a complete image frame can be decoded, all RTSP packets are decoded and the corresponding decoded image frame data are output. This ensures the integrity of the decoded image data and avoids the loss of decoded image data. Based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame, wherein the color space conversion processing can be converted from YUV (such as YUV420P) to RGB or BGR, so that the color space of the decoded image data can be flexibly converted to meet the subsequent reasoning requirements of different algorithm models.

[0087] In another embodiment, the inference node is used to cyclically read the image format conversion frames, and push the read image format conversion frames into the inference queue; perform inference operations on the image format conversion frames in the inference queue based on the producer-consumer model, thereby generating corresponding inference operation results, including:

[0088] The producer-consumer model of the inference node cyclically reads the image format conversion frame output by the video processing node, pushes the image format conversion frame into the inference queue at the back end of the producer-consumer model, and uses the conditional variable wake-up algorithm to perform inference operations on the image format conversion frames in the inference queue to generate corresponding inference operation results; and then returns the inference operation results to the front end of the producer-consumer model.

[0089] The beneficial effect of the above embodiment is that the inference node, as a node for implementing algorithmic inference on video data, mainly includes a producer-consumer model and a corresponding inference algorithm model, wherein the producer-consumer model is used to obtain the image format conversion frame output by the video processing node, and the corresponding inference algorithm model is used to perform corresponding algorithmic inference operations on the obtained image format conversion frame, and the producer-consumer model mainly includes a front end and a back end, the front end is responsible for obtaining the inference frame and blocking to wait for the inference result, and the back end is responsible for inference and returns the inference operation result to the front end, that is, the back end can be used to load the corresponding inference algorithm model. Further, the workflow of the producer-consumer model mainly includes: the inference node cyclically reads the decoded frame, and passes the decoded frame to the back end through the front-end interface Commit of the producer-consumer model, and blocks and waits for the inference result (pro.get_future) through the future-promise mechanism of C++. Once the front-end part passes in the data, the back-end pushes the data into the inference queue m_InputQueue, and then uses the conditional variable to wake up the algorithm inference object to obtain the inference queue data and perform inference. After the inference is completed, the inference result is returned to the front-end, so that the acquisition of the image format conversion frame and the inference operation can be effectively separated. Later, only the inference algorithm model loaded in the back-end needs to be changed to adjust the adaptation algorithm, reducing the workload required for the change of the inference algorithm. In addition, the inference node workflow mainly includes: the main program Main passes the loading method of the algorithm model object to the producer-consumer model (CPMInstance), initializes a CPMInstance instance object pInstance, pInstance calls the Start method to start the working thread, and then loads the model according to the loading method of the algorithm model object; the main program Main passes pInstance to the inference node TrNode, and calls the Start method of TrNode to start the working thread, and then TrNode loops to read the data (decoded frame output by the video processing node) and pushes the data through the front-end part of pInstance (Commit interface), and then infers through the back-end part of pInstance (Forward interface) and returns the inference result.

[0090] In another embodiment, the post-processing node is used to identify the inference operation result, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server, including:

[0091] The inference operation result is identified by the post-processing node to determine whether the inference operation result is complete; if the inference operation result is complete, the frame data output by the inference node is received; otherwise, the frame data output by the inference node is not received;

[0092] Image content recognition is performed on the frame data output by the inference node to obtain corresponding image content attribute information, and based on the image content attribute information, the received frame data is forwarded to the corresponding server.

[0093] The beneficial effect of the above embodiment is that the post-processing node can be used as a middleware node to receive the output of the reasoning node and forward it to other servers for use, which can ensure that the output of the reasoning node can be forwarded according to the data requirements of different servers, thereby improving the flexibility of data circulation. Specifically, the reasoning operation result is identified by the post-processing node to determine whether the reasoning operation result is complete, which can ensure the credibility of the frame data output by the reasoning node, and the image content of the frame data output by the reasoning node is identified to obtain the corresponding image content attribute information, so as to forward the received frame data to the corresponding server, ensuring that the server can obtain the corresponding frame data in a timely manner.

[0094] In general, the workflow dynamic configuration method and system based on the responsibility chain design pattern pulls the video stream through the video processing node and creates a decoder and a frame format converter, so as to obtain decoded image frame data from the video stream, and then convert it to obtain image format conversion frames; the image format conversion frames are cyclically read through the inference node and pushed into the inference queue, and then the image format conversion frames in the inference queue are inferred based on the producer-consumer model; the frame data output by the inference node is received through the post-processing node and forwarded to the corresponding server, which decomposes the video processing workflow into independent video processing nodes, inference nodes and post-processing nodes. Each node is responsible for a specific task and can interact stably and reliably. When the algorithm model needs to be re-adapted, it is only necessary to expand the algorithm model of the inference node, without re-encoding and adapting the entire workflow, thereby reducing the operational complexity of the workflow, shortening the calculation time of the workflow and improving the data processing accuracy of the workflow.

[0095] The above is only a specific implementation of the present invention, and any other improvements made based on the concept of the present invention are considered to be within the protection scope of the present invention.

Claims

1. A workflow dynamic configuration method based on the responsibility chain design pattern, characterized in that: include: Pull the video stream through the video processing node and obtain corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information; Based on the decoder, decoding the video stream to obtain decoded image frame data; and then based on the frame format converter, performing frame format conversion processing on the decoded image frame data to obtain an image format conversion frame; The image format conversion frames are read through the inference node loop, and the read image format conversion frames are pushed into the inference queue; based on the producer-consumer model, the image format conversion frames in the inference queue are inferred and operated, thereby generating corresponding inference operation results; The inference operation result is identified by a post-processing node, thereby receiving the frame data output by the inference node, and forwarding the received frame data to a corresponding server.

2. The workflow dynamic configuration method based on the responsibility chain design pattern as claimed in claim 1, characterized in that: Pull the video stream through the video processing node and obtain corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information; Based on the decoder, decoding the video stream to obtain decoded image frame data; Then, based on the frame format converter, performing frame format conversion processing on the decoded image frame data to obtain an image format conversion frame, including: Based on the input RTSP video stream URL, open the depacketizer of the video processing node; perform signaling negotiation based on the C++ API of FFmpeg through the depacketizer, so as to pull the stream from the RTSP video stream, and obtain the video code format and image resolution of the RTSP video stream as the video stream information; Based on the video code format and the image resolution, respectively create a corresponding decoder and a frame format converter, and start hardware acceleration for the decoder and the frame format converter; Based on the decoder, the RTSP packets are read cyclically. When all the RTSP packets in the decoder context can be decoded to obtain a complete image frame, all the RTSP packets are decoded and the corresponding decoded image frame data are output; then based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame.

3. The workflow dynamic configuration method based on the responsibility chain design pattern as claimed in claim 1, characterized in that: Pull the video stream through the video processing node and obtain the corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information, including: Step S1, using the following formula (1), the video processing node pulls the video stream and obtains the corresponding video stream information. In the above formula (1), S v represents the comprehensive score of the video stream information; T represents the total duration of the video stream; F(t) represents the frame quality score obtained at time t; R(t) represents the bit rate at time t; Q(t) represents the resolution score at time t; L(t) represents the brightness score at time t; α, β, γ, δ represent weight coefficients respectively; θ represents the nonlinear adjustment factor; Step S2, using the following formula (2), based on the video stream information, create a corresponding decoder: In the above formula (2), D represents the performance score of the decoder; C represents the encoding format parameters of the video stream; E represents the network delay score; μ1, μ2, μ3 represent weight coefficients respectively; η1, η2, η3 represent nonlinear influence factors respectively; represents the overall adjustment factor; Step S3, using the following formula (3), based on the video stream information, create a corresponding frame format converter, F t =(λ×S v ×D+γ×Ξ p ) k (3) In the above formula (3), F t represents the control signal after the frame format converter converts the external control signal into a frame format; λ represents the adjustment factor; γ represents the weight of the control environment factor; Ξ represents the external control signal; p represents the influencing factor of the external control signal; k represents the overall adjustment factor, which is used to control the strength of the final output signal.

4. The workflow dynamic configuration method based on the responsibility chain design pattern as claimed in claim 1, characterized in that: The image format conversion frames are read through the inference node loop, and the read image format conversion frames are pushed into the inference queue; Based on the producer-consumer model, an inference operation is performed on the image format conversion frame in the inference queue to generate a corresponding inference operation result, including: The producer-consumer model of the inference node cyclically reads the image format conversion frame output by the video processing node, pushes the image format conversion frame into the inference queue at the back end of the producer-consumer model, and uses the conditional variable wake-up algorithm to perform inference operation on the image format conversion frame in the inference queue, thereby generating corresponding inference operation results; and then returns the inference operation results to the front end of the producer-consumer model.

5. The workflow dynamic configuration method based on the responsibility chain design pattern as claimed in claim 1, characterized in that: The inference operation result is identified by a post-processing node, thereby receiving the frame data output by the inference node, and forwarding the received frame data to a corresponding server, including: Identify the inference operation result through a post-processing node to determine whether the inference operation result is complete; if the inference operation result is complete, receive the frame data output by the inference node; otherwise, do not receive the frame data output by the inference node; Perform image content recognition on the frame data output by the inference node to obtain corresponding image content attribute information, and forward the received frame data to the corresponding server based on the image content attribute information.

6. A workflow dynamic configuration system based on the responsibility chain design pattern, characterized by: include: The video processing node is used to pull the video stream and obtain the corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information; Based on the decoder, decoding the video stream to obtain decoded image frame data; and then based on the frame format converter, performing frame format conversion processing on the decoded image frame data to obtain an image format conversion frame; An inference node, used for cyclically reading image format conversion frames, and pushing the read image format conversion frames into an inference queue; performing inference operations on the image format conversion frames in the inference queue based on a producer-consumer model, thereby generating corresponding inference operation results; The post-processing node is used to identify the inference operation result, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server.

7. The workflow dynamic configuration system based on the responsibility chain design pattern as claimed in claim 6, characterized in that: The video processing node is used to pull the video stream and obtain corresponding video stream information, and create a corresponding decoder and frame format converter based on the video stream information; Based on the decoder, decoding the video stream to obtain decoded image frame data; Then, based on the frame format converter, performing frame format conversion processing on the decoded image frame data to obtain an image format conversion frame, including: Based on the input RTSP video stream URL, open the depacketizer of the video processing node; perform signaling negotiation based on the C++ API of FFmpeg through the depacketizer, so as to pull the stream from the RTSP video stream, and obtain the video code format and image resolution of the RTSP video stream as the video stream information; Based on the video code format and the image resolution, respectively create a corresponding decoder and a frame format converter, and start hardware acceleration for the decoder and the frame format converter; Based on the decoder, the RTSP packets are read cyclically. When all the RTSP packets in the decoder context can be decoded to obtain a complete image frame, all the RTSP packets are decoded and the corresponding decoded image frame data are output; then based on the frame format converter, the decoded image frame data is converted into a color space to obtain an image format conversion frame.

8. The workflow dynamic configuration system based on the responsibility chain design pattern as claimed in claim 6, characterized in that: The inference node is used to cyclically read the image format conversion frame, and push the read image format conversion frame into the inference queue; Based on the producer-consumer model, an inference operation is performed on the image format conversion frame in the inference queue to generate a corresponding inference operation result, including: The producer-consumer model of the inference node cyclically reads the image format conversion frame output by the video processing node, pushes the image format conversion frame into the inference queue at the back end of the producer-consumer model, and uses the conditional variable wake-up algorithm to perform inference operation on the image format conversion frame in the inference queue, thereby generating corresponding inference operation results; and then returns the inference operation results to the front end of the producer-consumer model.

9. The workflow dynamic configuration system based on the responsibility chain design pattern as claimed in claim 6, characterized in that: The post-processing node is used to identify the inference operation result, thereby receiving the frame data output by the inference node, and forwarding the received frame data to the corresponding server, including: Identify the inference operation result through a post-processing node to determine whether the inference operation result is complete; if the inference operation result is complete, receive the frame data output by the inference node; otherwise, do not receive the frame data output by the inference node; Perform image content recognition on the frame data output by the inference node to obtain corresponding image content attribute information, and forward the received frame data to the corresponding server based on the image content attribute information.

Images