Delay control method for streaming AI reasoning based on gstreamer

Abstract

The present invention discloses a delay control method for streaming media AI reasoning based on gstreamer, comprising the following steps: real-time monitoring of reasoning data frame information at the interface of the infer reasoning plug-in; obtaining the number of data frames waiting for reasoning, judging whether the number of data frames waiting for reasoning is greater than a maximum preset value, if so, discarding the current data frame and returning; if not, proceeding to the next step; obtaining the time difference between the timestamp of the last data frame added to the waiting reasoning and the timestamp of the current data frame, judging whether the time difference is less than a minimum preset value, if so, discarding the current data frame and returning; if not, proceeding to the next step; making the current data frame join the waiting queue of the infer reasoning plug-in. The present invention effectively controls the internal data frame cache quantity of the infer reasoning plug-in by externally monitoring the number of frames inside the inference plug-in to ensure that the output delay is within the control range, so as not to be accumulated and amplified all the time, and avoid program crash.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a delay control method for streaming media AI reasoning based on gstreamer. Background Art

[0002] The multimedia architecture solution based on gstreamer is currently the most widely used architecture solution (including media players, streaming media processors, etc.). The general architecture is as follows Figure 2 As shown, the source plug-in is the data source, which may be files, streaming media, or other applications; the queue plug-in plays the role of caching and threading in gstreamer; the infer plug-in is the inference plug-in for machine vision; the sink plug-in is the data output, which may be live broadcast, file or display, etc.

[0003] Common designs such as Figure 3 As shown in the figure, in this pipeline architecture design, each plug-in is directly called and will not be executed in an independent thread. The queue plug-in is used to divide threads and cache, and the leaky property of the queue plug-in is used to drop frames to maintain the real-time nature of the media.

[0004] In this architecture model, the prerequisite for maintaining real-time performance is that all plug-ins (except the queue plug-in) are directly called through, so that all data frames can be accumulated in the queue plug-in. The queue plug-in calculates how much buffer there is and then starts dropping frames to ensure real-time performance.

[0005] Gstreamer was designed earlier and was originally intended for streaming media. However, when applied to fields such as machine vision, this delay control method lost its original capabilities, because the CV filter plug-in for machine vision needs to use the SOC's neural network hardware or GPU hardware for inference, and often starts independent threads to process the inference logic, so the upstream data frames are cached inside the inference plug-in, causing the queue plug-in upstream of the inference plug-in to be unable to cache and accumulate, and the frame loss logic cannot be triggered. The resulting phenomenon is that if the inference computing power cannot keep up with the speed of the media stream, more and more data frames will accumulate inside the inference plug-in, and the output delay will become larger and larger.

[0006] One feasible solution is to contact the SOC solution provider directly to modify the internal logic of the inference plug-in. If the number of backlogged data frames exceeds the predefined number, the interface function will be blocked. Then each update needs to be synchronized, which poses a greater code risk. In addition, many SOC solution providers' inference plug-ins are not open source, so it is impossible to modify them yourself. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a delay control method for streaming media AI reasoning based on gstreamer.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] The delay control method of streaming media AI reasoning based on gstreamer includes the following steps:

[0010] Step S1, real-time monitoring of inference data frame information at the interface of the infer inference plug-in;

[0011] Step S2, obtain the number of data frames waiting for reasoning according to the inference data frame information, and determine whether the number of data frames waiting for reasoning is greater than the maximum preset value. If so, discard the current data frame and return; if not, enter step S3; obtain the number of data frames waiting for reasoning and determine whether the number of data frames waiting for reasoning is greater than the maximum preset value

[0012] Step S3, obtaining the time difference between the timestamp of the last data frame added to wait for inference and the timestamp of the current data frame, and judging whether the time difference is less than the minimum preset value, if so, discarding the current data frame and returning; if not, proceeding to step S4;

[0013] Step S4, adding the current data frame to the waiting queue of the infer inference plug-in.

[0014] Preferably, the inference data frame information includes the cumulative number of data frames whose inference is completed, the cumulative number of data frames to be added to the inference cache, and the timestamps of the data frames added to the inference cache.

[0015] Preferably, the number of data frames waiting for inference is equal to the cumulative number of data frames ready to be added to the inference cache minus the cumulative number of data frames whose inference has been completed.

[0016] Preferably, the interface of the infer inference plug-in includes an input port and an output port.

[0017] Preferably, the step S1 specifically includes the following steps: calling the prob probe function at the interface of the infer inference plug-in to monitor the inference data frame information at the interface of the infer inference plug-in in real time.

[0018] Based on the above technical solution, the beneficial effects of the present invention are:

[0019] 1) The present invention monitors the number of frames inside the inference plug-in externally, without relying on the prerequisite that there can be no cache and thread division inside the plug-in, and effectively controls the CV filter plug-in of the machine vision to use the neural network hardware or GPU hardware of the SOC to infer, and starts an independent thread to process the inference logic. The number of internal data frame caches of the inference plug-in ensures that the output delay is within the control range, does not accumulate and amplify, and avoids program crashes;

[0020] 2) The present invention does not need to modify the internal logic of the inference plug-in. By externally monitoring the number of frames inside the inference plug-in, there will be no technical coupling, avoiding the need to directly maintain the inference plug-in through the SOC solution provider, and no need for update synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0022] Figure 1 is a flow chart of a delay control method for streaming media AI reasoning based on gstreamer in an embodiment;

[0023] Figure 2 This is the AI ​​reasoning architecture diagram based on gstreamer;

[0024] Figure 3 This is the design diagram of the existing gstreamer multimedia architecture;

[0025] Figure 4 The figure is a diagram of an improved AI reasoning architecture based on gstreamer in an embodiment. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0027] like Figure 1 As shown, this embodiment provides a delay control method for streaming media AI reasoning based on gstreamer, comprising the following steps:

[0028] Step S1, add a prob probe function to the input port sink pad and output port src pad of the infer inference plug-in to dynamically track the number of data frames inside the infer inference plug-in, such as Figure 4 As shown, the prob probe function (prob src) at the output port records the cumulative number of data frames completed by reasoning; the prob probe function (prob sink) at the input port records the cumulative number of data frames ready to be added to the reasoning cache and the timestamp (PTS) of the data frames added to the reasoning cache;

[0029] Step S2, obtaining the number of data frames waiting for reasoning according to the reasoning data frame information, the number of data frames waiting for reasoning is equal to the cumulative number of data frames ready to be added to the reasoning cache minus the cumulative number of data frames whose reasoning is completed, and judging whether the number of data frames waiting for reasoning is greater than the maximum preset value, if so, discarding the current data frame and returning; if not, entering step S3;

[0030] Step S3, obtaining the time difference between the timestamp of the last data frame added to wait for inference and the timestamp of the current data frame, and judging whether the time difference is less than the minimum preset value, if so, discarding the current data frame and returning; if not, proceeding to step S4;

[0031] Step S4, adding the current data frame to the waiting queue of the infer inference plug-in.

[0032] The above is only a preferred implementation of the delay control method for streaming media AI reasoning based on gstreamer disclosed in the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the protection scope of the embodiments of this specification.

Claims

1. A delay control method for streaming media AI reasoning based on gstreamer, characterized in that: The steps include: Step S1, real-time monitoring of inference data frame information at the interface of the infer inference plug-in; Step S2, obtaining the number of data frames waiting for inference according to the inference data frame information, and determining whether the number of data frames waiting for inference is greater than a maximum preset value. If so, discarding the current data frame and returning; If not, proceed to step S3; Step S3, obtaining the time difference between the timestamp of the last data frame added to wait for inference and the timestamp of the current data frame, and determining whether the time difference is less than a minimum preset value. If so, discard the current data frame and return; If not, proceed to step S4; Step S4, adding the current data frame to the waiting queue of the infer inference plug-in.

2. The delay control method for streaming media AI reasoning based on gstreamer according to claim 1 is characterized in that: The inference data frame information includes the cumulative number of data frames whose inferences have been completed, the cumulative number of data frames ready to be added to the inference cache, and the timestamps of the data frames added to the inference cache.

3. The delay control method for streaming media AI reasoning based on gstreamer according to claim 1 is characterized in that: The number of data frames waiting for inference is equal to the cumulative number of data frames ready to be added to the inference cache minus the cumulative number of data frames whose inference has been completed.

4. The delay control method for streaming media AI reasoning based on gstreamer according to claim 1 is characterized in that: The interface of the infer inference plug-in includes an input port and an output port.

5. The delay control method for streaming media AI reasoning based on gstreamer according to claim 1 is characterized in that: The step S1 specifically includes the following steps: calling the prob probe function at the interface of the infer inference plug-in to monitor the inference data frame information at the interface of the infer inference plug-in in real time.

Images