Video monitoring real-time stream anomaly sensing and steady-state scheduling method

Abstract

The invention discloses a video monitoring real-time stream anomaly sensing and steady-state scheduling method, which is applied to a processing node for accessing, converging or forwarding a video stream. Multi-protocol video streams are uniformly adapted and written into a data distribution channel, and operation indexes such as an input rate, an output rate and a cache occupancy rate are collected according to a sampling period and stored into an annular buffer area; establishing a dynamic baseline in a preset time window to perform anomaly detection, judging states such as upstream input anomaly, downstream output congestion and media interruption in combination with index combination, and inhibiting state jitter by adopting jitter removal confirmation and minimum residence; session-level token bucket flow control parameters are generated according to the state and synchronized to a sending thread, and frame recovery is achieved by combining multi-level cache cooperation of memory cache and persistent cache with key frame priority reservation; and after the exception is removed, the sending rate is improved according to a staged recovery rule, and rollback is supported, so that closed-loop steady-state control is formed, and the continuity, the stability and the operability and maintainability under a weak network are improved.

Description

Technical Field

[0001] This invention relates to the field of video surveillance data transmission and processing technology, and in particular to a method for real-time anomaly detection and steady-state scheduling of video surveillance streams. Background Technology

[0002] Currently, video stream transmission in video surveillance services typically employs a cascading link structure: video is captured by front-end cameras, aggregated via access services, distributed through streaming media services, and finally transmitted to the client for display. The access service generally handles multi-protocol adaptation and initial traffic processing, playing a crucial role in anomaly detection and scheduling of the video stream. In existing technologies, common solutions for controlling the transmission of video surveillance streams mainly include the following categories: (1) Monitoring scheme based on fixed threshold: This is achieved by configuring a fixed buffer threshold or a fixed timeout. When the amount of buffered data exceeds the threshold or no data is transmitted within a specified time, actions such as dropping, reconnecting, or alarms are triggered.

[0003] (2) Passive scheduling scheme based on transport protocol stack: relying on the congestion control and retransmission mechanisms of the transport layer or protocol stack itself, the sending rate is passively reduced after congestion or packet loss is detected, and then gradually increased after the network recovers.

[0004] (3) Simple flow control scheme based on unified caching: Video streams from different protocols are converted into a unified format and placed into a shared buffer. When the buffer is full, a first-in-first-out (FIFO) or timestamp-based strategy is used to discard them. This usually does not fully consider the differences in importance of video frame types (e.g., key frames and non-key frames) in terms of playback continuity. However, the above-mentioned existing technologies still reveal the following shortcomings in multi-vendor heterogeneous device access and complex network environments: First, there is a lack of ability to differentiate the source of congestion. Existing solutions often struggle to effectively distinguish whether an anomaly originates from an upstream device's output / transmission failure or from a downstream client's limited receiving capacity or network bandwidth. The system typically only observes results such as buffer backlog, transmission obstruction, or packet loss, leading to the application of the same handling strategy to anomalies of different causes, lacking targeted scheduling optimization.

[0005] Secondly, the scheduling strategy is passive and lagging. Existing solutions often only trigger adjustments when the buffer is close to or has already overflowed, or when network congestion is already significant. This makes it difficult to combine traffic change trends for smooth control in a timely manner, which can easily lead to increased latency, worsened congestion, or jitter.

[0006] Secondly, the resource scheduling mechanism is simplistic. Existing solutions are relatively rigid in their caching strategies, lacking a collaborative and hierarchical scheduling mechanism that integrates multi-level caching such as memory and disk. In large-scale concurrency scenarios, relying entirely on memory caching will lead to significant resource pressure, while frequent reliance on disk read / write operations may introduce additional I / O overhead, making it difficult to meet real-time requirements.

[0007] Finally, there is insufficient basis for protocol adaptation and flow control. Different protocols lack unified status indicator collection and aggregation during the access and adaptation phases, resulting in a lack of multi-dimensional basis for flow control and making it difficult to adaptively adjust based on the real-time status of the video stream (such as bitrate fluctuation characteristics). In summary, how to achieve accurate perception of the real-time streaming status of video surveillance, effective differentiation of anomaly types, and the adoption of targeted steady-state scheduling strategies to improve the stability and controllability of real-time preview in scenarios with multi-vendor heterogeneous device access is a pressing technical problem that needs to be solved. Summary of the Invention

[0008] The purpose of this invention is to provide a method for anomaly detection and steady-state scheduling of real-time video surveillance streams. This method aims to overcome the shortcomings of existing video surveillance stream anomaly detection methods in multi-vendor heterogeneous access and complex network environments, such as reliance on static thresholds, difficulty in distinguishing between normal fluctuations and abnormal congestion, lack of effective identification of congestion sources, passive and lagging scheduling strategies, and a single cache and resource scheduling mechanism. By continuously sensing and collecting indicators of the real-time stream transmission status, it achieves effective identification and differentiation of anomaly types and causes, providing targeted scheduling basis for different anomaly scenarios. Furthermore, by incorporating steady-state control principles, it performs smooth scheduling and recovery when congestion trends occur or anomalies occur, reducing unnecessary frame drops and stuttering, controlling latency growth, and improving the stability of the recovery process. Simultaneously, in large-scale concurrent scenarios, it coordinates the hierarchical use and scheduling of cache resources such as memory and disk, reducing system resource pressure while meeting real-time low-latency requirements, ultimately improving the continuity, stability, and controllability of real-time video preview services under weak network fluctuation conditions.

[0009] To achieve the above objectives, this invention provides a method for real-time anomaly detection and steady-state scheduling of video surveillance streams, applied to processing nodes for video stream access, aggregation, or forwarding, comprising: 1) Process at least one real-time video stream, adapt video data from different protocols or sources to the same internal data format, and write it into the data distribution channel for one or more downstream processing modules to read; 2) Collect the running status indicators of the real-time video stream at a preset sampling period. The running status indicators include at least the input rate, output rate and buffer occupancy rate. Write the running status indicators at each sampling time into a circular buffer for storing the most recent sampling points. 3) Based on the operating status indicators within the circular buffer, a dynamic baseline is established for the input rate, output rate, and buffer occupancy rate within a preset time window, and anomaly detection is performed on the real-time video stream according to the dynamic baseline to obtain the abnormal status. 4) Based on the combination relationship between the abnormal state and the operating state index, the abnormal source is identified, and the output includes at least one of the following state categories: upstream input abnormal state, downstream output congestion state, and media interruption state. Frequent state switching is suppressed through de-jitter confirmation and minimum dwell mechanism. 5) Generate a flow control policy based on the state category and synchronize it to the sending thread. The flow control policy includes at least a token bucket-based sending rate control parameter. 6) During the flow control process, at least two levels of cache are used for coordinated scheduling. The at least two levels of cache include a memory cache and a persistent cache. When the downstream output is in a congested state, the overflow data is written to the persistent cache and the key frame data is kept in priority. When the downstream output congestion is relieved, the nearest key frame is located from the persistent cache and the output is restored. 7) When the abnormal state is resolved or the recovery conditions are met, the sending rate is gradually increased according to the phased recovery rules, and rollback or exit is executed based on the buffer occupancy rate and its changing trend during the recovery process to form a closed-loop steady-state scheduling.

[0010] Furthermore, the state categories include at least a normal state, an upstream input abnormal state, a downstream output congestion state, and a media interruption state, wherein the determination of the media interruption state includes not receiving media frames for a continuous preset time.

[0011] Furthermore, the determination of the downstream output congestion status includes meeting one of the following conditions: a) the output rate is lower than a preset ratio of the target rate and the cache occupancy rate shows an upward trend and continues to exceed a preset duration; b) the ratio of the input rate to the output rate exceeds a preset ratio threshold and the cache occupancy rate accumulates rapidly.

[0012] Furthermore, the determination of the abnormal state of the upstream input includes meeting one of the following conditions: a) the input rate is higher than the upper limit of the dynamic baseline and the cache occupancy rate suddenly increases; b) the input rate is lower than the lower limit of the dynamic baseline and causes the cache occupancy rate to be quickly emptied.

[0013] Furthermore, the de-jitter confirmation includes setting a confirmation counter for each state category and confirming the state switch only when the confirmation counter reaches a preset number of times threshold. The minimum dwell time mechanism includes setting a minimum dwell time for the upstream input abnormal state, the downstream output congestion state, and the media interruption state, and switching back to the normal state only when the running status index meets the normal conditions for a certain number of consecutive sampling cycles after the dwell time expires.

[0014] Furthermore, the flow control strategy adopts a token bucket model independent for each session. The token bucket parameters include the token generation rate and the burst rate. The token generation rate is adjusted according to the state category. Specifically, when the upstream input is in an abnormal state, the rate is reduced by a first scaling factor; when the downstream output is in a congested state, the rate is reduced by a second scaling factor; and when the state is normal, the rate is gradually increased by an increase factor without exceeding the maximum rate limit. The update results of the token bucket parameters are synchronized to the sending thread through atomic variables so that the sending thread can read the token bucket parameters without locking.

[0015] Furthermore, the at least two-level cache includes a first-level memory cache and a second-level persistent cache, wherein the first-level memory cache is a lock-free circular queue for caching hot data, and the second-level persistent cache is a memory-mapped file-based cache for caching overflow data when the first-level memory cache overflows, and during output scheduling, it preferentially reads from the first-level memory cache and reads from the second-level persistent cache when the first-level memory cache is empty.

[0016] Furthermore, during downstream output congestion, a frame-dropping-skeleton-preserving and key-frame-priority strategy is implemented. The frame-dropping-skeleton-preserving strategy includes discarding bidirectional prediction frames when the backlog delay exceeds a first threshold, and discarding forward prediction frames when the backlog delay exceeds a second threshold while retaining intra-frame coded keyframes.

[0017] Furthermore, a key frame index structure is established to support fast frame tracking after congestion relief. The key frame index structure records the frame number, timestamp, and offset of the key frame in the secondary persistent cache. When the downstream output congestion is relieved, the nearest key frame in the secondary persistent cache is located by searching the key frame index structure, and the output is resumed from the key frame while discarding the backlog data before the key frame to achieve fast frame tracking.

[0018] Furthermore, the phased recovery rule includes gradually increasing the transmission rate from a low rate level to a target rate level, maintaining a preset observation period in each stage, and entering the next stage when the cache occupancy rate is lower than the corresponding threshold, while limiting the increase in transmission rate per unit time to no more than a preset upper limit. The rollback includes rolling back the transmission rate to the previous stage level when the cache occupancy rate is higher than the rollback threshold or when the cache occupancy rate rises continuously for more than a preset duration.

[0019] The video surveillance real-time stream anomaly detection and steady-state scheduling method provided by this invention has at least the following beneficial effects: By unifying video data from different protocols or sources into the same internal data format and writing it into the data distribution channel for downstream modules to read, the access, aggregation, and forwarding processes are standardized, reducing the coupling of downstream modules to protocol differences and improving system scalability and reusability.

[0020] The system collects operational status indicators such as input rate, output rate, and buffer occupancy rate at a preset sampling period and writes them into a circular buffer that stores the most recent sampling points. This allows the system to retain short-term historical data with limited memory overhead, providing a reliable basis for subsequent baseline construction and trend judgment, and enhancing the continuity of detection and resistance to occasional jitter.

[0021] A dynamic baseline is established for input rate, output rate and buffer usage within a preset time window, and anomaly detection is performed accordingly. The threshold is adaptively adjusted according to changes in scene and load, avoiding the failure of fixed thresholds due to different camera bitrates, different network conditions or different business periods, thereby improving the accuracy and robustness of anomaly identification.

[0022] Based on the combination of abnormal states and operational status indicators, the system outputs status categories such as upstream input abnormal states, downstream output congestion states, and media interruption states, enabling structured attribution of abnormal sources. This facilitates operation and maintenance location (upstream, downstream, or media side) and provides a basis for differentiated flow control and caching strategies.

[0023] By suppressing repeated state transitions in the critical range through de-jitter confirmation and minimum dwell mechanism, the oscillations and jitters caused by frequent start-up and shutdown of flow control and frequent changes in transmission rate are avoided, thereby improving the stability and predictability of the system under complex network conditions.

[0024] A flow control policy containing token bucket parameters is generated and synchronized to the sending thread. The sending end performs rate control according to the policy, which can quickly converge the sending rate in the event of congestion or anomalies, and gradually increase it under normal conditions, subject to the maximum rate limit. At the same time, the results are updated synchronously through atomic variables, which facilitates lock-free reading by the sending thread, reduces lock contention overhead, and improves concurrency performance and real-time performance.

[0025] A two-level caching structure with at least two levels of memory caching and persistent caching is adopted. When downstream output is congested, overflow data is written to persistent cache, which reduces the risk of data loss caused by relying solely on memory caching and improves the carrying capacity and recoverability during sudden congestion and instantaneous jitter.

[0026] During downstream output congestion, a strategy of tiered discarding of predicted frames and priority retention of key frames is implemented. This can control the spread of backlog delays and retain the key data required for decoding and image reconstruction. In this way, the "usable image skeleton" can be maintained as much as possible when bandwidth is limited, thereby improving subjective viewing quality and decoding continuity.

[0027] A keyframe index structure is established to record the frame number, timestamp, and offset of the keyframe in the secondary persistent cache. This allows the system to quickly locate the nearest keyframe and resume output from that point when the downstream output congestion is relieved. At the same time, backlogged data before the keyframe is discarded, enabling fast frame tracking and reducing "delay trailing" caused by long-term backlog.

[0028] After the anomaly is resolved, the sending rate is gradually increased according to the phased recovery rules, and the buffer occupancy rate and its trend are observed in each phase. The system proceeds to the next phase only when the conditions are met; when the buffer occupancy rate exceeds the threshold or continues to rise, a rollback is executed. This closed-loop mechanism can avoid secondary congestion and overshoot caused by a one-time recovery, and improve the steady-state throughput and long-term operational stability of the system in dynamic network environments. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the method for real-time anomaly detection and steady-state scheduling of video surveillance according to the present invention. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0032] Example 1

[0033] like Figure 1 As shown, this embodiment provides a method for real-time anomaly detection and steady-state scheduling of video surveillance streams, applied to processing nodes for video stream access, aggregation, or forwarding, including: 1) At least one real-time video stream is accessed and processed, and video data from different protocols or sources is uniformly adapted to the same internal data format and written into the data distribution channel for one or more downstream processing modules to read. In this embodiment, the different protocols or sources may include at least one or more of RTSP, RTMP, GB / T 28181, HTTP-FLV, WebRTC, or vendor-specific protocols. The uniform adaptation may include standardizing fields such as the encapsulation format, timestamp / frame number, channel identifier, encoding type, and keyframe identifier of audio and video frames to form a unified internal frame structure or message structure. By masking protocol differences on the access side and writing them into the data distribution channel, downstream processing modules can obtain data in the same way, reducing redundant adaptation logic, lowering system coupling, and thus improving the portability and scalability of subsequent anomaly detection, flow control, and cache scheduling.

[0034] 2) The operating status indicators of the real-time video stream are collected at a preset sampling period. These indicators include at least the input rate, output rate, and buffer occupancy rate. The operating status indicators at each sampling time are written into a circular buffer used to store the most recent sampling points. In this embodiment, the sampling period can be set to a range of, for example, 100ms to 2s, according to the real-time requirements of the service. The input rate represents the amount of data or frames written to the data distribution channel per unit time. The output rate represents the amount of effective data or frames actually sent downstream by the sending thread per unit time. The buffer occupancy rate represents the proportion of occupancy of the primary memory buffer and / or the overall buffer, or the backlog latency. By writing each sampling point into the circular buffer and retaining only the most recent sampling points, a short-term historical sequence that can be used for trend judgment can be formed with relatively low memory overhead, improving the continuity of state perception and providing a stable data foundation for subsequent dynamic baseline calculation, thereby reducing the interference of occasional jitter on anomaly judgment.

[0035] 3) Based on the operating status indicators within the circular buffer, a dynamic baseline is established for the input rate, output rate, and buffer occupancy rate within a preset time window. Anomaly detection is then performed on the real-time video stream based on this dynamic baseline to determine the abnormal state. In this embodiment, the preset time window can correspond to the most recent N sampling points (e.g., 10-300 sampling points) in the circular buffer. The dynamic baseline can be generated using methods such as moving average, moving median, quantile upper and lower bounds, exponential moving average, or "mean ± k times standard deviation" to determine the baseline center value and its upper and lower bounds. When performing anomaly detection based on the dynamic baseline, criteria such as "input rate higher than the upper bound," "input rate lower than the lower bound," "output rate significantly lower than the target rate," and "buffer occupancy rate continuously increasing" can be used. This allows the threshold to adapt adaptively to different camera bitrate characteristics and network conditions, reducing false judgments caused by the failure of fixed thresholds in multiple scenarios and identifying congestion trends or abnormal fluctuations earlier.

[0036] 4) Based on the combination relationship between the abnormal state and the operating status indicators, the source of the abnormality is determined, and the output includes at least one of the following state categories: upstream input abnormal state, downstream output congestion state, and media interruption state. Frequent state switching is suppressed through debouncing confirmation and minimum dwell time mechanism. In this embodiment, the abnormality source determination can be comprehensively judged by combining multi-dimensional information such as "input rate - output rate difference / ratio", "buffer occupancy rate change trend", and "duration duration" to classify the abnormality into types such as upstream-side abnormality, downstream-side congestion, or media-side interruption. Debouncing confirmation can avoid frequent switching triggered by short-term jitter near the critical threshold; the minimum dwell time mechanism can ensure that after entering a certain abnormal state, the system remains in the shortest possible time to complete the effective scheduling action, thereby suppressing strategy oscillation, improving scheduling stability and predictability, and facilitating targeted handling of different abnormality causes.

[0037] 5) Generate a flow control strategy based on the state category and synchronize it to the sending thread. The flow control strategy includes at least a token bucket-based sending rate control parameter. In this embodiment, the flow control strategy can be generated at the session (stream) granularity, so that different video streams are isolated from each other and do not affect each other. The sending thread constrains the sending rhythm according to the token bucket parameter, so as to quickly reduce the sending rate and limit the burst when the downstream is congested or in an abnormal state, and then smoothly increase the sending rate when the state recovers, thereby achieving controllable rate convergence and recovery, reducing secondary congestion caused by instantaneous overshoot, and improving the continuity and stability of real-time preview.

[0038] 6) During flow control execution, at least two levels of caching are used for coordinated scheduling. The at least two levels of caching include a memory cache and a persistent cache. When the downstream output is congested, overflow data is written to the persistent cache and key frame data is prioritized for retention. When the downstream output congestion is relieved, the nearest key frame is located from the persistent cache and the output is restored. In this embodiment, the two-level cache coordination can be used to reduce the dependence on a single memory cache in large-scale concurrent scenarios: when the memory cache is close to full load, some backlogged data is selectively overflowed to the persistent cache to improve peak carrying capacity and reduce memory pressure. At the same time, prioritizing the retention of key frame data can ensure that there is the necessary reference for decoding and image reconstruction when the output is restored later. This can maintain the available image skeleton as much as possible when bandwidth is limited or the client processing capacity is insufficient, and shorten the recovery time after congestion is relieved.

[0039] 7) When the abnormal state is resolved or the recovery conditions are met, the transmission rate is gradually increased according to the phased recovery rules. During the recovery process, rollback or exit is performed based on the buffer occupancy rate and its changing trend to form a closed-loop steady-state scheduling. In this embodiment, phased recovery can gradually increase the transmission rate from a conservative level to the target level, and set an observation window at each stage to evaluate whether the buffer occupancy rate is stable or decreasing. When the buffer occupancy rate is detected to rise again or exceed the rollback threshold, it is promptly rolled back to the previous level, thus forming a closed-loop control process of increasing, observing, and rolling back. This avoids overshoot and oscillation caused by one-time recovery and improves steady-state throughput and long-term operational stability under weak network fluctuation environments.

[0040] As one implementation method, the state categories in this embodiment include at least a normal state, an upstream input anomaly state, a downstream output congestion state, and a media interruption state. The determination of the media interruption state includes the absence of media frames for a continuous preset duration. In this embodiment, the continuous preset duration can be set in conjunction with the frame rate and service tolerance (e.g., 1s to 10s), and can be further determined by combining the "difference between the timestamps of the most recently received frames." By separating the media interruption state from upstream anomalies and downstream congestion, misjudging situations such as "source-end disconnection / acquisition failure" as network congestion can be avoided, thereby allowing for the selection of more appropriate processing strategies (e.g., triggering reconnection, alarms, or degradation processing), improving the targeting and location efficiency of anomaly handling.

[0041] As one implementation method, the determination of downstream output congestion status in this embodiment includes satisfying one of the following conditions: a) the output rate is lower than a preset proportion of the target rate and the buffer occupancy rate shows an upward trend and continues to exceed a preset duration; b) the ratio of input rate to output rate exceeds a preset ratio threshold and the buffer occupancy rate accumulates rapidly. In this embodiment, by introducing a combination of conditions such as "the proportion of output rate lower than the target rate," "the upward trend of buffer occupancy rate," "the duration," and "the input / output rate ratio," false triggering caused by short-term network jitter or instantaneous transmission suppression can be reduced; among them, condition a is biased towards identifying congestion of "persistent transmission obstruction," and condition b is biased towards identifying congestion of "input significantly greater than output leading to rapid backlog." This combined determination is beneficial for identifying congestion trends earlier and triggering smooth flow control, reducing the probability of buffer overflow and latency propagation.

[0042] As one implementation method, the determination of the upstream input anomaly state in this embodiment includes meeting one of the following conditions: a) the input rate is higher than the upper bound of the dynamic baseline and the buffer occupancy rate suddenly increases; b) the input rate is lower than the lower bound of the dynamic baseline and causes the buffer occupancy rate to be quickly emptied. In this embodiment, condition a can be used to identify input-side pressure caused by upstream anomaly bursts (such as abnormal increase in device bitrate, excessively dense keyframes, abnormal repeated streaming, etc.); condition b can be used to identify input insufficiency caused by insufficient upstream supply (such as camera disconnection, encoding anomalies, link jitter causing intermittent upstream transmission). By comparing the upper and lower bounds of the dynamic baseline, "normal bitrate fluctuations" can be distinguished from "abnormal deviations," thereby improving the robustness of anomaly identification and providing a basis for subsequent differentiated scheduling strategies.

[0043] As one implementation method, the jitter confirmation in this embodiment includes setting a confirmation counter for each state category and confirming the state switch only when the confirmation counter reaches a preset threshold number of times. The minimum dwell time mechanism includes setting a minimum dwell time for upstream input abnormal state, downstream output congestion state, and media interruption state, and switching back to normal state only when the running state indicator meets the normal condition for a certain number of consecutive sampling cycles after the dwell time expires. In this embodiment, the confirmation counter can be accumulated according to the sampling cycle (e.g., switching only when the criterion is met for 3 to 10 consecutive sampling cycles), the minimum dwell time can be set according to different abnormal types (e.g., longer dwell time for congestion state, shorter dwell time for upstream abnormal state), and "returning to normal for a certain number of consecutive sampling cycles" can be required as the switchback condition. Through the above mechanism, the state can be suppressed from jumping back and forth around the threshold, avoiding oscillations caused by frequent start and stop of flow control and frequent switching of caching strategies, improving the overall scheduling stability, and reducing the adverse impact on the downstream playback experience.

[0044] As one implementation method, the flow control strategy in this embodiment adopts a token bucket model independent for each session. The token bucket parameters include the token generation rate and burst rate. The token generation rate is adjusted according to the state category: when the upstream input is in an abnormal state, it is reduced by a first scaling factor; when the downstream output is in a congested state, it is reduced by a second scaling factor; and when the state is normal, it is gradually increased by an increase factor, not exceeding the maximum rate limit. Furthermore, the updated token bucket parameters are synchronized to the sending thread via atomic variables, allowing the sending thread to read the token bucket parameters without locking. In this embodiment, the first and second scaling factors can be set as proportional coefficients less than 1, and can be set to different values ​​depending on the severity of the abnormality type (e.g., more conservative scaling when downstream is congested). The increase factor can be set as a gradual factor greater than 1 to achieve a "slow climb" recovery method. The maximum rate limit can be related to the target rate or link bandwidth assessment result. By synchronizing the token bucket parameters via atomic variables, multi-threaded lock contention can be reduced, allowing the sending thread to obtain the latest strategy in real time with lower overhead. This makes it more suitable for large-scale concurrent session scenarios, improving system real-time performance and throughput stability.

[0045] As one implementation, the at least two-level cache in this embodiment includes a first-level memory cache and a second-level persistent cache. The first-level memory cache is a lock-free circular queue used to cache hot data, and the second-level persistent cache is a memory-mapped file-based cache used to cache overflowed data when the first-level memory cache overflows. During output scheduling, data is preferentially read from the first-level memory cache, and when the first-level memory cache is empty, data is read from the second-level persistent cache. In this embodiment, the first-level memory cache carries low-latency hot data paths, suitable for real-time forwarding under normal conditions; the second-level persistent cache carries cold / backlogged data under congestion or sudden events, reducing peak memory usage and improving recoverability. Prioritizing reading from the first-level memory cache during output scheduling ensures minimal latency on normal paths; reading from the second-level persistent cache only when the first-level memory cache is empty or exhausted ensures orderly recovery of historical backlogged data while maintaining real-time performance, thus balancing real-time low latency and congestion resistance.

[0046] As one implementation method, this embodiment executes a frame-dropping-skeleton-preserving and keyframe-priority strategy during downstream output congestion. The frame-dropping-skeleton-preserving strategy includes discarding bidirectional prediction frames when the backlog delay exceeds a first threshold, and discarding forward prediction frames when the backlog delay exceeds a second threshold while retaining intra-frame encoded keyframes. In this embodiment, the first and second thresholds can be set to different levels, and a strategy selection that satisfies the condition that the first threshold is greater than or less than the second threshold is applied (e.g., more aggressively discarding bidirectional prediction frames to reduce backlog faster). Prioritizing keyframe retention ensures that the decoding reference chain remains intact, avoiding situations where "discarding keyframes leads to prolonged periods without displaying the image." By tiered discarding of different types of prediction frames, the spread of backlog delay can be controlled during congestion, and image continuity and availability can be maintained as much as possible under limited bandwidth, thereby reducing stuttering and improving the user's subjective experience.

[0047] As one implementation method, this embodiment establishes a keyframe index structure to support rapid frame tracking after congestion relief. The keyframe index structure records the frame number, timestamp, and offset of each keyframe in the secondary persistent cache. When downstream output congestion is relieved, the nearest keyframe in the secondary persistent cache is located by searching the keyframe index structure, and output is resumed from that keyframe while discarding backlogged data before that keyframe to achieve rapid frame tracking. In this embodiment, the keyframe index structure can be maintained incrementally by time or frame number and can be updated synchronously when writing to the secondary persistent cache. After congestion relief, by locating the "nearest keyframe" and discarding earlier backlogged data, long delays caused by replaying from too early a position can be avoided, thus achieving rapid return to real-time performance. This mechanism can shorten recovery time and improve the stability of the recovery phase, and is particularly suitable for repeated congestion-relief scenarios under weak network fluctuations.

[0048] As one implementation method, the phased recovery rule in this embodiment includes gradually increasing the transmission rate from a low rate level to a target rate level, maintaining a preset observation period in each stage, and entering the next stage when the buffer occupancy rate is lower than the corresponding threshold, while limiting the increase in transmission rate per unit time to no more than a preset upper limit. The rollback includes rolling back the transmission rate to the previous stage level when the buffer occupancy rate is higher than the rollback threshold or when the buffer occupancy rate rises continuously for more than a preset duration. In this embodiment, the preset observation period is used to verify whether the improved system has entered a new stable range; the upper limit is used to limit the rate change per unit time to avoid introducing secondary congestion due to excessively rapid recovery; the rollback threshold and duration condition can be used to identify the risk of "re-backlog during the recovery stage" and roll back in time. Through the above-mentioned phased improvement and rollback mechanism, a closed-loop steady-state control with buffer occupancy rate and its trend as feedback can be formed, reducing the probability of recovery jitter and oscillation, and improving the stability and controllability of long-term operation.

[0049] Example 2

[0050] Based on Embodiment 1, this embodiment provides a more refined implementation, still applied to processing nodes for video stream access, aggregation, or forwarding. The processing node may include an input adaptation layer, an indicator collector, an anomaly detector, a flow control strategy engine, a multi-level cache manager, a transmission scheduler, and a self-healing module. These modules work together to achieve unified video data access, operational status indicator sampling, dynamic baseline anomaly detection, status de-jittering and minimum dwell time, session-level flow control, multi-level cache collaboration, keyframe frame tracking and recovery, and anomaly self-healing and maintenance analysis, thereby achieving steady-state scheduling and closed-loop control. Steps 1) to 7) of this embodiment are consistent with Embodiment 1. This embodiment only supplements the optional refinements of each step without changing the basic process of Embodiment 1, as follows: 1) At least one real-time video stream is accessed and processed, and video data from different protocols or sources is uniformly adapted to the same internal data format and written into the data distribution channel for one or more downstream processing modules to read. As one implementation, the different protocols or sources mentioned in this embodiment include, but are not limited to, one or more of RTSP / RTP, PS / TCP, HTTP-FLV, and vendor-specific SDK callbacks. The uniform adaptation may include: defining a unified adapter interface, which at least includes a data receiving function; and encapsulating the accessed media frames into a unified internal message structure, which may include fields such as session identifier, device identifier, protocol type, frame data payload, timestamp or frame sequence number, and keyframe identifier. As one implementation, the data distribution channel in this embodiment is a data bus, which adopts a single-write, multi-read data broadcast mechanism, enabling the cache writing module, indicator acquisition module, and sending scheduling module to read data in the same internal data format in parallel, thereby reducing the overhead of repeated adaptation and copying.

[0051] 2) Collect the operating status indicators of the real-time video stream at a preset sampling period. The operating status indicators include at least the input rate, output rate, and buffer occupancy rate. Write the operating status indicators at each sampling time into a circular buffer used to store the most recent sampling points. As one implementation, the operating status indicators in this embodiment include: input rate (… ), output rate ( ) and cache utilization ( ); where the input rate ( The output rate represents the amount of data entering the processing node from the device side per unit time. ) represents the amount of data sent from the processing node to the downstream per unit time, and the cache utilization rate ( This represents the ratio of the current cache usage to the total cache capacity, with a value range of 0 to 1.

[0052] As one implementation method, the sampling period described in this embodiment is a configurable period, preferably 50ms to 500ms, more preferably about 100ms; the sampling method can be implemented by counting the data bus input flow and the transmission queue (or the number of successfully transmitted bytes), and can adopt lock-free counting or low-lock contention method to reduce sampling overhead.

[0053] As one implementation method, the circular buffer in this embodiment is a fixed-size circular structure used to store the most recent N sampling points, where N is a configurable integer, preferably 20 to 1000, and more preferably about 50 sampling points.

[0054] 3) Based on the operating status indicators within the circular buffer, a dynamic baseline is established for the input rate, output rate, and buffer occupancy rate within a preset time window. Anomaly detection is then performed on the real-time video stream based on this dynamic baseline to obtain anomaly status. As one implementation method, in this embodiment, the dynamic baseline is calculated using an exponentially weighted moving average (EWMA) to obtain a smoothed mean. ), and combined with standard deviation ( Constructing dynamic baseline intervals , where the coefficient ( ) is a configurable coefficient, preferably 2 to 4, more preferably about 3.

[0055] As one implementation, the Exponentially Weighted Moving Average (EWMA) update satisfies:

[0056] in This is the current sampled value. It is a smoothing factor, with a value range of 0 to 1, which can be configured to balance response speed and anti-jitter capability.

[0057] The standard deviation Updates can be done using a recursive method:

[0058] As one implementation method, the preset time window in this embodiment covers a sampling history of several seconds to several minutes; for example, under a 100ms sampling period, tens to hundreds of sampling points can be selected for baseline updates so that the baseline can be adaptively adjusted with changes in network and bit rate.

[0059] 4) Based on the combination relationship between the abnormal state and the operating state index, the abnormal source is identified, and the output includes at least one of the following state categories: upstream input abnormal state, downstream output congestion state, and media interruption state. Frequent state switching is suppressed through de-jitter confirmation and minimum dwell mechanism. As one implementation method, the state categories described in this embodiment include at least normal state, upstream input abnormal state, downstream output congestion state, and media interruption state.

[0060] As one implementation method, the determination of the downstream output congestion state in this embodiment includes satisfying one of the following conditions: a) Output rate ( The cache occupancy rate is lower than the preset percentage of the target rate (target rate, target_rate) and the cache utilization rate is ( The cache utilization rate shows an upward trend and continues to exceed a preset duration; wherein the preset ratio is preferably 0.5 to 0.9, more preferably about 0.7 to 0.8; the upward trend can be characterized by the cache utilization rate change rate (cache utilization rate change rate, dBF), wherein the cache utilization rate change rate (dBF) is the cache utilization rate per unit time (… The rate of change of the ) is preferably a threshold of about 1% / s to 10% / s; the duration is preferably 0.5s to 10s, more preferably about 1s to 5s; b) The ratio of input rate to output rate Exceeding the preset ratio threshold and cache utilization ( Rapid accumulation; wherein the ratio threshold is preferably 1.05 to 2.0, more preferably about 1.2.

[0061] As one implementation method, the determination of the upstream input abnormal state in this embodiment includes satisfying one of the following conditions: a) Output rate ( ) is higher than the upper bound of the dynamic baseline of the input rate and the cache utilization rate ( A sudden increase; b) Input rate ( The input rate falls below the lower bound of the dynamic baseline and causes a decrease in cache occupancy. Quickly empty the container.

[0062] As one implementation method, the determination of the media interruption state in this embodiment includes not receiving media frame data for a continuous preset duration. The preset duration is a configurable duration, preferably 0.5s to 20s, and more preferably about 2s to 10s.

[0063] As one implementation method, the de-jitter confirmation in this embodiment includes setting a confirmation counter for each state category, and confirming the state switch only when the confirmation counter reaches a preset number of times threshold; the preset number of times threshold is a configurable integer, preferably 3 to 20, and more preferably about 5.

[0064] As one implementation method, the minimum dwell time mechanism described in this embodiment includes setting minimum dwell time for upstream input abnormal state, downstream output congestion state and media interruption state respectively, and switching back to normal state only when the running status index meets the normal conditions for a certain number of consecutive sampling cycles after the dwell time expires; wherein the minimum dwell time is preferably in the range of 0.5s to 60s; the number of sampling cycles that meet the normal conditions for a certain number of consecutive cycles is preferably 2 to 10, more preferably about 3.

[0065] 5) Generate a flow control policy based on the state category and synchronize it to the sending thread. The flow control policy includes at least a token bucket-based sending rate control parameter. As one implementation method, the flow control strategy described in this embodiment adopts a token bucket model that is independent for each session. The token bucket parameters include the token generation rate (rate) and the burst size or bucket capacity (burst_size). The sending thread reads data from the cache and sends it based on the token bucket authorization.

[0066] In one implementation, this embodiment adjusts the token generation rate according to the state category: when the upstream input is in an abnormal state, it is reduced by a first scaling factor; when the downstream output is in a congested state, it is reduced by a second scaling factor; when the state is normal, it is gradually increased by a boosting factor, but does not exceed the maximum rate limit (maximum rate limit, max_rate). The first and second scaling factors are both coefficients less than 1, preferably 0.4 to 0.95; the boosting factor is a coefficient greater than 1, preferably 1.01 to 1.2.

[0067] As one implementation method, in this embodiment, the update result of the token bucket parameter is synchronized to the sending thread through an atomic variable or a lock-free synchronization mechanism, so that the sending thread can read the token bucket parameter without a lock, thereby reducing the lock contention overhead in high-concurrency session scenarios.

[0068] As one implementation method, this embodiment can record an exception log and release or reset some resources in the media interruption state, and trigger a reconnection process when the configuration allows.

[0069] 6) During the flow control process, at least two levels of cache are used for coordinated scheduling. The at least two levels of cache include a memory cache and a persistent cache. When the downstream output is in a congested state, the overflow data is written to the persistent cache and the key frame data is kept in priority. When the downstream output congestion is relieved, the nearest key frame is located from the persistent cache and the output is restored. In one implementation, the at least two-level cache in this embodiment includes a first-level memory cache (L1) and a second-level persistent cache (L2). The first-level memory cache (L1) is a lock-free circular queue used to cache frequently accessed data, and the second-level persistent cache (L2) is a memory-mapped file-based cache used to cache overflowed data when the first-level memory cache (L1) overflows. Both the capacity of the first-level memory cache (L1) and the capacity of the second-level persistent cache (L2) are configurable. The first-level memory cache (L1) is preferably 64KB to 8MB, and the second-level persistent cache (L2) is preferably 1MB to 1GB.

[0070] In one implementation, under normal conditions, data is preferentially written to the first-level memory cache (L1) and the sending scheduler preferentially reads data from the first-level memory cache (L1) for transmission. Under downstream output congestion, when the first-level memory cache (L1) reaches the threshold or is fully loaded, the overflow data is written to the second-level persistent cache (L2), and the sending scheduler preferentially reads data from the first-level memory cache (L1) and reads data from the second-level persistent cache (L2) when the first-level memory cache (L1) is empty.

[0071] As one implementation method, this embodiment executes a frame dropping and skeleton preservation and key frame priority strategy during downstream output congestion. The frame dropping and skeleton preservation strategy includes dropping bidirectional prediction frames when the backlog delay exceeds a first threshold and dropping forward prediction frames when the backlog delay exceeds a second threshold while retaining intra-frame coded key frames. The first threshold and the second threshold are configurable delay thresholds, preferably 100ms to 5s.

[0072] As one implementation method, this embodiment establishes a keyframe index structure to support fast frame tracking after congestion relief. The keyframe index structure records the frame number, timestamp, and offset of the keyframe in the secondary persistent cache (L2). The keyframe index structure can be a fixed-size circular array with a capacity of configurable entries, preferably 50 to 5000 entries, more preferably about 100 entries. During the recovery phase, the nearest keyframe is located by searching the keyframe index structure, and the output is resumed from that keyframe, while discarding the backlog data before that keyframe.

[0073] 7) When the abnormal state is resolved or the recovery conditions are met, the sending rate is gradually increased according to the phased recovery rules, and rollback or exit is executed based on the buffer occupancy rate and its changing trend during the recovery process to form a closed-loop steady-state scheduling.

[0074] As one implementation method, the phased recovery rule in this embodiment includes gradually increasing the sending rate from a low rate level to a target rate level, and maintaining a preset observation period in each stage; when the cache occupancy rate ( When the value is below the corresponding threshold and the trend is stable, proceed to the next stage; at the same time, limit the increase in the transmission rate per unit time to no more than a preset upper limit. The observation period is preferably 1s to 30s; the upper limit is preferably 1% to 10% per second.

[0075] As one implementation method, the rollback described in this embodiment includes when the cache occupancy rate ( ) higher than the fallback threshold or cache occupancy ( If the transmission rate continuously increases for more than a preset duration, the transmission rate will be rolled back to the previous level or a more conservative level; the rollback threshold and duration are both configurable parameters; the exit includes reaching the target rate and the buffer occupancy rate ( The recovery process will exit after a preset duration within the preset stable range.

[0076] As one implementation method, this embodiment may also include dynamic bitrate and connection optimization: when the upstream input is in an abnormal state and the device supports dynamic bitrate, the bitrate reduction control of the device encoding parameters is triggered; the bitrate reduction step size and lower limit are configurable parameters, and after the bitrate reduction is lifted, it can be delayed for a preset time and then restored in stages according to the preset step size; if the abnormal state is entered again during the recovery process, the recovery is stopped or rolled back.

[0077] As one implementation method, this embodiment may also include protocol switching and fallback logic: when protocol instability or long-term congestion is detected to meet preset trigger conditions, the protocol switches between multiple bearer modes according to preset priorities, and keeps the session identifier unchanged to be transparent to the upper layer; when the new protocol is running stably and the system load meets the conditions, the original protocol is created to probe the connection in parallel. If the probe is successful, the protocol falls back; if the probe fails, the protocol enters a degraded mode and retryes intermittently over a longer period of time. A suppression strategy is set for the number of consecutive failures to avoid frequent switching.

[0078] As one implementation method, this embodiment may also include abnormal snapshots and root cause analysis: an abnormal snapshot is generated when a state switch, protocol switch, or manual triggering occurs. The snapshot includes basic session information and the input rate over a recent period of time. ) / Output rate ( ) / Cache utilization ( The system can analyze the sequence and statistics, frame loss statistics, current flow control parameters, parameter versions, and node resource usage, and output root cause analysis conclusions based on preset rules.

[0079] The above embodiments are merely exemplary implementations of the present invention. For those skilled in the art, various equivalent modifications or substitutions can be made to the threshold, time window, cache capacity, scaling factor, recovery phase division, and triggering conditions without departing from the concept of the present invention, and all such modifications or substitutions should fall within the protection scope of the present invention.

[0080] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for real-time anomaly detection and steady-state scheduling of video surveillance streams, applied to processing nodes for video stream access, aggregation, or forwarding, characterized in that: include: 1) Process at least one real-time video stream, adapt video data from different protocols or sources to the same internal data format, and write it into the data distribution channel for one or more downstream processing modules to read; 2) Collect the running status indicators of the real-time video stream at a preset sampling period. The running status indicators include at least the input rate, output rate and buffer occupancy rate. Write the running status indicators at each sampling time into a circular buffer for storing the most recent sampling points. 3) Based on the operating status indicators within the circular buffer, a dynamic baseline is established for the input rate, output rate, and buffer occupancy rate within a preset time window, and anomaly detection is performed on the real-time video stream according to the dynamic baseline to obtain the abnormal status. 4) Based on the combination relationship between the abnormal state and the operating state index, the abnormal source is identified, and the output includes at least one of the following state categories: upstream input abnormal state, downstream output congestion state, and media interruption state. Frequent state switching is suppressed through de-jitter confirmation and minimum dwell mechanism. 5) Generate a flow control policy based on the state category and synchronize it to the sending thread. The flow control policy includes at least a token bucket-based sending rate control parameter. 6) During the flow control process, at least two levels of cache are used for coordinated scheduling. The at least two levels of cache include a memory cache and a persistent cache. When the downstream output is in a congested state, the overflow data is written to the persistent cache and the key frame data is kept in priority. When the downstream output congestion is relieved, the nearest key frame is located from the persistent cache and the output is restored. 7) When the abnormal state is resolved or the recovery conditions are met, the sending rate is gradually increased according to the phased recovery rules, and rollback or exit is executed based on the buffer occupancy rate and its changing trend during the recovery process to form a closed-loop steady-state scheduling.

2. The method according to claim 1, characterized in that: The state categories include at least normal state, upstream input abnormal state, downstream output congestion state, and media interruption state, wherein the determination of the media interruption state includes not receiving media frames for a continuous preset time.

3. The method according to claim 2, characterized in that: The determination of downstream output congestion status includes meeting one of the following conditions: a) the output rate is lower than a preset ratio of the target rate and the cache occupancy rate shows an upward trend and continues to exceed a preset duration; b) the ratio of input rate to output rate exceeds a preset ratio threshold and the cache occupancy rate accumulates rapidly.

4. The method according to claim 2, characterized in that: The determination of the abnormal state of the upstream input includes meeting one of the following conditions: a) the input rate is higher than the upper limit of the dynamic baseline and the cache occupancy rate suddenly increases; b) the input rate is lower than the lower limit of the dynamic baseline and causes the cache occupancy rate to be quickly emptied.

5. The method according to claim 2, characterized in that: The de-jitter confirmation includes setting a confirmation counter for each state category and confirming the state switch only when the confirmation counter reaches a preset number of times threshold. The minimum dwell time mechanism includes setting a minimum dwell time for upstream input abnormal state, downstream output congestion state and media interruption state respectively, and switching back to normal state only when the running status index meets the normal conditions for a certain number of consecutive sampling cycles after the dwell time expires.

6. The method according to claim 2, characterized in that: The flow control strategy adopts a token bucket model independent for each session. The token bucket parameters include the token generation rate and the burst rate. The token generation rate is adjusted according to the state category. Specifically, when the upstream input is in an abnormal state, the rate is reduced by a first scaling factor; when the downstream output is in a congested state, the rate is reduced by a second scaling factor; and when the state is normal, the rate is gradually increased by an increase factor, but does not exceed the maximum rate limit. Furthermore, the update results of the token bucket parameters are synchronized to the sending thread through atomic variables so that the sending thread can read the token bucket parameters without locking.

7. The method according to claim 1, characterized in that: The at least two-level cache includes a first-level memory cache and a second-level persistent cache, wherein the first-level memory cache is a lock-free circular queue used to cache hot data, and the second-level persistent cache is a cache based on a memory-mapped file used to cache overflow data when the first-level memory cache overflows, and during output scheduling, it preferentially reads from the first-level memory cache and reads from the second-level persistent cache when the first-level memory cache is empty.

8. The method according to claim 2, characterized in that: During downstream output congestion, a frame-dropping-skeleton-preserving and key-frame-priority strategy is implemented. The frame-dropping-skeleton-preserving strategy includes discarding bidirectional prediction frames when the backlog delay exceeds a first threshold, and discarding forward prediction frames when the backlog delay exceeds a second threshold while retaining intra-frame coded keyframes.

9. The method according to claim 7, characterized in that: To support fast frame tracking after congestion is relieved, a key frame index structure is established. The key frame index structure records the frame number, timestamp, and offset of the key frame in the secondary persistent buffer. When the downstream output congestion is relieved, the nearest key frame in the secondary persistent buffer is located by searching the key frame index structure, and the output is resumed from the key frame, while the backlog data before the key frame is discarded to achieve fast frame tracking.

10. The method according to claim 1, characterized in that: The phased recovery rule includes gradually increasing the transmission rate from a low rate level to a target rate level, maintaining a preset observation period in each stage, and entering the next stage when the cache occupancy rate is lower than the corresponding threshold, while limiting the increase in transmission rate per unit time to no more than a preset upper limit. The rollback includes rolling back the transmission rate to the previous stage level when the cache occupancy rate is higher than the rollback threshold or when the cache occupancy rate rises continuously for more than a preset duration.

Images