Quality monitoring abnormity positioning method and system for multi-source heterogeneous media data

Abstract

The invention discloses a quality monitoring abnormity positioning method and system for multi-source heterogeneous media data, and particularly relates to the technical field of quality monitoring of the multi-source heterogeneous media data.The method comprises the steps that through access and time base normalization, sources such as RTSP, HLS / DASH, offline files and message buses are mapped to a unified time reference, and out-of-order rearrangement and late cutting are carried out; basic quality elements are extracted from A / V / S in a sliding window, a dynamic threshold value is formed by adopting a trimming mean value and a median absolute deviation, and a window-level quality event is generated; using event anchor points as common reference to estimate cross-modal time delay and fuse evidences, outputting scene / mode labels by means of a lightweight online classifier, entering a confidence reduction mode when the evidences are missing, and outputting an availability bitmap and a reason code; and then event confirmation, time-space range refinement and hierarchical root cause posterior determination are completed, influence measurement and priority are calculated, and finally an anomaly positioning and root cause analysis record set including an event timeline, severity / influence degree, root causes and disposal suggestions is formed.

Description

Technical Field

[0001] This invention relates to the field of multi-source heterogeneous media data quality monitoring technology, and more specifically, to a method and system for locating anomalies in the quality monitoring of multi-source heterogeneous media data. Background Technology

[0002] With the continued widespread adoption of streaming acquisition and broadcasting technologies such as RTSP and HLS / DASH, quality monitoring and anomaly localization for multi-source heterogeneous media data have become crucial for operational support. Existing solutions often rely on single-modal or weakly coupled multi-modal metrics, leading to common problems such as: the inability to unify PTS / DTS and arrival times from different sources to the same time base, resulting in difficulties in cross-modal correlation and backtracking; the lack of engineered out-of-order reordering and monotonicity guarantees when dealing with variable frame rates, out-of-order, and late segments, easily causing statistical bias and false alarms; and thresholds that are mostly fixed empirical values ​​without adaptive adjustment based on content format and network operation, resulting in inconsistent alarm sensitivity in high-speed motion scenes or static interview scenarios.

[0003] On the other hand, existing toolchains do not adequately support the auditability of evidence, lacking a mechanism to package robust statistics, dynamic thresholds, evidence pointers, and availability bitmaps within a window, making it difficult to support subsequent root cause analysis and verification; the lack of a unified characterization of relative time delay, phase consistency, and event anchors between A / V / S makes it difficult for the same objective defect to form sparse and stable candidate events in multimodal scenarios; and the lack of explicit confidence reduction strategies when facing upstream missing data or incomplete observations leads to unclear boundaries in localization conclusions.

[0004] In summary, a multimodal quality monitoring and anomaly localization technology oriented towards a sliding window under a unified time benchmark is needed. This technology should be able to bridge across modalities using event anchor points as a common reference, combine a lightweight online classifier to obtain scene / pattern labels to adaptively adjust fusion weights and judgment thresholds, and maintain interpretable, auditable, and engineering-applicable alarm standards even in scenarios with missing evidence. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a method and system for quality monitoring and anomaly location of multi-source heterogeneous media data to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for quality monitoring and anomaly localization of multi-source heterogeneous media data includes the following steps:

[0008] Step 1: Map the original timestamps from each source to the system's unified time base, perform out-of-order rearrangement and late-arrival discard or late-arrival truncation processing to obtain an ordered sequence of audio, video, and subtitles aligned under the unified time base;

[0009] Step 2: Under a unified time reference, extract the basic quality vector and modal structure features of audio, video and subtitle modalities respectively using a sliding analysis window W(t). Generate dynamic thresholds based on robust statistics to form a feature record sequence that includes video basic quality vector, audio basic quality vector, subtitle basic quality vector, robust statistics, dynamic thresholds, quality event list, evidence pointer and availability bitmap.

[0010] Step 3: Using the feature record sequence as input, construct a sparse event anchor set under a unified time reference, estimate cross-modal delay and phase consistency, and output scene / mode labels M through an online learning classifier based on neutral context fields; aggregate multimodal evidence around the event anchors, calculate the consistency score of candidate quality events, and output a bridging analysis record set containing event timelines, scene / mode labels M, candidate event types, start and end times, severity, confidence, trigger evidence summaries, and availability bitmaps;

[0011] Step 4: Using the bridging analysis record set and feature record sequence as input, perform temporal continuity verification and confirmation on candidate quality events, refine spatial or semantic scope localization, obtain posterior root cause distribution based on multi-evidence fusion, calculate unified impact metric and priority score, and output an anomaly localization and root cause analysis record set that includes event start and end time, spatial or semantic scope description, severity, impact metric, posterior root cause distribution, reason code, availability bitmap, and operation and maintenance handling suggestions.

[0012] In a preferred embodiment, in step one, the process of mapping the original timestamps from each source to the system's unified time base is achieved through linear mapping and minor correction, specifically including a proportional correction coefficient, an initial phase offset, and a time variation correction amount constrained within a safe range.

[0013] In a preferred embodiment, in step two, the process of generating a dynamic threshold based on robust statistics includes: for any metric within the window, calculating its trimmed mean and median absolute deviation, and setting a dynamic threshold based on the product of the sum of the two and a robust coefficient; a window-level quality event is generated only when the duration of continuous over-threshold exceeds the minimum significant duration of the metric.

[0014] In a preferred embodiment, the process of constructing a sparse event anchor set in step three includes: when a shot change, inter-frame difference energy abrupt change, or structural coding step occurs on the video side, and an energy envelope or spectral centroid transition occurs simultaneously on the audio side, and a line-level time code density abrupt change or large gap closure occurs on the subtitle side, a single anchor point is established in the least common time neighborhood of the signal, and the mode involved in alignment and the triggering evidence are recorded; for time periods with only single modal significance, the single modal anchor point is retained and the availability bitmap is explicitly marked.

[0015] In a preferred embodiment, in step three, the scene / mode label M is extracted by a lightweight online classifier. The classifier takes a neutral context field from the feature record sequence as input, and the field includes one or more of the following: motion level, voice activity ratio, caption duty cycle, bitrate and resolution range, arrival jitter and buffer usage. When the classifier is running, it takes the context statistics of the anchor neighborhood as input and outputs the current scene or mode label M.

[0016] In a preferred embodiment, in step three, the process of calculating the consistency score of the candidate quality event includes: weighted summation of normalized anomaly metrics related to the anchor point neighborhood within the window; and generating a candidate quality event when the consistency score exceeds a dynamic threshold and the duration spans the minimum significant duration.

[0017] In a preferred embodiment, in step four, the process of obtaining the posterior root cause distribution based on multi-evidence fusion includes: using a log-linear model to robustly fuse the evidence in order to calculate the posterior confidence of each candidate root cause.

[0018] In a preferred embodiment, the process of calculating a unified impact metric in step four includes: combining the event duration, severity, affected coverage, and scene weighting function, and obtaining the impact metric through weighted summation; the priority score is based on the impact metric and incorporates event novelty and recurrence factors.

[0019] In a preferred embodiment, the following modules are included:

[0020] The access and time base normalization module is used to map the original timestamps from various sources to the system's unified time base, perform out-of-order rearrangement and late-arrival discard or late-arrival truncation processing, and obtain an ordered sequence of audio, video and subtitles aligned under the unified time base.

[0021] The basic quality and modal feature extraction module is used to extract basic quality vectors and modal structure features of audio, video and subtitle modalities respectively under a unified time reference using a sliding analysis window W(t). Based on robust statistics, dynamic thresholds are generated to form a feature record sequence containing video basic quality vector, audio basic quality vector, subtitle basic quality vector, robust statistics, dynamic thresholds, quality event list, evidence pointer and availability bitmap.

[0022] The Cross-Modal Bridging and Scene / Mode Label M module is used to construct a sparse event anchor set under a unified time reference, take the feature record sequence as input, estimate cross-modal latency and phase consistency, and output scene / mode labels M through an online learning classifier based on neutral context fields; it aggregates multimodal evidence around the event anchors, calculates the consistency score of candidate quality events, and outputs a bridging analysis record set containing event timelines, scene / mode labels M, candidate event types, start and end times, severity, confidence, trigger evidence summaries, and availability bitmaps;

[0023] The anomaly localization and root cause analysis module is used to verify and confirm the temporal continuity of candidate quality events by taking the bridging analysis record set and feature record sequence as input, perform refined localization of spatial or semantic scope, obtain the posterior root cause distribution based on multi-evidence fusion, calculate a unified impact metric and priority score, and output an anomaly localization and root cause analysis record set that includes event start and end time, spatial or semantic scope description, severity, impact metric, posterior root cause distribution, reason code, availability bitmap, and operation and maintenance handling suggestions.

[0024] The technical effects and advantages of this invention are as follows:

[0025] This invention uses a unified time base and a sliding window W(t) to integrate multi-source access, feature extraction, and bridging analysis. By performing out-of-order rearrangement and late-arrival discarding / truncation on RTSP, HLS / DASH, offline files, and message bus data, it ensures strict monotonic alignment of A / V / S under a unified time base. Within W(t), a dynamic threshold is established using the trimmed mean and median absolute deviation, triggering quality events only when the minimum significant duration is exceeded. Combining event anchors and minimal evidence, it achieves traceable, replayable, and auditable statistical standards, significantly reducing false alarms and missed alarms caused by jitter, missing fragments, and variable frame rates.

[0026] The system aggregates multimodal evidence around event anchors, extracts scene / pattern labels M online using neutral context fields, and adaptively adjusts fusion weights and judgment thresholds. When evidence is missing, it enters a reduced confidence mode using availability bitmaps and reason codes to ensure clear conclusion boundaries. Subsequently, it obtains the posterior root cause distribution through a log-linear model and outputs bridging analysis record sets and anomaly localization and root cause analysis record sets with unified impact metrics and priority scores, achieving an end-to-end closed loop from alarm to handling recommendations, combining cross-scenario robustness with engineering feasibility. Attached Figure Description

[0027] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0028] Figure 1 This is a flowchart illustrating the method for locating anomalies in the quality monitoring of multi-source heterogeneous media data according to the present invention.

[0029] Figure 2 This is a schematic diagram illustrating the core principle of the cross-modal bridging analysis and anomaly localization mechanism of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of the quality monitoring and anomaly location system for multi-source heterogeneous media data of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] For ease of reading and implementation, this paper makes the following stipulations regarding several terms: RTSP refers to Real-Time Streaming Protocol; HLS refers to HTTP Live Streaming based on HTTP adaptive transmission; DASH refers to Dynamic Adaptive Streaming over HTTP adaptive bitstream standard; HDFS refers to Hadoop Distributed File System; GOP refers to Group of Pictures image frame structure; PTS refers to Presentation Time Stamp; DTS refers to Decoding Time Stamp; late-drop / late-trunc refer to late-arrival drop and late-arrival truncation, respectively, referring to the processing strategies for data frames or fragments that arrive after exceeding the analysis window or playback time limit; the former is directly discarded, while the latter truncates and retains late-arriving content while maintaining temporal consistency. A pipeline refers to the processing pipeline, which is a directed processing flow composed of parallel modules such as demultiplexing, decoding or resampling, feature extraction, detection, and aggregation. GStreamer and DeepStream are multimedia processing and parallel encoding / decoding frameworks. In this paper, they are only used as optional implementation examples for quickly assembling multiple real-time processing streams and hardware acceleration. Kafka is a distributed publish-subscribe messaging middleware used to decouple data access, computation, and archiving. Unless otherwise stated, the above product names are merely exemplary implementations. This invention is not limited to any specific vendor or product, and can be replaced with frameworks or middleware with equivalent functionality without changing the functionality and effect.

[0034] This invention provides a method for quality monitoring and anomaly location of multi-source heterogeneous media data, such as... Figure 1 As shown, it includes the following steps:

[0035] Step 1: Multi-source access and time base normalization

[0036] The system first establishes stable connections with various media sources through a horizontally scalable access gateway and streaming processing framework. These sources include real-time streaming based on session and adaptive protocols, such as RTSP, HLS, and DASH, as well as offline file batch processing, covering media files on object storage and HDFS, and segmented data or feature streams pushed by a message bus, which can use Kafka topics. While maintaining transparency to the upstream, the access side automatically identifies and decapsulates protocols and containers, parses and outputs unified metadata for subsequent calculations, including video bitrate, resolution, frame rate, GOP structure, audio sampling rate, channel layout, encapsulation timestamps PTS and DTS, arrival time, and jitter statistics. All of this metadata and media payload are bound to corresponding source and session identifiers to ensure cross-source traceability and playback consistency.

[0037] To eliminate the impact of device local clock skew, initial phase difference caused by network paths, and random jitter on downstream analysis, this step maps the raw time metrics from various sources to the system time base. (in milliseconds), and enforces the monotonicity of this benchmark on the window scale. Specifically, for any source i, the source-side timestamp The system calculates its mapping value to a unified time base. .in, Indicates source clock and The proportional correction factor is used to compensate for constant frequency drift; Indicates the initial phase offset at the time of session establishment; This is a small, time-varying correction amount used to absorb sudden arrival jitter, time rollback, or packaging anomalies, and its amplitude is constrained to... Within a safe range to ensure that the monotonication process does not introduce irreversible distortion.

[0038] The above parameters are updated online throughout the session lifecycle using a combination of sliding estimation and robust statistics. When switching sources or reconnecting, a warm start is performed based on the statistics of the most recent stable segment to shorten the convergence time and avoid amplification of cold start jitter.

[0039] After time base normalization, the system uses a timestamp-ordered circular buffer as the convergence and rearrangement carrier for the access layer. For out-of-order or late-arriving segments, if their... If a segment falls within the reorderable window without disrupting the monotonicity of the submitted sequence, it is inserted into the circular buffer according to its time position, and the order index of adjacent segments is corrected. If it exceeds the reorder window or causes an unacceptable risk of reversal, the segment is labeled with a late-drop / late-trunc quality tag and the minimum evidence is retained. The minimum evidence is the index and summary statistics, so that subsequent anomaly attribution can identify incomplete observations due to non-algorithmic reasons.

[0040] For variable frame rate content and encapsulation differences across coding levels, the system performs equivalent repair without altering the semantics of the original sampling points: on the one hand, irregular frame intervals are converted to... On the one hand, the system employs a strictly monotonically increasing sequence. On the other hand, it performs semantically preserved annotation and segmented submission for segments with time jumps, rollbacks, and repeated markings to prevent subsequent statistical features from being biased due to inconsistent time scales. For audio-side buffer layer switching or time base drift, the system prioritizes a sign-consistent phase continuity strategy to reduce auditory breaks. For subtitle / timecode text, line / word-level timecode coverage and legality checks are used as preconditions for submission to prevent empty segments or out-of-bounds timecodes from entering the unified timeline.

[0041] To ensure the auditability and replayability of this step, all key decisions during the access and time base normalization process are structured and written into the access log stream. These key decisions include parameter estimation, anomaly criteria, rearrangement and discard operations, tag injection, and capacity adjustment. The log stream, along with source-level time mapping parameters, serves as time-axis evidence output along with the media and feature channels, providing stable and reproducible time-axis input for subsequent feature extraction, cross-modal alignment, and dynamic modeling. It also provides necessary corroborating evidence for alarm verification and root cause analysis. For ease of engineering implementation, the access and normalization unit and streaming processing skeleton can be implemented using a multi-parallel pipeline framework. For example, GStreamer / DeepStream can be used for decapsulation and parallel encoding / decoding, while a Kafka feature bus can carry the publication / subscription of time base mapping parameters and metadata. Health indicators and backpressure signals can be exposed through observability probes to achieve decoupling from the control plane.

[0042] The final output of this step in the process is three categories and a unified time base. Aligned ordered sequences, corresponding to the three modalities A, V, and S, where A, V, and S represent audio, video, and subtitles / time-coded text, respectively; each sequence element contains a source identifier, a session identifier, and a context identifier. The output includes timestamps and durations, structured decapsulated metadata, and their quality tags; additionally, it includes estimated time-mapping parameter trajectories for each source and the complete access log stream. These outputs form the temporal basis for subsequent multimodal feature calculations within the window W(t), event anchor generation, and cross-modal alignment, enabling interpretable statistics, modeling, and alerting in subsequent steps on the same timeline.

[0043] Step 2: Basic quality and modal feature extraction

[0044] In a preferred embodiment, the system is based on the output of step one and has been aligned to a unified time base. The three types of ordered sequences (A, V, and S) are analyzed using a sliding analysis window. The system extracts and integrates basic quality elements and modal structure features using a specific method. The window length L is set to a range of seconds to tens of seconds in engineering practice, based on source frame rate, bit rate fluctuations, and service real-time requirements; for example, L=5 seconds. To avoid bias in feature estimation caused by inconsistent sampling densities from different sources, the system performs modal analysis on each modality without altering the original sampling semantics. Lightweight resampling is performed, triggering interpolation or thinning only during periods of significant sparsity or sudden density, and inference and downsampling class labels are injected into the interpolation and thinning segments respectively to ensure auditability.

[0045] For video modalities, the system forms a basic quality vector within each window, encompassing four dimensions: brightness, sharpness, temporal continuity, and structural coding. Brightness and contrast are statistically analyzed using frame-level histograms and robustly aggregated within the window to identify low-order distortions such as black levels, white levels, and color casts. Sharpness is achieved through stable estimation of edge intensity and high-frequency energy. A typical approach is to apply a discrete second-order Laplacian operator to the brightness plane and calculate the variance within the window to characterize blur trends. Temporal continuity is monitored for freezes and dropped frames using inter-frame difference energy. The system defines the short-time inter-frame difference energy as... ;in express time The brightness pixel value, Indicates that two adjacent frames are in The time interval is D, where D is the set of sampled pixels.

[0046] If within several consecutive frames If the frame consistently falls below the window adaptive threshold and is accompanied by abnormal PTS / DTS intervals, the system generates evidence of frozen / potentially dropped frames and locates the corresponding frame number. At the structural coding level, it captures structural quality degradation caused by encoding / decoding and network collaboration through indicators such as GOP regularity, reference layer switching, resolution / bitrate steps, and field / frame format consistency. These indicators are aggregated into a video fundamental quality vector within the window. It also includes time anchors for triggering evidence and a minimized image summary, such as the downsampled block average and boundary activity, to support the interpretability of subsequent anomaly localization.

[0047] For audio modalities, the system constructs a fundamental audio quality vector based on energy, spectral structure, and waveform robustness. At the energy level, equivalent sound pressure level (ESR) and root mean square (RMS) energy are used to describe loudness stability, while peak-to-average power ratio (PAPR) is calculated to identify over-compression and peak clipping. Spectral structure focuses on window statistics of the spectral centroid, effective bandwidth, and harmonic-to-noise ratio (NNR) to characterize bandwidth limitations, narrowband howling, or increased noise floor. Waveform robustness is assessed through zero-crossing rate stability, DC bias, and continuous saturation segment detection to identify anomalies and distortions in the analog-to-digital channels. To reflect perceptible interruptions in the time domain, the system tracks the proportion of silence and speech activity within a window and distinguishes between program silence, coded silence frames, and link interruption silence through endpoint consistency detection. The resulting quantitative information forms the fundamental audio quality vector. It also records the occurrence range and magnitude estimate of each type of anomaly, which are used as inputs for subsequent cross-modal correlation and root cause indication.

[0048] For subtitle / timecode-infused text modalities, the system prioritizes verifying internal timecode consistency and coverage integrity. For each window, the system calculates the time coverage of valid lines, average and maximum adjacent gaps, overlaps, and boundary violations, and performs a rapid consistency check on character set legality and rendering usability. For timecode-infused text injected from external ASR or OCR pipelines, the system uses only the temporal continuity and content duty cycle of the self-modal as the basic quality characterization without introducing cross-modal coupling judgments, avoiding premature speech / lip-sync scene inference in this step. The results of these checks constitute the subtitle basic quality vector. It also outputs evidence pointers located at the row level when there are time out-of-bounds errors, overlapping coverage, or abnormally sparse conditions.

[0049] To improve the adaptability of quality elements under different content and network conditions, this step employs robust aggregation of various raw metrics within the window and provides adaptive criteria. The system defines any metric. The window robustness center and dispersion are respectively the trimmed mean. Absolute deviation from median Therefore, a dynamic threshold is set. ;in A robustness coefficient corresponding to the metric type, used to adjust alarm sensitivity; and All values ​​are calculated within W(t) and updated as the window slides. Samples falling outside the criterion are labeled as suspicious segments, but the system only generates window-level quality events when the duration of consecutive out-of-threshold events exceeds the minimum significant duration of the metric, in order to suppress false alarms caused by occasional spikes. For segments with upstream late-drop / late-trunc labels or significant missing packet indications, the criterion will automatically switch to a reduced confidence mode and include an incomplete observation reason code in the output, thereby maintaining the interpretable boundary of the conclusion while maintaining recall.

[0050] In scenarios where reference signals or benchmark copies exist, such as comparing the master copy after transcoding, the system extracts anchor frames / segments within a window for lightweight full-reference evaluation to supplement the deficiencies of no-reference and semi-reference metrics. For video, the system selects several frames uniformly distributed along the time axis before and after structural events, calculates the error statistics of the luminance and chrominance planes, and projects them into composite metrics such as structural similarity and peak signal-to-noise ratio; for audio, the system aligns to... Based on this, the short-time signal difference energy and cross-correlation peak position offset are calculated to identify time base drift and amplitude distortion. The above full-reference results do not directly replace the no-reference conclusions, but rather serve as a weighted term for quality evidence and explicitly output their applicable premises, such as an intact reference link and effective time base alignment, to avoid misleading results when the reference is missing or mismatched.

[0051] The output of this step is a time-sliding sequence of feature records. ,in Includes robust statistics within the window, dynamic thresholds, a list of quality events, evidence pointers, and an availability bitmap. For metrics that are uncomputable due to upstream missing slices, downsampling, or interpolation, Clearly state the unavailable / inferred status along with the reason code and the scope of impact.

[0052] To support subsequent cross-modal alignment and extraction of scene / mode labels M, It also carries neutral context fields that are distinctive for the business, such as motion level, voice activity ratio, bitrate and resolution range, caption duty cycle and line density, etc., but does not make cross-modal causal judgments in this step. All outputs are published as structured messages through the feature bus and maintain a one-to-one correspondence with the time axis side evidence generated in step one, thus forming a reproducible, interpretable and anomaly-friendly quality element foundation when entering the bridging analysis and rolling prediction stage.

[0053] Step 3: Cross-modal bridging analysis and scene / mode label M extraction

[0054] The system uses the feature record sequence output in step two. Using the time axis evidence provided in step one as input, and based on a unified time reference... The above steps complete the bridging and alignment of the A / V / S three modalities, evidence fusion, and online extraction of scene / pattern labels M. The goal of this step is to integrate the various modalities within a sliding window. The local quality features formed within the data are transformed into cross-modal consistent candidate quality events, providing sparse and stable event anchors, offset estimates, and confidence quantifications for subsequent localization and root cause analysis. To avoid irreversible rewriting of upstream data, bridging analysis aligns and infers based solely on features and metadata. All inference results are output as structured bypass channels, without altering the original media and feature stream.

[0055] To establish a common reference across modalities, the system first... Internally generated sparse event anchor set The formation of anchor points follows the principle of temporal saliency consistency from different modalities: when a shot change, inter-frame difference energy abrupt change, or structural coding step occurs on the video side, and a transition in the energy envelope or spectral centroid occurs simultaneously on the audio side, and a density abrupt change in line-level timecode or large gap closure occurs on the subtitle side, the system establishes a single anchor point within the least common time neighborhood of these signals. The system records the modalities involved in alignment and the triggering evidence. For periods where only a single modality is significant while other modalities are unavailable or incompletely observed, the system retains the single-modal anchor points and explicitly labels the availability bitmap without forcing alignment. In subsequent fusion stages, such anchor points will be processed using a confidence reduction strategy to ensure that the conclusion boundaries are clear and auditable.

[0056] After establishing the anchor point in the time domain, the system quantifies the relative time delay and phase consistency between modes to identify cross-modal synchronization shifts and correct subsequent event timing. To improve robustness, the system does not directly use the original waveform, but instead... We select low-dimensional projection signal pairs with alignment indices, such as macroblock activity or inter-frame difference energy sequence on the video side and short-time energy envelope on the audio side; line duty cycle or time coverage sequence on the subtitle side and speech activity ratio, construct cross-modal cross-correlation function and estimate delay peak position.

[0057] After obtaining the cross-modal temporal relationships, the system revolves around each anchor point. Gathering from Based on the evidence fragments, a multimodal consistency score is constructed for candidate quality events to achieve the transformation from measuring anomalies to event hypotheses. To this end, the system will compare the evidence fragments within the window with... The neighborhood-related normalized outlier measure is denoted as and with As a consistency score; among which For the feature weights under the current scene / mode label M, satisfying ≥0 and .when When the dynamic threshold constructed from the trimmed mean and MAD is exceeded and the duration spans the minimum significant duration for that event type, the system... Candidate quality events are generated, along with the event category, start and end times, severity, and confidence level. Severity is typically derived from an empirical mapping of the impact of anomalies on business operations; confidence level uses a monotonic mapping function σ(⋅) to compress the strength and usability of evidence into the [0,1] interval, for example... ,in , To map the contributions of different pieces of evidence to the confidence space using nonnegative coefficients. For example, the conservatism coefficient. =0.8; This represents the proportion of unavailable or downweighted evidence at this anchor point. To avoid overfitting to a single strong piece of evidence, σ(⋅) is chosen as a saturated monotonic function with gentle compression applied to sharp peaks, making multi-source consistency take precedence over single-source bursts.

[0058] The extraction and updating of scene / pattern labels M are integral to this step and directly affect the fusion weights and judgment thresholds. The system uses... Based on neutral context fields, including motion level, speech activity ratio, caption duty cycle, bitrate and resolution range, arrival jitter and buffer usage, a lightweight online classifier capable of distinguishing content form and network running state is trained. During runtime, the current M is output as the input and output of the context statistics of the anchor neighborhood, and high-frequency jitter is avoided by time hysteresis and confidence threshold. The lightweight online classifier can be any of the following or their equivalent variants: online logistic regression, linear classifier based on stochastic gradient descent, or average perceptron.

[0059] Specifically, motion level refers to the intensity of motion in the image over time. It can be robustly statistically and normally normalized within W(t) based on block-level motion vectors, optical flow amplitude, or boundary activity of adjacent frames, with a value ranging from zero to one, where zero represents near-static motion and one represents violent motion.

[0060] The percentage of speech activity refers to the proportion of the total window duration marked as having speech activity within W(t). The value ranges from zero to one, and is statistically analyzed based on the audio track, with environmental noise and music segments differentiated and smoothed.

[0061] The subtitle duty cycle refers to the proportion of the actual visible duration of the subtitles within W(t) to the total window duration. Its value ranges from zero to one.

[0062] The bitrate and resolution range refers to the range of observed video bitrate and resolution values ​​within W(t), as well as the current level range. Bitrate is expressed in kilobits per second or megabits per second, and resolution is expressed as width x height. Step events and stability are also recorded to characterize the adaptive bitstream state and resolution level. This parameter provides evidence for identifying network limitations or changes in transcoding layers.

[0063] Arrival jitter refers to the degree of fluctuation in the interval between fragmented or packet arrivals relative to the expected interval. It is characterized by the standard deviation, percentile, or jitter index within W(t), with units of milliseconds. The data originates from the arrival timestamp of the access layer, excluding the effects of retransmissions and packet merging. This parameter reflects the instability of network latency and, together with the buffering strategy, determines the risk of rebuffering.

[0064] Buffer occupancy refers to the instantaneous depth of the playback buffer and its quantile statistics within W(t). The unit is seconds, and the value reported by the player or software development kit is preferred. By default, it can be estimated from the arrival rate and consumption rate.

[0065] Intuitively, when M represents a high-speed, sparsely spoken sports segment, the system reduces its sensitivity to energy fluctuations between video frames and increases its weighting of audio continuity; when M represents a stable studio talk show scene, the system increases its sensitivity to frozen / repeating frames and tightens the A / V synchronization threshold. Updates to M are not written back to upstream metrics, but only through weighting. The adaptive dynamic threshold is reflected in the fusion decision layer, ensuring robustness and interpretability for different business models.

[0066] Considering the common observational contradictions and evidence conflicts in real-world data transmission, this step explicitly addresses these conflicts. During periods with upstream late-drop / late-trunc or significant packet loss, the system automatically enters a confidence-reducing mode for relevant anchor points: firstly, it raises the consistency score threshold required to generate candidate events; secondly, it moderately increases β in the confidence calculation to penalize missing evidence; and thirdly, it adds a reason code for incomplete observations to all candidate events triggered by single-modal evidence and includes an availability bitmap of the affected metrics in the output. If different modalities give mutually exclusive hypotheses for the same time period, such as video freezing while audio is highly active and subtitles are continuous, the system prioritizes the interpretation that best matches the content format based on M, while retaining the suboptimal hypothesis as a backup branch. The root causes of the disagreement—near the threshold boundary, alignment uncertainty, and reference mismatch—are written into the circumstantial evidence field for further analysis in subsequent steps.

[0067] The output of this step is the bridging analysis record set. For each sliding window endpoint t, It should include at least: an event timeline and index after anchor point aggregation; the scenario / mode label M that takes effect at time t and its confidence and hysteresis status; the type, start and end, severity, confidence, trigger evidence summary and availability bitmap for each candidate quality event; the availability bitmap refers to a mask that marks the availability of each modality and its metric in a binary manner within the sliding analysis window under a unified time benchmark. 1 indicates that the unpacking and calculation in this cycle is successful and reliable, and 0 indicates that it is unavailable due to missing packets / alignment defects / timeouts, etc. It is used to quantify the proportion of missing evidence and adjust the fusion weight and judgment threshold in conjunction with the reduced confidence mode, and is written into the bridging analysis record set along with the reason code for audit traceability.

[0068] Step 4: Anomaly Localization and Root Cause Analysis

[0069] In a preferred embodiment, the system outputs the bridging analysis record set in step three. The characteristic sequence formed in step two As input, at a unified time base The system completes the confirmation, fine-grained location and hierarchical root cause determination of candidate quality events, and generates executable alarms and handling suggestions for operation and maintenance and business sides.

[0070] To achieve the transition from candidate to confirmed, the system first... The system performs continuous re-examination and time-series cleaning of the event timeline. For similar candidate events, if their start and end intervals overlap or the gap is less than the minimum significant interval threshold, they are considered different observation segments of the same potential event and merged on the time axis. When an event crosses an anchor point and the cross-modal delay estimate is stable within the confidence interval, the system determines the unified start and end points according to the strongest evidence time anchoring principle. The strongest evidence time anchoring principle means that when multiple modal candidates exist simultaneously, the time point or interval endpoint where the net evidence strength or consistency score after availability discount reaches the global maximum value is used as the unified start and end anchor points. When there is a cross-modal fixed delay and its estimate falls within the confidence interval, it is first synchronously shifted according to that delay before anchoring. To avoid misjudging instantaneous noise as a stable defect, the system continuously tracks the net evidence strength and availability bitmap within the window for each event. Only when the net strength after availability discount still continuously exceeds the shortest significant duration of the confirmation threshold is it marked as a confirmed quality event; otherwise, it remains in the observation state with an attached reason code for continued observation. The threshold is confirmed to be not a fixed constant, but is adaptively set with reference to the trimmed mean and MAD output in step two, and is tightened or loosened in real time as the scene / mode label M changes, so as to maintain a stable alarm caliber under different content formats such as high-speed motion and static broadcast.

[0071] After confirming the temporal dimension, the system refines the spatial and semantic scope. For video modalities, without altering the original sampled semantics, the system uses macroblock activity, edge energy, and error masks obtained from decoding residuals as clues to further refine the spatial and semantic scope. A normalized anomaly map is constructed from the aligned frame sequence, and morphological agglomeration is performed to integrate discrete anomaly blocks into consistent bounding boxes of regional defects, providing normalized coordinates, coverage area, and duration, thereby distinguishing between global freeze / resolution drop and local mosaic / screen tearing. For audio modalities, the system applies phase continuity constraints to anomaly segments with zero cross-rate and harmonic-to-noise ratio to locate the start, end, and bandwidth of continuous saturation instantaneous pops and narrowband howls. For subtitle / timecode-enabled text modalities, the system uses line coverage and out-of-bounds statistics as the core to determine the line segments and byte ranges where large areas of empty segments overlap and out-of-bounds and unrenderable encoded data occur. The spatial or semantic ranges of different modalities are unified under the same event identifier through the cross-modal delay and anchor point set obtained in step three, and their respective availability and uncertainty descriptions are given in the output, enabling subsequent processing strategies to be accurately targeted according to the impact area.

[0072] Root cause analysis relies on a multi-layered evidence attribution map, prioritizing the distinction between five upstream elements—source content / acquisition end transmission network link encoding / decoding parameters and transcoding strategies, time base and synchronization captions / text pipeline—and the downstream element—rendering / playback—representing responsibility boundaries. The system uses evidence fragments converged in the bridging analysis as input to construct a framework targeting the candidate root cause set { The posterior confidence estimate of} is robustly fused using a log-linear model: ;in This represents the set of evidence for the event, which includes cross-modal consistency scores, latency estimation significance, upstream late-drop / late-trunc labels, network arrival jitter statistics, GOP structural transitions, audio saturation, subtitle out-of-bounds density, etc. fr(⋅) is the evidence feature function related to the root cause, such as packet loss fingerprints, bitrate step, phase drift, and subtitle overlap rate. The non-negative fusion weights learned from historical annotations and offline playback characterize the evidence r for root causes. The support level; M is the scenario / pattern label obtained in step three, used to adaptively adjust the direction of evidence interpretation under different content and network states. This posterior distribution does not require a single definite conclusion. When the highest posterior is insufficient to form an exclusive judgment, the system retains the top K high posterior root causes as concurrent hypotheses and provides reason codes for insufficient discrimination and suggestions for additional evidence collection, such as requesting more granular network statistics from upstream or triggering short-term reference sampling, to avoid mishandling caused by insufficient evidence.

[0073] To provide quantifiable impact to the business side, the system calculates a unified impact metric and priority score for each confirmed event. The impact metric integrates two dimensions—time and space—as well as cross-modal coverage, and is defined as follows: ;in This is the normalized result of the event duration over the session quantiles. For severity, such as freeze amplitude, mute energy loss, and the normalized result of unrenderable subtitle proportions, This refers to the affected coverage, such as the percentage of screen area or the percentage of audio bandwidth that is impaired. This is a scene weighting function used to reflect the differences in the subjective impact of the same objective defects under different content formats; The weights are non-negative and their sum is one, for example... Priority scoring in Based on this, the novelty and recurrence factors of the event are superimposed to suppress the pollution of the real-time alarm channel by historical remnants, and to merge and suppress the frequency of repeated events on the same link within a short period of time, so as to ensure that the alarm channel is bounded and operable.

[0074] Once the root causes and effects are identified, the system generates an anomaly localization and root cause analysis record set. Its content covers unique event identifiers and indexes, unified start and end times and cross-modal delay corrections, spatial or semantic range descriptions, severity and impact metrics, posterior root cause distributions and their aforementioned concurrency assumptions, reason codes and availability bitmaps, correlation pointers with the time-axis evidence from step one, and engineering-oriented handling recommendations. These recommendations do not directly intervene in media content but are output to the control plane and operations ticketing system as strategic suggestions. For example, for packet loss and freezing caused by network congestion, it suggests temporarily increasing the access buffer limit and enabling stronger FEC; for excessive compression of transcoding parameters, it suggests reverting to a conservative code control template; for time base drift, it suggests restarting time synchronization or enabling tighter monotonicity constraints on the access side; and for subtitles exceeding limits, it suggests rolling back the subtitle packaging and rendering pipeline version. All recommendations include the basis for their generation and the expected scope of impact, enabling human-machine collaborative decision-making at different service levels and time periods.

[0075] To more intuitively demonstrate the core mechanisms and data flow processes in steps three and four of this solution, a schematic diagram of the core principles of the specially constructed architecture is shown below. Figure 2As shown in the figure, this diagram reveals the complete technical path from multimodal feature input to the generation of executable alerts: First, the multimodal features generated in step two are synchronously input into three parallel processing units: event anchor generation, cross-modal latency estimation, and scene label extraction. Subsequently, the outputs of these units are merged in the evidence fusion stage to generate candidate quality events. These candidate events then enter the anomaly localization stage, undergoing event confirmation and spatiotemporal localization, multi-evidence root cause determination, and impact calculation, ultimately forming executable alerts and handling suggestions for operations and maintenance. The feedback path shown by the dashed line in the diagram is particularly crucial, indicating that the results of root cause determination can be dynamically fed back to the scene label extraction process. This demonstrates the system's ability to adapt based on historical evidence, ensuring the accuracy and robustness of root cause analysis across different business scenarios.

[0076] Example 2

[0077] The design of the quality monitoring and anomaly location system for multi-source heterogeneous media data of this invention is based on the method in Embodiment 1, specifically as follows: Figure 3 The following modules are shown:

[0078] The access and time base normalization module is used to establish connections with RTSP, HLS / DASH, offline files and message buses, complete the decapsulation of protocols and containers, and output metadata and media payloads in a unified format. Without rewriting the source sampling semantics, it maps the timestamps of each source to a unified time base and performs out-of-order reordering and late pruning in a circular buffer. At the same time, it generates and outputs access log streams and time axis side evidence, and exposes health and back pressure indicators to the outside world, providing a traceable time sequence and evidence basis for subsequent processing.

[0079] The basic quality and modal feature extraction module is used to analyze multimodal data such as audio, video, and subtitles under a unified time reference. It extracts elements such as brightness, sharpness, temporal continuity, coding structure, energy spectrum structure, rendering, and timecode coverage within a sliding window, and generates dynamic thresholds and window-level quality events using robust statistics. The module explicitly marks unusable or inferred states for uncomputable metrics, along with reason codes and impact ranges, and outputs a feature record sequence. The sequence includes robust statistics, dynamic thresholds, a list of quality events, evidence pointers, and an availability bitmap, while also carrying neutral context fields that are distinguishable for business purposes for subsequent extraction and updating of scene / mode labels M.

[0080] The cross-modal bridging and scene / mode label M module is used to aggregate multimodal evidence with event anchors as a common reference, estimate cross-modal latency and phase consistency, and form candidate events and consistency scores. The module has a built-in online lightweight classifier that takes context statistics as input and outputs the current scene / mode label M under hysteresis and confidence threshold constraints, and adaptively adjusts the fusion weights and judgment thresholds accordingly. When upstream fragments are missing or evidence is lacking, it enters a confidence reduction mode, raises the consistency threshold, punishes missing evidence, and carries an availability bitmap and reason code in the output. The module outputs a bridging analysis record set, which includes the event timeline after anchor aggregation, M and its confidence and hysteresis states, candidate event elements, and evidence pointers.

[0081] The anomaly localization and root cause analysis module takes a bridging analysis record set and a feature record sequence as input, and performs timeline cleaning and event confirmation, spatial and semantic range refinement localization, hierarchical root cause determination, and generation of handling suggestions. It robustly weights and fuses evidence such as cross-modal consistency, network jitter, coding structure transitions, audio saturation, and subtitle intrusion, and retains concurrency hypotheses and additional evidence collection suggestions when discrimination is insufficient. The module calculates a unified impact metric and priority score, and finally outputs an anomaly localization and root cause analysis record set. This record set includes a unified start and end date and delay correction for events, spatial and semantic range, posterior root cause distribution, reason codes and availability bitmaps, and handling suggestions for operation and maintenance and business.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for quality monitoring abnormality localization of multi-source heterogeneous media data, characterized in that, The method comprises the following steps: Step 1: mapping original timestamps of each source to a system unified time reference, performing out-of-order rearrangement and late discard or late cut processing, and obtaining audio, video and subtitle ordered sequences aligned under the unified time reference; Step 2: under the unified time reference, extracting basic quality vectors and modality structure features of audio, video and subtitle modes respectively by using a sliding analysis window W(t), generating dynamic thresholds based on robust statistics, and forming a feature record sequence containing video basic quality vectors, audio basic quality vectors, subtitle basic quality vectors, robust statistics, dynamic thresholds, quality event lists, evidence pointers and availability bitmaps; Step 3: taking the feature record sequence as input, constructing a sparse event anchor point set under the unified time reference, estimating cross-modality time delay and phase consistency, and outputting scene / mode labels M by an online learning classifier according to neutral context fields; Gathering multi-modal evidence around the event anchor point, calculating the consistency score of the candidate quality event, and outputting a bridging analysis record set containing event timeline, scene / mode label M, candidate event type, start and end time, severity, confidence, trigger evidence summary and availability bitmap; Step 4: taking the bridging analysis record set and the feature record sequence as input, reviewing and confirming the time continuity of the candidate quality event, performing refined positioning in the spatial or semantic range, obtaining a posteriori root cause distribution based on multi-evidence fusion, calculating a unified impact measure and priority score, and outputting an abnormal positioning and root cause analysis record set containing event start and end time, spatial or semantic range description, severity, impact measure, posteriori root cause distribution, reason code, availability bitmap and operation and maintenance disposal suggestion.

2. The method of claim 1, wherein: In step 1, the process of mapping original timestamps of each source to a system unified time reference is achieved through linear mapping and small correction, which specifically includes a proportion correction coefficient, a starting phase offset, and a time variation correction quantity constrained within a safe range.

3. The method of claim 1, wherein: In step 2, the process of generating dynamic thresholds based on robust statistics includes: for any quantity within the window, calculating its trimmed mean and median absolute deviation, and setting the dynamic threshold based on the product of the sum of the two and a robust coefficient; only when the continuous over-threshold duration exceeds the minimum significant duration of the quantity, a window-level quality event is generated.

4. The method of claim 1, wherein: In step 3, the process of constructing a sparse event anchor point set includes: when there is a shot cut, interframe difference energy mutation or structure coding step on the video side, there is an energy envelope or spectral centroid transition on the audio side, and there is a line-level time code density mutation or large gap closure on the subtitle side, a single anchor point is established within the minimum common time neighborhood of the signal, and the aligned participating modes and trigger evidence are recorded; for time periods with only single-modality significance, single-modality anchor points are retained and the availability bitmap is explicitly labeled.

5. The method of claim 1, wherein: In step 3, the scene / mode label M is extracted by a lightweight online classifier, and the classifier takes the neutral context fields in the feature record sequence as input, which includes one or more of motion level, speech activity proportion, subtitle duty cycle, code rate and resolution interval, arrival jitter and buffer occupancy. The classifier takes the context statistics of the anchor neighborhood as input and outputs the current scene or mode label M at runtime.

6. The method of claim 1, wherein: In step three, the process of calculating the consistency score of the candidate quality event includes: weighting and summing the normalized anomaly metrics related to the anchor neighborhood within the window; generating the candidate quality event when the consistency score exceeds the dynamic threshold and the duration spans the minimum significant length.

7. The method of claim 1, wherein: In step four, the process of obtaining the posterior root cause distribution based on multi-evidence fusion includes: using a log-linear model to robustly fuse the evidence to calculate the posterior confidence of each candidate root cause.

8. The method of claim 1, wherein: In step four, the process of calculating the unified impact metric includes: synthesizing the event duration, severity, affected coverage, and scene weight function to obtain the impact metric by weighted summation; the priority score is superimposed on the impact metric based on the factors of event freshness and recurrence.

9. A quality monitoring anomaly localization system for multi-source heterogeneous media data, characterized in that, The positioning system is used to implement the method of any one of claims 1-8, comprising the following modules: The access and time base normalization module is used to map the original timestamps of each source to a system unified time reference, perform out-of-order rearrangement and late discard or late truncation processing, and obtain an audio, video and subtitle ordered sequence aligned under the unified time reference; The basic quality and modal feature extraction module is used to extract basic quality vectors and modal structure features under the unified time reference, respectively for audio, video and subtitle modalities, in a sliding analysis window W(t), generate dynamic thresholds based on robust statistics, and form a feature record sequence containing video basic quality vectors, audio basic quality vectors, subtitle basic quality vectors, robust statistics, dynamic thresholds, quality event list, evidence pointer and availability bitmap; The cross-modal bridging and scene / mode label M module is used to input the feature record sequence, construct a sparse event anchor set under the unified time reference, estimate the cross-modal time delay and phase consistency, and output the scene / mode label M according to the neutral context field through online learning classifier; The multi-modal evidence around the event anchor is converged, the consistency score of the candidate quality event is calculated, and the bridging analysis record set containing the event timeline, scene / mode label M, candidate event type, start and end time, severity, confidence, trigger evidence summary and availability bitmap is outputted. The abnormal positioning and root cause analysis module is used to input the bridging analysis record set and the feature record sequence, review and confirm the time continuity of the candidate quality event, perform refined positioning in spatial or semantic range, obtain the posterior root cause distribution based on multi-evidence fusion, calculate the unified impact metric and priority score, and output the abnormal positioning and root cause analysis record set containing the event start and end time, spatial or semantic range description, severity, impact metric, posterior root cause distribution, reason code, availability bitmap and operation and maintenance handling suggestion oriented.

Images