Video conference audio and video quality comprehensive detection system fused with deep learning

Abstract

The invention relates to the technical field of multimedia communication, and discloses a video conference audio and video quality comprehensive detection system fused with deep learning, which comprises a demultiplexing module, a video processing module, an audio processing module, an alignment cache module, an analysis module and a judgment module. The demultiplexing module separates audio and video code streams and analyzes timestamps; the video processing module analyzes a code stream grammar structure, motion vectors are aggregated by using a weighting coefficient based on a space division size and a residual code amount, a video feature sequence is generated, and pixel reconstruction is not needed; the audio processing module calculates a framing energy index to generate an audio feature sequence; the alignment cache module maps the feature sequence to a unified time axis; the analysis module calculates a cross-correlation function to determine a peak value and an optimal time lag; and the judgment module judges the playing quality state through activeness gating, hysteresis comparison and an asymmetric synchronous tolerance interval. According to the method, the compressed domain features and a hierarchical judgment mechanism are utilized, so that the calculation consumption is reduced, and the detection accuracy and robustness are improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia communication technology, specifically to a comprehensive audio and video quality testing system for video conferencing that integrates deep learning. Background Technology

[0002] With the widespread adoption of remote work and online collaboration, video conferencing has become a core tool for cross-regional communication, and users are increasingly demanding higher audio-visual quality during meetings. Among the many quality experience indicators, the smoothness of the video and the synchronization between audio and video are key factors determining user experience. Freezing of the video or audio-visual desynchronization significantly reduces the efficiency of information transmission and can even cause visual fatigue and discomfort among participants.

[0003] In existing video conferencing quality monitoring systems, the detection of frozen frames primarily relies on pixel-level analysis techniques. These methods typically require completely decoding the received compressed video stream into the original image sequence, then determining whether the image is frozen by calculating the sum of absolute differences (SAD) between adjacent frames or extracting dense optical flow features. However, video decoding and pixel-level calculations are computationally intensive tasks, consuming extremely high amounts of processor power and memory bandwidth. When media servers need to handle hundreds or thousands of concurrent sessions simultaneously, or when running on edge nodes and mobile terminals with limited computing power, this full-decoding detection method often fails to meet real-time requirements and may even preempt normal audio and video processing resources, affecting the stability of the meeting itself.

[0004] Furthermore, existing solutions for detecting audio-video synchronization issues also have certain limitations. Traditional synchronization detection often relies on comparing the Display Timestamp (PTS) in the transport protocol header. While this method is simple to implement, it only reflects the time relationship of data packets at the transmission layer and cannot perceive the actual playback time difference caused by differences in decoding time, rendering queue congestion, or playback device clock drift. Some solutions using signal processing techniques attempt to use the cross-correlation between audio waveforms and video temporal features for detection, but in their decision logic, they usually use symmetrical time thresholds, assuming that sound preceding the image and sound following the image have an equal impact on the user experience. This is inconsistent with the characteristics of human psychoacoustics and visual perception. The human eye is far more sensitive to "sound preceding image" than "sound lagging behind image," and using symmetrical thresholds can easily lead to inconsistencies between the detection results and the user's subjective experience. At the same time, existing cross-correlation detection algorithms often fail in low-information scenarios such as silent backgrounds or solid-color PPT presentations common in meetings due to the lack of effective signal gating and state smoothing mechanisms, resulting in drastic fluctuations in the calculated correlation coefficients and frequent false alarms. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a comprehensive audio and video quality testing system for video conferencing that integrates deep learning. This system solves the problems of high computational resource consumption for full decoding analysis, failure to adapt audio-visual synchronization detection to differences in human visual perception, and inaccurate detection due to susceptibility to network interference in existing technologies.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a comprehensive audio and video quality testing system for video conferencing that integrates deep learning, comprising a demultiplexing module, a video processing module, an audio processing module, an alignment buffer module, an analysis module, and a decision module.

[0007] The demultiplexing module is used to receive multimedia data packets, separate the multimedia data packets into video basic streams and audio basic streams, and parse the timestamps.

[0008] The video processing module parses the syntax structure of the basic video bitstream and extracts motion vectors, spatial partition size, and residual bit quantity. This module does not perform pixel reconstruction; instead, it aggregates motion vectors using weighting coefficients calculated based on the spatial partition size and residual bit quantity to generate a video feature sequence. These weighting coefficients are positively correlated with the residual bit quantity and negatively correlated with the spatial partition size, and are used to perform a weighted summation of the motion vector magnitudes of the coding units, thereby obtaining the original weighted aggregated features of the current video frame. If the current frame is identified as an intra-coded frame, the video feature values ​​of the previous frame are reused, or the average of the feature values ​​from the preceding and following predicted frames is used.

[0009] The audio processing module decodes the basic audio bitstream and segments it into frames according to the video frame, calculating the energy index to generate an audio feature sequence. When calculating the energy index, it obtains the normalized amplitude values ​​of all sampling points within the audio frame, calculates the root mean square of the normalized amplitude values, and uses a base-10 logarithm multiplied by 20 to convert the root mean square to a logarithmic energy value in decibels to maintain consistency with physical dimensions.

[0010] The alignment buffer module uses timestamps to map video and audio feature sequences to the same timeline and stores them aligned within a sliding window. This module is configured with a dual-channel synchronous circular buffer, the physical capacity of which is greater than the sum of the detection window length and the preset network jitter tolerance. The system also utilizes the sampling clock frequency of the current session to uniformly convert the raw timestamps into display timestamps in milliseconds for establishing timing mapping.

[0011] The analysis module calculates the cross-correlation function of video and audio feature sequences within a sliding window, determining the peak cross-correlation value and the optimal time lag. Specifically, it calculates the normalized cross-correlation coefficient within a preset integer search space, and the optimal time lag is the relative time offset that maximizes the normalized cross-correlation coefficient.

[0012] The decision module determines playback quality states, including video freezes and audio-visual asynchrony, based on the cross-correlation peak value and the optimal time lag. This module executes layered decision logic including activity gating detection, content consistency judgment, and time alignment judgment. The content consistency judgment uses hysteresis comparison logic to evaluate the cross-correlation peak value; the time alignment judgment determines whether the millisecond time delay value falls within a preset synchronization tolerance range. This range is asymmetrically set, allowing the maximum absolute value of audio lead to be less than the maximum absolute value of audio lag. Furthermore, the decision module performs state smoothing processing, filtering transient anomalies through a state confirmation sliding window.

[0013] This invention provides a comprehensive audio and video quality testing system for video conferencing that integrates deep learning. It has the following beneficial effects: 1. This invention adopts a video feature extraction mechanism based on the compression domain. It directly parses the bitstream syntax structure through the video processing module without performing inverse transformation and pixel reconstruction, which significantly reduces the consumption of computing resources. At the same time, by using a fusion weighting algorithm based on spatial partitioning size and residual bit quantity, it can specifically enhance the feature weights of high-frequency texture areas such as facial lip movements and body movements. While ensuring the sensitivity of dynamic image capture, it realizes the lightweight deployment of the system on edge computing nodes or resource-constrained terminal devices.

[0014] 2. This invention establishes a cross-modal alignment and cross-correlation analysis model based on absolute timestamps, which solves the problem of timing misalignment caused by inconsistent audio and video sampling frequencies and network transmission jitter. By discretizing and mapping continuous audio streams to video frame granularity through the alignment buffer module, and combining it with the asymmetric synchronization tolerance interval setting that conforms to the characteristics of human eye perception, the system can accurately quantify audio-visual synchronization deviation, effectively avoid misjudgment caused by a single symmetric threshold setting, and improve the consistency between detection results and user subjective perception.

[0015] 3. This invention sets up a layered decision logic that includes activity gating, content consistency verification and state smoothing, effectively filtering out detection interference caused by background silence, solid color images and instantaneous network fluctuations. The decision module outputs the final quality label based on the state statistics results in the sliding window, eliminating random jumps in single-frame detection results and ensuring the stability and reliability of video freezing and audio-visual desynchronization diagnosis in low signal-to-noise ratio or complex network environments. Attached Figure Description

[0016] Figure 1 This is a structural diagram of the system functional modules of the present invention; Figure 2 This is a flowchart of the audio and video quality detection method of the present invention; Figure 3 This is a temporal domain comparison diagram of the video and audio features of the present invention; Figure 4 This is the cross-correlation analysis and decision graph of the present invention.

[0017] Among them, 110 is the demultiplexing module; 120 is the video processing module; 130 is the audio processing module; 140 is the alignment buffer module; 150 is the analysis module; and 160 is the decision module. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example: Please see the appendix Figure 1 This invention provides a comprehensive audio and video quality testing system for video conferencing that integrates deep learning. This system can be flexibly deployed in media forwarding servers (SFU / MCU), edge computing nodes, or user terminal devices in a video conferencing architecture for real-time monitoring of audio and video quality.

[0020] In this embodiment, the video conferencing quality detection system includes a demultiplexing module 110. The demultiplexing module 110 is connected to a network communication interface and is used to receive multimedia data packets from the video conferencing. The demultiplexing module 110 separates the multimedia data packets into a video basic stream and an audio basic stream according to a transmission protocol (such as RTP / Real-Time Messaging Protocol, RTMP). The video basic stream contains compressed video data frames arranged in chronological order, and the audio basic stream contains compressed audio data frames arranged in chronological order. Furthermore, the demultiplexing module 110 is also responsible for parsing the presentation timestamp (PTS) information indicating the playback order from the header information of the multimedia data packets.

[0021] The system further includes a video processing module 120 for receiving the basic video bitstream. Unlike traditional pixel-domain detection, which requires inverse transformation and pixel reconstruction, the video processing module 120 does not require pixel-level reconstruction. Instead, it extracts motion vectors, spatial partitioning dimensions, and residual code amounts for coding units within video frames by parsing the syntax hierarchy of the basic video bitstream (performing entropy decoding). The video processing module 120 calculates weighting coefficients for corresponding coding units based on the spatial partitioning dimensions and residual code amounts, and uses these weighting coefficients to perform weighted aggregation of motion vectors, thereby generating a video feature sequence characterizing the dynamic range of the video frame.

[0022] The system also includes an audio processing module 130, which decodes the basic audio stream to obtain digital audio signals and performs frame-by-frame processing on the digital audio signals according to the duration corresponding to the video frames. The audio processing module 130 calculates the energy index of each frame of audio signal and generates an audio feature sequence characterizing the level of speech activity.

[0023] To enable cross-modal analysis, the system includes an alignment cache module 140, which is connected to both the video processing module 120 and the audio processing module 130. The alignment cache module 140 maintains a sliding window storage area with a preset time length, using unified timestamp information to map video and audio feature sequences to the same timeline for aligned storage.

[0024] The analysis module 150 is connected to the alignment buffer module 140 and is used to read the video feature sequence and audio feature sequence within the sliding window. The analysis module 150 calculates the cross-correlation function of the video feature sequence and audio feature sequence under different time delays, and obtains the cross-correlation peak and the corresponding optimal time delay parameter through a search algorithm.

[0025] Finally, the decision module 160 is connected to the analysis module 150. Based on the statistical mean of the video feature sequence, the statistical mean of the audio feature sequence, and the calculation results of the cross-correlation function, the decision module 160 determines the current video conference playback quality status. Playback quality status includes video freeze status and audio-visual desynchronization status. The decision module 160 ultimately outputs the detection results, including status indicators and synchronization offset parameters.

[0026] The functional modules of a video conferencing quality testing system can be specified as hardware components in electronic devices or software instructions executed by a processor.

[0027] Electronic devices include a processor, which serves as the core of computation, used to parse and execute computer program instructions. The processor can be implemented as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), or a Field-Programmable Gate Array (FPGA). The processor performs logical operations to achieve core functions such as stream demultiplexing, syntax element parsing, feature-weighted calculation, cross-correlation analysis, and state determination.

[0028] Electronic devices include memory, which is communicatively connected to a processor, for storing computer program instructions and temporary data during processing. The memory also acts as a data buffer, caching the video base stream, audio base stream, extracted syntax element data, and aligned feature sequences.

[0029] The electronic device includes a communication interface for establishing a communication connection with an external network, acquiring real-time video conferencing stream data, and sending the detection results to a monitoring terminal or log server. A bus connects these components, enabling data transmission and control signal interaction.

[0030] Please see the appendix Figure 1 The detection method provided by this invention can monitor the image freeze and audio-visual synchronization status in real time during video conferencing based on the compression domain information of video encoding and the temporal energy information of audio signals.

[0031] The method first performs step S100, receiving the video conference transport stream, demultiplexing the transport stream into independent video basic streams and audio basic streams, and parsing the timestamp information used to indicate the playback order.

[0032] Step S100 specifically includes: capturing Real-time Transport Protocol (RTP) or Real-Time Messaging Protocol (RTMP) data packets via the network interface. The protocol parser removes the transport layer encapsulation header, identifies the payload type, and separates the Video Elementary Stream and Audio Elementary Stream. It should be noted that the specific protocol parsing details for transport stream demultiplexing are well-known technologies in the field of communications and will not be elaborated here. Simultaneously, the system extracts the Presentation Time Stamp (PTS) from the data packet header to establish a unified system time base for timing alignment of video and audio frames in subsequent steps.

[0033] Then, step S200 is executed to parse the syntax elements in the basic video bitstream, extract the motion vector, spatial partition size and residual code amount of the coding unit, and perform weighted calculation on the motion vector based on the spatial partition size and residual code amount to generate a video feature sequence.

[0034] In this embodiment, step S200 is designed based on the following physical principle: In video conferencing scenarios, the speaker's lip movements and facial expressions typically correspond to subtle changes in image texture and non-rigid motion. In video coding, these regions are usually divided into smaller prediction units (small size) by the encoder, and a large amount of error data (high residual) remains after prediction. Conversely, background regions are typically divided into larger sizes with smaller residuals. Therefore, combining size and residual information can effectively extract speech-related facial motion features from the entire image using weighted extraction.

[0035] Step S200 further includes the following sub-steps: S201: The current video frame data is read using the bitstream parser to identify the Network Extraction Unit (NALU) type. If the current frame is a prediction frame (P-frame or B-frame) containing motion information, its slice header information is parsed, and each coding unit within the frame is traversed. Coding units correspond to macroblocks or Coding Tree Units (CTUs) in different video coding standards. If the current frame is an intra-coded frame (I-frame / IDR frame), since it does not contain motion vector information, the system executes a feature preservation strategy, directly reusing the feature values ​​of the previous frame or interpolating them as the average of the preceding and following P-frames to avoid non-physical zeroing fluctuations in the feature sequence.

[0036] S202, for the current video frame (non-I-frame) in the... Each coding unit directly extracts three feature parameters from the code stream syntax: motion vector. Space division dimensions and residual code .

[0037] Among them, motion vector Includes horizontal components and vertical components Calculate the motion intensity of the coding unit. The calculation formula is as follows: ; in, The physical displacement modulus of the encoding unit is characterized.

[0038] Space division dimensions This represents the pixel area covered by the encoding unit.

[0039] Residual code The length of the binary data used to record the predictive residual transform coefficients in this coding unit is obtained by calculating the number of bits occupied by the corresponding syntax element in the bitstream.

[0040] S203, Constructing Multi-Dimensional Fusion Weights This weight combines spatial partitioning granularity with residual information density to characterize the visual prominence of the image region corresponding to the coding unit. The calculation formula is as follows: ; in, In fact, it implies the concept of residual bit density, which reflects the texture complexity and the amount of prediction error information per unit pixel area; and This is a preset normalization balance coefficient used to adjust the proportion of geometric dimensional features and residual features in the weight calculation. In this embodiment, The recommended value range is 16 to 64. The recommended value range is 0.5 to 2.0.

[0041] S204, calculate the... The original weighted aggregate features of each video frame This generates a video feature sequence. The calculation formula is as follows: ; in, For the first The total number of coding units contained in a video frame. To prevent extremely small constants with a denominator of zero. Reflects the first The instantaneous dynamic range of a visible region (such as a face or body movement area) in a video frame.

[0042] The method includes step S300, which involves decoding and framing the basic audio bitstream, calculating the energy index of each frame, and generating an audio feature sequence.

[0043] Step S300 specifically includes: decoding the basic audio bitstream to obtain digital audio sampling data in PCM format. Based on the video frame rate parsed in step S100, the audio sampling data is divided into audio frames with a duration equal to the display duration of the video frames. For the... The corresponding video frame Each audio frame contains [number] audio frames. A sequence of sampling points Calculate the root mean square energy of this audio frame. The calculation formula is as follows: ; in, For the first The normalized amplitude values ​​at each sampling point (range [-1.0, 1.0]). The root mean square is used here instead of the simple sum of squares to maintain dimensional consistency with the "20log" formula in subsequent decibel conversions.

[0044] The method includes step S400, which uses timestamp information to align video feature sequences and audio feature sequences, calculates the cross-correlation function of the two within a sliding window, and determines the cross-correlation peak and the corresponding time lag.

[0045] Step S400 specifically includes: establishing a length of The sliding window reads the time-aligned final video feature sequences from the buffer. and the final audio feature sequence Calculate the time lag of two sequences at different times. Normalized cross-correlation function The calculation formula is as follows: ; in, The relative time offset within the detection range (usually in frames). and These are the video feature values ​​and audio feature values ​​at the corresponding moments within the sliding window, respectively. and These are the mean values ​​of the video feature sequence and the audio feature sequence within the window, respectively. and These are the standard deviations of the video feature sequence and the audio feature sequence within the window, respectively. This represents the summation index or the offset of a historical data point within the sliding window.

[0046] The normalized cross-correlation function under different hysteresis was calculated. Then, the system searches within the preset integer search space. Find the peak value within the time frame and determine the optimal time lag. and the maximum cross-correlation number The calculation formula is as follows: ; ; in, The detection time window range is limited to the maximum search radius (e.g., the number of frames corresponding to 500ms); The maximum cross-correlation coefficient represents the degree of similarity between the video and audio waveforms under optimal alignment, and is used to determine whether the content matches. The optimal time lag represents the relative displacement that maximizes audiovisual similarity, which is the synchronization deviation value ultimately measured by the system. Indicates in set Within the range, find the function The maximum possible value; Indicates in set Within the range, make the function The variable that gets the maximum value The value of .

[0047] Step S500 employs a hierarchical decision mechanism to transform the correlation statistics into discrete states at the business layer. This decision logic is based on the following physical assumptions: First, the signal must be statistically significant (non-silent); second, the waveform shapes must be similar (homogeneity); and finally, the waveform's offset on the time axis must be within the perception threshold (synchronicity).

[0048] Specifically, step S500 includes the following sub-steps: Step S510: Perform activity gating detection.

[0049] Because cross-correlation algorithms are prone to producing random, high-correlation artifacts when processing low signal-to-noise ratio signals (such as silent or solid-color backgrounds), pre-gating is necessary. The system checks the current audio energy characteristics. Check if it exceeds the valid speech start threshold (e.g., 0.15), and simultaneously check video features. Is it greater than the minimum motion threshold (e.g., 0.08)? If any condition is not met, or if the previous step has been marked as "VIDEO_FREEZE", the system determines that it is currently in an invalid detection period, outputs the status as "IDLE", and maintains the synchronization status of the previous moment or outputs unknown.

[0050] Step S520: Perform a content consistency judgment.

[0051] After passing through the gating point, the system first determines whether the action in the video and the sound in the audio belong to the same source. This step is achieved by evaluating the magnitude of the correlation coefficient, which is independent of time delay. The system uses a hysteresis comparator (SchmittTrigger) logic. Make a decision to prevent the state from bouncing around the threshold: If the current state is "asynchronous", only when A state flip is triggered only when the value is set to 0.65, indicating a content match. If the current state is "synchronized", only when A content mismatch is only considered when the value is set to 0.35. If... If the conditions are below the above, it indicates a significant difference in the shape of the video waveform envelope compared to the audio waveform envelope (e.g., a person nodding in the video while speaking in the audio). In this case, regardless of time delay... Regardless of the value, the system interprets it as "CONTENT_MISMATCH" (content mismatch).

[0052] Step S530: Perform time alignment determination.

[0053] The system only evaluates after the content consistency determination in step S520 has been passed. The system will use frames as units. Convert to time delay : ; in, This represents the average interval between video frames (in seconds).

[0054] In the cross-correlation definition of this embodiment, the physical meaning of the time delay direction is defined as follows: like This means video sequence With backward-shifted audio sequences Matching means that the actual audio signal arrives later than the video signal, i.e., audio lag. like This means that the video sequence matches the forward-shifted audio sequence, indicating that the actual audio signal arrives earlier than the video signal, i.e., audio lead.

[0055] System settings for synchronization tolerance range Based on reference standards and the subjective tolerance of real-time conferencing, the human eye has low sensitivity to "sound lagging behind the picture" but is very sensitive to "sound preceding the picture." Therefore, this embodiment uses an asymmetric threshold setting: (Maximum allowed audio advance, i.e.) The negative absolute value does not exceed 60ms). =+150ms (maximum allowable audio lag, i.e.) Positive values ​​do not exceed 150ms.

[0056] The logic behind the judgment is as follows: like The result is determined to be "SYNC_OK" (audio-visual synchronization). like or The system is identified as "SYNC_DELAY" (audio-visual asynchrony) and the specific delay is recorded. .

[0057] Step S540: State smoothing and final output.

[0058] Because network jitter can cause instantaneous deviations in the correlation calculation of a single frame, directly outputting the instantaneous state will lead to false alarms. The system introduces a length of... (For example The system uses a sliding window to confirm the status of frames. It executes either a "majority vote" or "continuous confirmation" logic: the system only updates the global status flag when the percentage of identical states (such as "SYNC_DELAY") within the sliding window exceeds 75%. For example, the system only throws an "audio-visual desynchronization alarm" event to the upper-layer application if delays in multiple consecutive frames are detected, thereby filtering out occasional detection fluctuations caused by keyframe (I-frame) refreshes or momentary packet loss, ensuring the stability of the diagnostic results.

[0059] During step S100, the video conferencing quality detection system accesses the real-time media transmission link of the conference via a network communication interface. Since video and audio data are usually transmitted mixed in the network with different sampling frequencies and encapsulation formats, directly comparing their data packet sequences cannot reflect the true synchronization relationship. Therefore, the system first needs to perform stream separation and time base unification operations.

[0060] In practical implementation, the video conferencing transport stream received by the system is encapsulated based on the Real-time Transport Protocol (RTP) or the Real-Time Messaging Protocol (RTMP). The demultiplexing module 110 reads the header information of the received data packets and identifies the type of payload according to the transport protocol specifications. The system obtains the payload type values ​​(PayloadTypeID) and their respective sampling clock frequencies for the video and audio streams in the current session by parsing the Session Description Protocol (SDP) or out-of-band signaling information. Based on this configuration information, the demultiplexing module 110 checks the payload type field or the synchronization source identifier (SSRC) in the data packet header, logically separating data packets belonging to the video channel from those belonging to the audio channel.

[0061] During demultiplexing, the system synchronously performs timestamp parsing and conversion operations. The raw timestamps in the RTP protocol are monotonically increasing counts based on a specific clock frequency. Since the clock frequencies of different media streams are typically inconsistent (e.g., video streams often use 90kHz, while audio streams often use 48kHz), the raw values ​​cannot be directly used for cross-modal synchronization calculations. Therefore, the system uniformly maps the raw timestamps of video and audio to the system's absolute display time (PresentationTimeStamp, PTS) in milliseconds (ms).

[0062] Let the clock frequency of the stream to which the currently processed media data packet belongs be . The original timestamp recorded in the protocol header of this data packet is The offset of the base timestamp recorded at the initial moment of session establishment or the moment of reset is . The system calculates the display timestamp corresponding to the data packet based on the following formula. : ; in, The value is determined based on the configuration parameters obtained during the demultiplexing phase; This is used to eliminate the influence of random initial values ​​and ensure that the timeline starts counting from zero or a fixed offset point; the coefficient 1000 is used to convert seconds to milliseconds. Through the above conversion, each frame of data in both the video and audio base streams is given a time stamp with a uniform dimension.

[0063] Furthermore, to address packet loss or out-of-order delivery issues in network transmission, the demultiplexing module uses the sequence number in the packet header to perform integrity checks on arriving data packets. The system maintains a jitter buffer of preset size (e.g., 200ms to 500ms duration) to rearrange out-of-order data packets according to their sequence numbers and remove packets with duplicate sequence numbers, ensuring that the video and audio base streams output to subsequent processing modules are monotonically increasing in timing. If a discontinuity in sequence numbers is detected, the module marks the discontinuity in the output stream to trigger frame interpolation or interpolation logic in subsequent feature extraction steps.

[0064] After completing the logical separation and timestamp standardization of the basic video and audio streams, the system establishes a unified timing alignment and buffering mechanism. Since video signals are discrete frame sequences in time, while audio signals are continuous waveform samples, and their arrival times are inconsistent due to jitter during network transmission, this invention constructs a ring-shaped buffer structure based on an absolute time axis to discretize the continuous audio stream and align it to the temporal granularity of the video frames.

[0065] Step S110: Construct a dual-channel synchronous circular buffer. The system allocates a video circular buffer in memory to store video data and an audio circular buffer to store audio data. To meet the requirements of real-time sliding window analysis and accommodate network jitter, the physical capacity of the buffers is determined. Set to be greater than the detection window length With maximum network jitter tolerance sum.

[0066] ; in, The time span used for cross-correlation analysis is typically set to 2000ms to 5000ms to cover at least one complete voice interaction phrase; The buffer margin reserved to resist network transmission jitter is set based on the current network round-trip time (RTT) statistics, typically 1.5 to 2 times the RTT. A circular buffer structure is adopted, avoiding frequent memory allocation and deallocation operations through the circular movement of read and write pointers.

[0067] Step S120: Perform timestamp-based video frame slot mapping. The system treats each frame in the video base stream as a reference point on the timeline. For each arriving video frame, the system reads its normalized display timestamp. This is mapped to the corresponding logical location in the video circular buffer. The system calculates the time difference between the current frame and the previous frame. ,like If the frame interval exceeds a preset threshold (e.g., 1.5 times the standard frame interval), it is determined that a frame has been dropped. The system marks the corresponding empty position in the buffer as "invalid" and performs zero-padding or linear interpolation on the feature value at that position in subsequent calculations to prevent feature sequence breakage.

[0068] Step S130: Perform temporal segmentation and alignment of the audio data. To achieve cross-modal analysis, the system needs to segment the continuous audio stream into audio segments in the temporal domain that match the display duration of the video frames. Let the display timestamp of the current video frame be... The display timestamp of the next video frame is If network latency causes... Not yet available; the system uses the average frame interval of the current video stream. To make an estimate, that is, to let .

[0069] Assume the audio stream has a sampling rate of (Unit: Hz). The system calculates the starting global index of the audio sampling points within the corresponding time period of this video frame. and end global index : ; ; The coefficient 1000 is used to convert timestamps in milliseconds to seconds. This is to map the global index to a physical capacity of... The audio circular buffer is used by the system to map the above indexes to physical address ranges through modulo operations. This ensures that the subsequently extracted audio energy features maintain a strict one-to-one correspondence with the video compression domain features in the time dimension.

[0070] Step S140: Maintain the update logic for the sliding window. The system maintains a write pointer pointing to the latest playback time. and a read pointer to the start time of the analysis window As new data packets continue to arrive, Move forward. When and The time difference between them exceeds the set window length. At that time, the system synchronously moved forward. This covers the old data area.

[0071] During step S200, the video processing module 120 does not perform complete pixel reconstruction, but only performs entropy decoding and syntax parsing. The process of obtaining compressed domain syntax elements in step S200 specifically includes the following sub-steps: Step S210: Bitstream Structure Parsing and Frame Type Filtering. The system sequentially reads the Network Abstraction Layer Units (NALUs) in the basic video bitstream. The parser reads the NALU header bytes to identify the slice units containing video image data. The parser reads the slice header to identify the encoding type of the current slice. If the current slice is an I-slice (IntraSlice), since it does not contain motion vectors, the system marks the frame as a "reset frame". To maintain the smoothness of the feature sequence, the system reuses the final features of the previous valid P-frame or B-frame. It can be used as an alternative value for the current frame, or set as a preset baseline activity value.

[0072] Step S220: Traversal and size extraction of the prediction unit. The parser performs entropy decoding, traversing each basic coding tree unit (CTU) within the current slice in scanning order. The parser recursively locates the smallest prediction unit (PU). In this embodiment, each prediction unit is defined as a basic unit for feature computation. Regarding the first Each prediction unit determines the pixel width it covers. and height Calculate the spatial division dimensions of this unit. .in, The smaller the value, the more fragmented the region is during encoding, which usually corresponds to high-frequency textures or edge regions of the image.

[0073] Step S230, analysis and reconstruction of motion vectors (MV). For the first MV in the P-slice or B-slice... Each prediction unit (MTU) extracts the motion vector difference (MVD) and reference frame index from the bitstream, and reconstructs the absolute motion vector by combining them with the predicted motion vectors (MVP). Calculate the motion intensity of this unit. If the current prediction unit is in IntraMode, then let If the current prediction unit is in Skip Mode, then the motion vector of the reference block is directly copied according to the standard; if copying is not possible, then... .

[0074] Step S240: Statistics of Residual Bits. After parsing the motion prediction information, the bitstream stores the transform coefficients of the prediction residuals. The parser records the data entering the next step... The position of the code stream read pointer before the residual data area of ​​each prediction unit And the position of the bitstream read pointer when parsing is complete. Calculate the first... Residual code quantity per coding unit .in, This indicates the number of physical bits consumed by the prediction unit in the compressed bitstream to describe texture details and prediction errors.

[0075] The spatial partitioning size of each prediction unit was obtained according to step S200. and residual code Then, the system executes steps S250 to S270.

[0076] Step S250: Calculate the residual code density. The system defines the residual code density by calculating the average residual bits per unit pixel area. This parameter can distinguish between noise in a large, smooth background and key motion details in a small area.

[0077] Step S260: Construct a fusion weighted model based on geometric dimensions and residuals. The system calculates the weights. .in, These are the geometric weighting coefficients. This represents the residual weighting coefficient.

[0078] Step S270: Perform weighted aggregation calculation to generate frame-level features. The system iterates through the current video frames. All valid prediction units within the range, using the calculated weights Regarding exercise intensity Perform a weighted average to generate the original weighted aggregated features of the video frame. .

[0079] Step S280: Temporal smoothing of feature values. Due to the periodic changes in I / P / B frames within the video coding structure, the original feature values ​​fluctuate even under uniform motion. The system employs an exponentially weighted moving average (EWMA) algorithm to perform low-pass filtering on the sequence. ; in, This is a smoothing factor, and a value range of 0.6 to 0.8 is recommended. This is the final video feature sequence input to the cross-correlation analysis module 150.

[0080] During step S300, the system decodes the basic audio bitstream, standardizes its format, and performs frame segmentation.

[0081] Step S310, Audio Decoding and Format Normalization. The system reads the audio payload data from the audio circular buffer and decodes it to reconstruct the PCM format digital sampled signal. To eliminate hardware differences, the system sets a unified target audio format: sampling frequency. Hz, mono. If it is multi-channel, it is mixed into mono. Normalize the PCM data to [ ]. The range is 1.0, 1.0], and DC removal processing is performed.

[0082] Step S320: Frame-by-frame extraction of audio data. Based on the time mapping relationship established in step S100, audio data aligned with the current video frame time is extracted from the audio circular buffer, and the current audio frame vector is constructed. .

[0083] Step S330: Calculate the root mean square energy of the short-time frame. The calculation formula is as follows: .

[0084] Step S340: Perform logarithmic domain transformation and decibel mapping. Convert the root mean square energy value to a logarithmic energy value in decibels. .

[0085] Step S350: Perform dynamic noise thresholding. The system maintains a background noise estimate. And set a dynamic noise threshold. .in, For signal-to-noise ratio tolerance, it is typically set to 10dB to 15dB. If Then the energy value will be forcibly set to the lower limit of the silent reference. (e.g., -90dB) to eliminate background noise interference.

[0086] Step S360: Feature Normalization and Temporal Smoothing. To make audio and video features comparable in numerical space, the system maps the effective energy features to the [0,1] interval. Furthermore, to match the temporal rate of change of video features, the system uses an exponentially weighted moving average algorithm to generate the final audio feature sequence. .

[0087] After obtaining the video compression domain feature sequences respectively and audio temporal energy feature sequence Then, the system proceeds to step S400, where the normalized cross-correlation function (NCCF) is used to quantify the similarity between the two in the time-series waveforms.

[0088] Step S410: Define the time delay search range and analysis window. The system sets the maximum search radius. (Corresponding to the maximum tolerable time delay, such as the number of frames corresponding to 500ms), thereby determining the integer search space. At the same time, the length of the core analysis window is set to... frame.

[0089] Step S420: Construct the data sequence required for analysis. Analysis module 150 reads a length of [length missing] from the alignment cache module. The final video feature sequence and the final audio feature sequence To support sliding calculations within the search space, the system needs to ensure that the audio data read covers an extended range including the hysteresis offset.

[0090] Step S430: Calculate the normalized cross-correlation function using a sliding motion. The system traverses the search space. Each integer time delay within For each The system first calculates the mean of the video feature sequence within the current sliding window. Standard deviation and the mean of the audio feature sequence. Standard deviation Then, the normalized cross-correlation function is calculated according to the following formula. : ; in, The summation index within the sliding window. and These are the feature values ​​at the corresponding time points within the window.

[0091] Step S440: Extract peak correlation and synchronization status. The system calculates all corresponding Then, a peak search operation is performed to determine the maximum number of cross-correspondences. and the corresponding optimal time lag : ; ; Used to determine whether the content matches in subsequent judgments. This indicates the degree of synchronization deviation of the current audiovisual signal.

[0092] At the same time, the system executes steps S450 to S460, namely the video freeze detection logic.

[0093] Step S450: Calculate short-time statistics for video features. The system defines the observation window. Calculate the original feature sequence of the video mean and standard deviation .

[0094] Step S460: Perform dual threshold decision and state accumulation. The system sets the amplitude silence threshold. (e.g., 0.05) and noise threshold (e.g., 1×10) 41×10 4). If and If the current frame is deemed a suspected frozen frame, then the frame is considered a frozen frame if the number of suspected frozen frames exceeds a certain time threshold. When the system marks the status as "VIDEO_FREEZE" for 3 consecutive seconds (e.g.), the status is marked as "VIDEO_FREEZE".

[0095] To aid in understanding the present invention, the following detailed explanation is provided using the specific application scenario of "remote court hearings in high-latency network environments." Assume a court is conducting a cross-border remote video hearing. Due to congestion in the cross-border network link, audio data packets suffer more severe jitter and congestion during transmission than video data packets, resulting in a significant delay in sound transmission to the judge.

[0096] Please see the appendix Figure 3 , Figure 3 This paper demonstrates the temporal comparison of video and audio features of the present invention. When the system receives an audiovisual clip of a judge asking "Please state the defendant's name," the video processing module extracts the features of that frame. Due to the detailed division of the coding units in the mouth region and the large residual, the video feature sequence generated by the system is represented by a solid black line in the figure. This solid line shows a clear peak of motion at the horizontal axis of 1.0 second, with a normalized feature amplitude of 0.85, reflecting the facial muscle movements of the judge during the speech. Simultaneously, the audio feature sequence generated by the audio processing module is represented by a dashed black line in the figure for distinction. Since the actual sound signal arrives later than the video, the peak of the audio feature sequence appears at the system absolute time of 1.25 seconds, with a normalized amplitude of 0.9. (See Appendix...) Figure 3 In the diagram, there is a clear horizontal distance between the video peak and the audio peak. The chart visually indicates the time misalignment between the two peaks by a double-arrow line segment connecting them. The misalignment duration is calculated to be 250 milliseconds.

[0097] Please see the appendix Figure 4 , Figure 4 This demonstrates the cross-correlation analysis and decision logic of the present invention. The analysis module 150 performs cross-correlation calculations within a 3-second sliding window, and the resulting cross-correlation function curve is represented as a fluctuating black solid line in the graph. The system searches for the maximum value in the curve and finds that when a lag of +250 milliseconds is introduced, the overlap of the feature peaks of the video and audio is highest. This point is marked as a black solid circle (detected peak value) in the graph. The ordinate (maximum cross-correlation coefficient) corresponding to this peak value is as high as 0.92, satisfying the content consistency judgment. (Appendix) Figure 4 The system also sets a synchronization tolerance range, visually represented by a light gray background area. The left boundary of this area is the upper limit of allowed lead (-60 milliseconds), and the right boundary is the upper limit of allowed lag (+150 milliseconds). In this detection, the solid black dot representing the current state is located at +250 milliseconds on the horizontal coordinate, clearly falling outside the right side of the light gray synchronization tolerance range. This intuitively shows that the calculated delay (+250 milliseconds) exceeds the maximum threshold of audio lag allowed by the system (+150 milliseconds). Based on this, the decision module 160 ultimately outputs the state as "audio-visual desynchronization (SYNC_DELAY)" and records the specific delay as +250 milliseconds.

Claims

1. A comprehensive audio and video quality testing system for video conferencing integrating deep learning, characterized in that, include: The demultiplexing module (110) is used to receive multimedia data packets, separate the multimedia data packets into video basic streams and audio basic streams, and parse the timestamps. The video processing module (120) is used to parse the syntax structure of the basic video bitstream without pixel reconstruction, extract motion vectors, spatial partition size and residual code amount, aggregate the motion vectors using weighting coefficients calculated based on the spatial partition size and residual code amount, and generate a video feature sequence. The audio processing module (130) is used to decode the basic audio bitstream and divide it into frames according to video frames, calculate energy index and generate audio feature sequence; The alignment cache module (140) is used to map the video feature sequence and the audio feature sequence to the same timeline using the timestamp and store them in alignment within a sliding window; Analysis module (150) is used to calculate the cross-correlation function of the video feature sequence and the audio feature sequence within the sliding window, and to determine the cross-correlation peak and the optimal time lag. The decision module (160) is used to determine the playback quality status, including video freezing and audio-visual asynchrony, based on the cross-correlation peak value and the optimal time lag.

2. The video conferencing audio and video quality comprehensive testing system integrating deep learning according to claim 1, characterized in that, When generating a video feature sequence, the video processing module (120) calculates the weighting coefficients for each coding unit in the current video frame; The values ​​of the weighting coefficients are positively correlated with the residual code amount and negatively correlated with the spatial partitioning size; The video processing module (120) uses the weighting coefficients to perform a weighted summation of the motion vector magnitudes of the coding unit to obtain the original weighted aggregate features of the current video frame.

3. The video conferencing audio and video quality comprehensive testing system integrating deep learning according to claim 2, characterized in that, When processing the basic video stream, if the video processing module (120) identifies the current frame as an intra-coded frame that does not contain motion vectors, it reuses the video feature value of the previous frame or uses the average of the feature values ​​of the predicted frames before and after as the feature value of the current frame.

4. The video conferencing audio and video quality comprehensive testing system integrating deep learning according to claim 1, characterized in that, When calculating the energy index, the audio processing module (130) obtains the normalized amplitude value of all sampling points in the audio frame, calculates the root mean square of the normalized amplitude value, and converts the root mean square into a logarithmic energy value in decibels. The conversion process uses a base-10 logarithm of 20 to maintain consistency with the dimensions of the physical field quantities.

5. A comprehensive audio and video quality testing system for video conferencing integrating deep learning as described in claim 1, characterized in that, The alignment cache module (140) is configured with a dual-channel synchronous ring buffer, and the physical capacity of the ring buffer is set to be greater than the sum of the detection window length and the preset network jitter tolerance. The integrated detection system also utilizes the sampling clock frequency of the current session to uniformly convert the original timestamps of the video and audio base streams into display timestamps in milliseconds, which are used to establish a timing mapping in the circular buffer.

6. The video conferencing audio and video quality comprehensive testing system integrating deep learning according to claim 1, characterized in that, When calculating the cross-correlation function, the analysis module (150) calculates the normalized cross-correlation coefficient between the video feature sequence and the audio feature sequence within a preset integer search space. The optimal time lag is the relative time offset that maximizes the normalized cross-correlation coefficient, and the peak cross-correlation value is the maximum value of the normalized cross-correlation coefficient within the search space.

7. The video conferencing audio and video quality comprehensive testing system integrating deep learning according to claim 1, characterized in that, The decision module (160) executes hierarchical decision logic including activity gating detection, content consistency decision and time alignment decision; The activity gating detection is used to check whether the amplitudes of the current audio feature sequence and video feature sequence are higher than the preset effective speech threshold and minimum motion threshold, respectively, before performing cross-correlation analysis.

8. A comprehensive audio and video quality testing system for video conferencing integrating deep learning according to claim 7, characterized in that, The specific method by which the decision module (160) executes the content consistency decision is as follows: The cross-correlation peak value is evaluated using hysteresis comparison logic; When the current state is asynchronous, content matching is determined only when the cross-correlation peak value is greater than the set high threshold. When the current state is synchronized, content mismatch is determined only when the peak value of the cross-correlation is less than the set low threshold.

9. A comprehensive audio and video quality testing system for video conferencing integrating deep learning according to claim 7, characterized in that, When performing time alignment determination, the decision module (160) converts the optimal time lag into a time delay millisecond value and determines whether the time delay millisecond value falls within a preset synchronization tolerance range. The synchronization tolerance interval is set asymmetrically, and the maximum absolute value of audio leading allowed by the synchronization tolerance interval is less than the maximum absolute value of audio lagging allowed.

10. A comprehensive audio and video quality testing system for video conferencing integrating deep learning according to claim 1, characterized in that, The decision module (160) is also used to perform state smoothing processing and maintain a state confirmation sliding window. Only when the proportion of frames determined to be in the same abnormal state in the sliding window exceeds a preset proportion will the final playback quality state detection result be output.

Images