Real-time audio and video end-to-end delay statistical method based on bidirectional time synchronization

Abstract

The invention discloses a real-time audio and video end-to-end delay statistical method based on two-way time synchronization, and particularly relates to the technical field of real-time audio and video communication and network performance measurement, which comprises the following steps: S1, time synchronization: a plug flow client and a pull flow client respectively calculate clock skew with an SFU server through two-way message exchange; the method comprises the following steps: S1, a flow pushing end converts an acquisition timestamp and deviation into SFU reference time, and the SFU reference time and stage time consumption metadata are embedded into SEI information to be sent along with a video stream, S3, an SFU server forwards the video stream containing the SEI information, and S4, delay calculation and decomposition: a flow pulling end analyzes the SEI information, calculates total end-to-end delay in combination with the deviation of the flow pulling end, and decomposes the time consumption of each stage. Through high-precision time synchronization and time data losslessly carried by SEI information, non-intrusive, automatic, full-link-covered and decomposable end-to-end delay accurate statistics is realized, and the bottleneck problems that a traditional method is high in invasiveness, low in precision and incapable of positioning are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of real-time audio and video communication and network performance measurement technology, and more specifically, to a real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization. Background Technology

[0002] Real-time audio and video communication technology, as the core supporting modern internet applications such as video conferencing, online education, interactive live streaming, and cloud gaming, relies heavily on end-to-end latency for user experience. High latency can lead to problems such as interactive stuttering and audio-visual asynchrony, severely degrading communication quality. Therefore, accurately and efficiently calculating end-to-end latency is crucial for monitoring, evaluating, and optimizing service quality.

[0003] Currently, the commonly used delay statistics methods in the industry mainly have the following limitations:

[0004] First, there are invasive and inaccurate solutions. For example, the sending end overlays a visible timer or timestamp watermark on the video screen, and the receiving end identifies and calculates the time difference through image recognition or camera capture. This method not only changes the original video content and affects the user experience, but its accuracy is also severely limited by the accuracy of the image recognition algorithm and the frame rate of the camera capture. It cannot achieve automated and high-precision measurement and is difficult to apply on a large scale.

[0005] Secondly, there is the solution that can only measure part of the link. Another common approach is to calculate based on the timestamp of network packets, such as using time information in the RTP / RTCP protocol to estimate network transmission delay. However, this method can only capture the transmission time of packets on the network path, and completely ignores the delay introduced by the crucial acquisition, encoding, decoding, and rendering links in the audio and video data processing chain. Therefore, its measurement results cannot truly reflect the end-to-end delay of the entire link perceived by the user.

[0006] Furthermore, while there are general solutions for time synchronization, such as network time protocols, their synchronization accuracy is usually in the millisecond range in complex internet environments, and it is difficult to eliminate the errors introduced by network path asymmetry, which cannot meet the stringent requirements of real-time audio and video applications for microsecond-level synchronization accuracy.

[0007] In summary, existing technologies cannot provide an end-to-end latency statistics solution that simultaneously satisfies non-intrusiveness, high accuracy, full-link coverage, and latency decomposition capabilities. This results in developers and operations personnel lacking effective tools when locating performance bottlenecks, making it difficult to quickly identify whether the problem occurs at the acquisition end, network transmission, or playback end.

[0008] Therefore, in view of the above situation, the present invention provides a real-time audio and video end-to-end delay statistics method based on bidirectional time synchronization. Summary of the Invention

[0009] To overcome the aforementioned deficiencies of the prior art, this invention provides a real-time audio and video end-to-end delay statistics method based on bidirectional time synchronization to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization, comprising the following steps:

[0011] S1. Time synchronization: The push streaming client and pull streaming client exchange bidirectional messages with the SFU server to calculate the clock deviation between their local clock and the SFU server clock.

[0012] S2. Data Acquisition and Embedding: The streaming client acquires video frames, records the acquisition timestamp based on its local clock, combines the acquisition timestamp with the clock offset to convert it into an SFU reference timestamp, and generates SEI information containing the SFU reference timestamp and stage duration metadata, which is sent to the SFU server along with the corresponding encoded video frames.

[0013] S3, Forwarding: The SFU server receives and forwards the video stream containing SEI information to the streaming client.

[0014] S4. Delay Calculation and Decomposition: The streaming client parses the SEI information to obtain the SFU base timestamp and stage time metadata. It then converts these to the local time base of the streaming client by combining its own clock deviation. The total end-to-end delay is calculated by comparing it with the rendering timestamp. At the same time, the total delay is decomposed by combining the stage time metadata and the timestamp recorded locally by the streaming client.

[0015] Preferably, in step S1, the bidirectional message exchange is based on the HTTP protocol;

[0016] The formula for calculating the clock deviation is:

[0017]

[0018] Where T1 is the client's request sending timestamp, T2 is the SFU's request receiving timestamp, T3 is the SFU's response sending timestamp, and T4 is the client's response receiving timestamp.

[0019] Preferably, step S1 further includes a dynamic error compensation mechanism, specifically: performing multiple bidirectional message exchanges, filtering the calculated multiple clock deviations, and / or introducing a dynamic calibration factor based on historical synchronization data to adjust the clock deviations in real time.

[0020] Preferably, in step S2, the stage time metadata included in the SEI information includes at least the acquisition time and encoding time, and the SEI information is processed by lossless compression or encryption before embedding.

[0021] Preferably, in step S3, the SFU server adds an SFU receiving timestamp and a forwarding timestamp to the video data packet when forwarding it.

[0022] Preferably, in step S4, the delayed decomposition includes at least:

[0023] Based on the encoding time in the SEI information, the processing time at the streaming end is obtained;

[0024] Calculate network transmission time based on the SFU receive timestamp, forwarding timestamp, and pull-end packet receive timestamp;

[0025] The decoding time is calculated based on the start and end timestamps of the streaming end.

[0026] Preferably, it also includes S5, performance optimization:

[0027] The calculated latency data is reported in real time, and the encoding parameters of the streaming end are dynamically adjusted or alarms are triggered based on this.

[0028] Preferably, the streaming client is configured with modules for time synchronization, SEI information generation and embedding;

[0029] The SFU server is configured with modules for providing time synchronization services and forwarding video streams;

[0030] The streaming client is configured with modules for time synchronization, SEI information parsing, and latency calculation decomposition.

[0031] The technical effects and advantages of this invention are as follows:

[0032] 1. This invention calculates the clock deviation between the client and the server through bidirectional message exchange, which can effectively offset the error caused by the asymmetry of network latency between the request path and the response path, thereby obtaining high-precision time synchronization at the microsecond level. On this basis, the synchronized absolute timestamp is embedded into the video stream through supplementary enhancement information units, so that the pull end can obtain the accurate time reference of the push end's acquisition time. The final calculated latency covers all links such as acquisition, encoding, network transmission, decoding, and rendering, realizing true full-link latency statistics, rather than one-sided network latency.

[0033] 2. This invention utilizes the Supplemental Enhancement Information Unit (SUE) in the video coding standard as a carrier of metadata. This unit is encoded and transmitted together with the video frame, without occupying video pixel data or affecting the normal decoding and display of the main video stream. It is completely transparent to the user. More importantly, this method not only transmits the absolute timestamp in the SUE, but also embeds the time consumption of each processing stage. Combined with the decoding and rendering timestamps recorded locally at the streaming end and the forwarding timestamps injected by the selective forwarding unit server, the system can accurately decompose the total end-to-end latency into each processing stage, enabling developers and maintenance personnel to quickly locate performance bottlenecks and providing clear data support for system optimization.

[0034] 3. The time synchronization process of this invention only occurs during connection establishment or periodically, rather than operating on each frame of video. The communication overhead is extremely low, and the bandwidth occupied by the supplementary enhancement information unit itself is minimal and negligible after compression. The method relies on a general selective forwarding unit server and the standard H.264 / HEVC encoding specification in its architecture, without requiring modification to the existing core audio and video transmission links. Therefore, it has strong versatility and can be widely used in real-time communication systems based on selective forwarding unit architecture, such as video conferencing, live streaming, and cloud gaming. In addition, since the latency data can be calculated and reported in real time, the system parameters can be dynamically adjusted based on this data, significantly improving the system's adaptability and user experience. Attached Figure Description

[0035] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1: Basic Method Flow

[0038] This invention provides a real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization, specifically including the following steps:

[0039] S1, Time Synchronization

[0040] Before or simultaneously with the start of audio and video streaming, the push client (such as a mobile phone or computer) and the pull client (such as another mobile phone or TV) exchange messages bidirectionally with the selective forwarding unit server to calculate the clock deviation between their respective local clocks and the selective forwarding unit server's clock, denoted as Δ.推流 and Δ 拉流 .

[0041] S2. Data Acquisition and SEI Information Embedding

[0042] The streaming client starts the camera to capture video frames. For each video frame, perform the following operations:

[0043] Record the local timestamp when the frame was captured, and mark it as the capture timestamp. 本地 ;

[0044] The video frames are fed into the encoder for encoding, and the local timestamp at the end of encoding is recorded and marked as the encoding end timestamp. 本地 ;

[0045] Calculate the data acquisition time. Data acquisition time = encoding start timestamp 本地 -Collection timestamp 本地 (The encoding start time can be approximated as the encoding end time of the previous frame or the system call time). Calculate encoding time: Encoding time = Encoding end timestamp 本地 - Encoding start timestamp 本地 ;

[0046] Using the clock offset Δ calculated in step S1 推流 Convert the collection timestamp to a selective forwarding unit reference timestamp: release time 选择性转发单元 =Collection timestamp 本地 +Δ 推流 ;

[0047] Construct a supplementary and enhanced information payload that includes the following metadata;

[0048] Release time: The absolute acquisition time of the selective forwarding unit baseline;

[0049] Data collection time: The time spent in the data collection phase;

[0050] Encoding time: Time spent in the encoding phase;

[0051] Frame sequence number: The video frame sequence number used for tracking.

[0052] The supplemental enhancement payload is encoded together with the corresponding video frame and sent to the selective forwarding unit server along with the video stream.

[0053] S3, Selective forwarding unit forwarding

[0054] The selective forwarding unit server receives video streams from push clients. The selective forwarding unit does not need to parse or modify the content in the supplementary enhancement information, but treats it as a transparent data unit and forwards it directly to one or more pull clients according to the subscription relationship.

[0055] S4, Delayed Calculation and Decomposition

[0056] The streaming client performs the following operations:

[0057] The video stream is received from the selective forwarding unit, decoded, and video frames and embedded supplementary enhancement information are obtained.

[0058] Release time was extracted from supplementary and enhanced information. 选择性转发单元 Metadata such as collection time and encoding time;

[0059] Using the clock offset Δ calculated in step S1 拉流 Convert the selective forwarding unit's reference timestamp back to the collection time under the local time reference of the pull stream endpoint: Collection Time 在拉流端 =Published time 选择性转发单元 -Δ 拉流 ;

[0060] Record the local timestamp at which the current frame begins rendering to the screen, and mark it as the rendering timestamp. 本地 ;

[0061] Calculate total end-to-end delay: end-to-end 延迟 =Rendering timestamp 本地 -Collection time 在拉流端 ;

[0062] A preliminary decomposition of the total delay is performed:

[0063] Total streaming time = Acquisition time + Encoding time (obtained directly from supplementary enhancement information)

[0064] Decoding time at the streaming end = Decoding end timestamp 本地 -Decode start timestamp 本地 (Obtained by recording the start and end times of decoding);

[0065] Rendering time on the streaming end = Rendering timestamp 本地 -Decoding end timestamp 本地 (Estimation)

[0066] Network and queuing time ≈ end-to-end 延迟 -(Total streaming time + Decoding time + Rendering time) (estimated using the difference method).

[0067] Example 2: This example includes high-precision time synchronization and selective forwarding unit timestamps.

[0068] This embodiment refines steps S1 and S2 based on embodiment 1.

[0069] In the time synchronization of step S1, bidirectional message exchange is based on the Hypertext Transfer Protocol, and the specific process is as follows:

[0070] The streaming client generates a Hypertext Transfer Protocol (HTTP) request to acquire or submit data, and records its local timestamp T1 at the moment the request is sent.

[0071] The selective forwarding unit server records its local timestamp T2 the instant it receives the request;

[0072] When the selective forwarding unit server processes a request and prepares to send a response, it records its local timestamp T3 again.

[0073] The selective forwarding unit server fills T2 and T3 into the Hypertext Transfer Protocol response body and sends it back to the streaming client;

[0074] The moment the streaming client receives the response, it records its local timestamp T4;

[0075] The streaming client uses the following formula to calculate its clock skew with the selective forwarding unit, effectively offsetting the errors caused by network transmission path asymmetry:

[0076]

[0077] In step S3, when the selective forwarding unit server forwards video data packets to the streaming client, it adds two timestamps to the data packets at the application layer:

[0078] Receive timestamp 选择性转发单元 The moment when the selective forwarding unit receives this data packet from the push stream end;

[0079] Forwarding timestamp 选择性转发单元 The moment when the selective forwarding unit prepares to send this data packet to the streaming end;

[0080] In the delay decomposition of S4, the timestamps added by the selective forwarding unit can be used to calculate network transmission time more accurately:

[0081] Uplink network time ≈ Receive timestamp 选择性转发单元 -(Published Time) 选择性转发单元 +Δ 推流 (Δ here) 推流 Used to convert the release time selective forwarding unit back to the push-stream time for alignment calculation, or directly use the packet sending time recorded by the push-stream.

[0082] Selective forwarding unit processing and queuing time = forwarding timestamp 选择性转发单元 - Receive timestamp 选择性转发单元 ;

[0083] Downlink network time ≈ Reception time在拉流端 - Forwarding timestamp 选择性转发单元 (wherein, receiving time) 在拉流端 (The local time at which the streaming end received the data packet);

[0084] Total network transmission time = Uplink network time + Selective forwarding unit processing and queuing time + Downlink network time.

[0085] Example 3 includes dynamic error compensation and performance optimization.

[0086] This embodiment further enhances the robustness and application value of the system based on the above embodiments.

[0087] In the time synchronization of step S1, a dynamic error compensation mechanism is introduced;

[0088] Multiple measurements and filtering: The client (pull / pull stream) continuously initiates three Hypertext Transfer Protocol time synchronization requests, obtaining three initial clock offset values ​​Δ1, Δ2, and Δ3. Subsequently, a filtering algorithm, such as the moving average method, is used: ( (Or Kalman filtering) to smooth random errors and obtain the final clock offset value used for calculation;

[0089] Dynamic calibration: The client resynchronizes its time every 60 seconds. The system records historical clock deviation values ​​and calculates the clock drift rate. A dynamic calibration factor α (e.g., α = 1.000001 means 1 microsecond fast per second) is introduced to fine-tune the clock deviation value used between two synchronizations: Δ 已调整 =Δ 前次 +α*Time elapsed;

[0090] In the performance optimization in step five:

[0091] The total latency and the time consumed at each stage, calculated by the streaming end, are reported in real time to a monitoring backend system via a signaling channel (such as a web socket). This system can:

[0092] Real-time monitoring and alarms: The latency curve is dynamically displayed on the console interface. When the total latency exceeds the set threshold (e.g., 400 milliseconds), the system automatically triggers an alarm to notify the operation and maintenance personnel.

[0093] Dynamic strategy adjustment: The monitoring system or client software toolkit itself can automatically adjust based on latency data. For example, when the network transmission time is detected to be consistently too high, the push end can be instructed to reduce the video encoding bitrate or resolution. When the decoding time is abnormal, the pull end can be instructed to switch to a more efficient decoder or reduce the rendering quality.

[0094] Bottleneck identification: By analyzing the proportion of time spent in each stage, performance bottlenecks can be quickly identified. For example, if encoding time accounts for the largest proportion, the performance of the streaming device or encoding parameters should be optimized. If network transmission time accounts for the largest proportion, the network status should be checked or the link should be switched.

[0095] It is worth noting that in this invention:

[0096] Streaming client: A terminal device that captures, encodes, and sends audio and video streams;

[0097] Streaming client: A terminal device that receives, decodes, and renders audio and video streams;

[0098] SFU: Selective Forwarding Unit, a type of media server responsible for receiving streams from push clients and forwarding them to one or more pull clients as needed;

[0099] End-to-end latency: refers to the total time elapsed from the moment a video frame is captured by the push client to the moment the pull client renders that frame onto the screen;

[0100] SEI: Supplemental Enhancement Information, is a data unit specified in video coding standards such as H.264 / HEVC that can be used to carry additional information, such as timestamps and user-defined data. It can be transmitted along with the video stream without affecting the decoding of the main video stream.

[0101] Time difference: The deviation between the client's local clock and the SFU server clock;

[0102] Stage time consumption: refers to the time consumed in a specific processing step within the end-to-end latency, such as acquisition time, encoding time, network transmission time, decoding time, rendering time, etc.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization, characterized in that: Includes the following steps: S1. Time synchronization: The push streaming client and pull streaming client exchange bidirectional messages with the SFU server to calculate the clock deviation between their local clock and the SFU server clock. S2. Data Acquisition and Embedding: The streaming client acquires video frames, records the acquisition timestamp based on its local clock, combines the acquisition timestamp with the clock offset to convert it into an SFU reference timestamp, and generates SEI information containing the SFU reference timestamp and stage duration metadata, which is sent to the SFU server along with the corresponding encoded video frames. S3, Forwarding: The SFU server receives and forwards the video stream containing SEI information to the streaming client. S4. Delay Calculation and Decomposition: The streaming client parses the SEI information to obtain the SFU base timestamp and stage time metadata. It then converts these to the local time base of the streaming client by combining its own clock deviation. The total end-to-end delay is calculated by comparing it with the rendering timestamp. At the same time, the total delay is decomposed by combining the stage time metadata and the timestamp recorded locally by the streaming client.

2. The real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization according to claim 1, characterized in that: In step S1, the bidirectional message exchange is based on the HTTP protocol; The formula for calculating the clock deviation is: Where T1 is the client's request sending timestamp, T2 is the SFU's request receiving timestamp, T3 is the SFU's response sending timestamp, and T4 is the client's response receiving timestamp.

3. The real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization according to claim 1, characterized in that: Step S1 further includes a dynamic error compensation mechanism, specifically: performing multiple bidirectional message exchanges, filtering the calculated clock deviations, and / or introducing a dynamic calibration factor based on historical synchronization data to adjust the clock deviations in real time.

4. The real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization according to claim 1, characterized in that: In step S2, the SEI information includes at least the acquisition time and encoding time in the stage time metadata, and the SEI information is processed by lossless compression or encryption before embedding.

5. The real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization according to claim 1, characterized in that: In step S3, the SFU server adds an SFU receiving timestamp and a forwarding timestamp to the video data packets when forwarding them.

6. The real-time audio and video end-to-end latency statistics method based on bidirectional time synchronization according to claim 1, characterized in that: In step S4, the delayed decomposition includes at least: Based on the encoding time in the SEI information, the processing time at the streaming end is obtained; Calculate network transmission time based on the SFU receive timestamp, forwarding timestamp, and pull-end packet receive timestamp; The decoding time is calculated based on the start and end timestamps of the streaming end.

7. The real-time audio and video end-to-end delay statistics method based on bidirectional time synchronization according to claim 1, characterized in that: It also includes S5 and performance optimizations: The calculated latency data is reported in real time, and the encoding parameters of the streaming end are dynamically adjusted or alarms are triggered based on this.

8. The real-time audio and video end-to-end delay statistics method based on bidirectional time synchronization according to claim 1, characterized in that: The streaming client is configured with modules for time synchronization, SEI information generation and embedding; The SFU server is configured with modules for providing time synchronization services and forwarding video streams; The streaming client is configured with modules for time synchronization, SEI information parsing, and latency calculation decomposition.

Images