A method, system, apparatus and device for latency statistics

Abstract

The embodiment of the present specification provides a time delay statistical method, system, device and equipment. The scheme comprises: a receiving end receives a data frame sent by a sending end through a first communication channel and a frame identifier of the data frame; the receiving end time information of the data frame is recorded correspondingly to the frame identifier, and the receiving end time information can include a processing completion timestamp of the data frame at the receiving end; wherein the time of the sending end and the receiving end is synchronized; time delay statistical information is received through a second communication channel; the time delay statistical information contains a target frame identifier of a target data frame and first time information; the first time information can include a collection timestamp of the target data frame; the second time information of the target data frame is queried from the receiving end time information based on the target frame identifier; and an end-to-end time delay from the sending end to the receiving end is calculated according to a target processing completion timestamp in the second time information and the collection timestamp in the first time information.

Description

Technical Field

[0001] This specification relates to the field of time delay statistics technology in one or more embodiments, and particularly to a time delay statistics method, apparatus and device. Background Technology

[0002] With the development of real-time audio and video communication technologies, especially the widespread adoption of WebRTC, low-latency live streaming, cloud rendering, and multimodal interaction, users are demanding higher levels of "face-to-face" interactive experiences. End-to-end latency of media streams is one of the core metrics for measuring user experience, and accurately measuring end-to-end latency is a crucial prerequisite for performance optimization and ensuring Quality of Service (QoS) and Quality of Experience (QoE).

[0003] Therefore, how to provide a delay statistics method to accurately measure end-to-end delay has become an urgent technical problem to be solved. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a delay statistics method, apparatus, and device to improve the accuracy of delay statistics without intruding on the media stream.

[0005] According to a first aspect of one or more embodiments of this specification, a delay statistics method is provided, applied at a receiving end, comprising: The system receives data frames sent by the sending end and the frame identifier of the data frames through the first communication channel. The receiving end time information of the data frame is recorded corresponding to the frame identifier, and the receiving end time information includes the processing completion timestamp of the data frame at the receiving end; the sending end and the receiving end are time synchronized. The delay statistics are received through a second communication channel; the delay statistics include the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame. Based on the target frame identifier, query the second time information of the target data frame from the time information of the receiving end; The end-to-end delay from the sending end to the receiving end is calculated based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

[0006] According to a second aspect of one or more embodiments of this specification, a delay statistics method is provided, applied at a transmitting end, comprising: The sending end sends a data frame and its frame identifier to the receiving end through the first communication channel; the sending end and the receiving end are time-synchronized. The delay statistics information is sent to the receiving end through the second communication channel. The delay statistics information includes the target frame identifier of the target data frame and the first time information. The first time information includes the acquisition timestamp of the target data frame. The latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0007] According to a third aspect of one or more embodiments of this specification, a latency statistics method is provided, applied to the middle end of a service processing link, comprising: The system receives data frames sent by the sending end and the frame identifier of the data frames through the first communication channel. The data frame and the frame identifier are sent to the receiving end through the first communication channel; the time of the intermediate end, the sending end, and the receiving end are synchronized; The system receives first latency statistics sent by the sending end through a second communication channel; the first latency statistics include the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame. The second delay statistics are sent to the receiving end through the second communication channel; the second delay statistics include the first delay statistics. The second latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0008] According to a fourth aspect of one or more embodiments of this specification, a latency statistics system is provided, comprising: a transmitter and a receiver, wherein the transmitter and the receiver are time-synchronized. The transmitting end is configured to transmit data frames and frame identifiers of the data frames through a first communication channel; and to transmit latency statistics through a second communication channel, wherein the latency statistics include the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame. The receiving end is configured to receive data frames and frame identifiers of the data frames sent by the sending end through a first communication channel; record receiving end time information of the data frames corresponding to the frame identifiers, the receiving end time information including the processing completion timestamp of the data frames at the receiving end; receive the delay statistics information through a second communication channel; query the second time information of the target data frames according to the target frame identifier; and calculate the end-to-end delay from the sending end to the receiving end based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

[0009] According to a fifth aspect of one or more embodiments of this specification, a delay statistics device is provided, applied at a receiving end, comprising: The first receiving module is used to receive data frames sent by the sending end and the frame identifier of the data frames through the first communication channel; A recording module is used to record the receiving end time information of the data frame corresponding to the frame identifier, the receiving end time information including the processing completion timestamp of the data frame at the receiving end; the sending end and the receiving end are time synchronized; The second receiving module is used to receive latency statistics information through the second communication channel; the latency statistics information includes the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame; The query module is used to query the second time information of the target data frame from the receiving end time information based on the target frame identifier; the second time information includes the target processing completion timestamp of the target data frame at the receiving end; The first calculation module is used to calculate the end-to-end delay from the sending end to the receiving end based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

[0010] According to a sixth aspect of one or more embodiments of this specification, a delay statistics device is provided, applied at a transmitting end, comprising: A first transmitting module is configured to transmit a data frame and a frame identifier of the data frame to a receiving end through a first communication channel; the transmitting end and the receiving end are time-synchronized. The second sending module is used to send latency statistics information to the receiving end through the second communication channel. The latency statistics information includes the target frame identifier of the target data frame and first time information. The first time information includes the acquisition timestamp of the target data frame. The latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0011] According to a seventh aspect of one or more embodiments of this specification, a latency statistics device is provided, applied at the middle end of a service processing link, comprising: The first receiving module is used to receive data frames sent by the sending end and the frame identifier of the data frames through the first communication channel; A first transmitting module is used to transmit the data frame and the frame identifier to the receiving end through the first communication channel; the intermediate end, the transmitting end, and the receiving end are time-synchronized; The second receiving module is used to receive first delay statistics information sent by the sending end through a second communication channel; the first delay statistics information includes a target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame; The second sending module is used to send second delay statistics to the receiving end through the second communication channel; the second delay statistics include the first delay statistics. The second latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0012] According to an eighth aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer instructions, implements the steps of the latency statistics method.

[0013] One embodiment of this specification can achieve at least the following beneficial effects: The receiving end receives data frames and frame identifiers sent by the sending end through a first communication channel, records the receiving end time information of the data frames corresponding to the frame identifiers, and receives latency statistics information through a second communication channel. Based on the target frame identifier in the statistics information, the receiving end queries the second time information of the target data frame from the receiving end time information. The end-to-end latency is calculated based on the acquisition timestamp of the target data frame in the latency statistics information and the target processing of the target data frame in the second time information. This method, by separating the latency statistics information from the media stream during transmission, ensures that the data frames themselves do not carry any timestamp information, achieving zero-intrusion latency statistics of the original media stream, and also decoupling the latency statistics process from the media stream transmission process. In addition, the sending and receiving ends synchronize their times in advance. The receiving end directly uses the difference between the target processing completion timestamp of the target data frame recorded locally and the acquisition timestamp of the target data frame carried in the statistical information to calculate the end-to-end delay, which improves the accuracy of delay statistics.

[0014] In addition, the latency statistics are transmitted through an independent communication channel, so the receiving end can obtain the first-time information required for latency statistics without parsing the media stream, eliminating the measurement bias introduced by parsing streaming media and further improving the accuracy of latency statistics. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of a delay statistics method provided in the embodiments of this specification; Figure 2 This is a flowchart illustrating a delay statistics method applied to a receiving end, as provided in the embodiments of this specification. Figure 3 This is a flowchart illustrating a latency statistics method applied to the transmitting end, as provided in the embodiments of this specification. Figure 4 This is a flowchart illustrating a latency statistics method applied to an intermediate end in a business processing link, as provided in the embodiments of this specification. Figure 5 This is a schematic diagram of the structure of a delay statistics system provided in the embodiments of this specification; Figure 6 This is a swimlane diagram of a time delay statistics method provided in the embodiments of this specification; Figure 7 This is a flowchart of a delay statistics method provided in the embodiments of this specification; Figure 8 This is a schematic diagram of a delay statistics device applied to a receiving end, as provided in the embodiments of this specification. Figure 9 This is a schematic diagram of a delay statistics device applied to a transmitting end, provided in an embodiment of this specification. Figure 10 This is a schematic diagram of the structure of a latency statistics device applied to the middle end of a service processing link, as provided in the embodiments of this specification. Figure 11This is a schematic diagram of the structure of a computer device provided in the embodiments of this specification. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0018] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0019] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0020] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0021] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0022] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0023] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0024] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0025] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0026] The following explains the terms and concepts used in one or more embodiments of this specification.

[0027] Supplemental Enhancement Information (SEI) is an additional data mechanism in video coding standards such as H.264 / AVC, H.265 / HEVC, and H.266 / VVC. It is used to embed non-essential but useful metadata into the video bitstream and is often used to transmit metadata such as timestamps and subtitles.

[0028] Data Channel is a bidirectional communication channel defined in WebRTC that can transmit arbitrary custom data in addition to audio and video streams.

[0029] Round-Trip Time (RTT) is a core metric in network communication, referring to the total time it takes for a data packet to travel from the sender to the receiver and back.

[0030] Real-time Transport Protocol (RTP) is used for the encapsulation and transmission of audio and video data. Media streams are encapsulated in RTP packets for transmission.

[0031] End-to-end latency is the total time it takes for an audio or video frame to be processed by the receiving end from the moment it is captured, through encoding, transmission, forwarding, and decoding.

[0032] Multimodal backend links refer to backend processing service chains that support multiple modal inputs such as audio, video, and text, and typically include multiple intermediate processing nodes (such as access gateways, AI inference engines, media servers, etc.).

[0033] Service link: refers to the complete business processing link from the sending end, through zero or one or more intermediate processing nodes, to the receiving end. Multimodal backend links can be part of the service link.

[0034] With the rapid development of real-time audio and video communication technologies, especially the widespread application of WebRTC, low-latency live streaming, cloud rendering, and multimodal interaction, users are placing increasingly higher demands on "face-to-face" interactive experiences. End-to-end latency of media streams is one of the core indicators for measuring user experience, and accurately measuring end-to-end latency is an important prerequisite for performance optimization and ensuring Quality of Service (QoS) and Quality of Experience (QoE).

[0035] In existing technologies, after acquiring a media stream frame, the transmitting end typically reads the local system clock immediately to generate an acquisition timestamp. It then calls the encoder interface to encapsulate this timestamp into a custom SEI (Search Engine Identity), inserting it into the header of the corresponding media stream data frame. The complete media stream data frame carrying the SEI timestamp is then packaged into an RTP packet and sent. Upon receiving the media stream data frame, the receiving end records the current system time as the arrival time of the media stream data frame. It then parses the SEI to obtain the acquisition time and estimates the latency based on the difference between the arrival time and the acquisition time.

[0036] Existing methods for inserting SEI information into media stream data frames require each node to frequently parse the SEI field in RTP packets to extract timestamp information. However, SEI parsing typically relies on the complete decoding process and is susceptible to factors such as decoding time, frame out-of-order delivery, and packet loss / retransmission. On one hand, the processing delay introduced by the decoding process itself is included in the latency measurement results, causing inaccurate latency measurements. On the other hand, when frame out-of-order delivery or packet loss occurs, SEI information may not be obtained in a timely manner or correctly associated with the corresponding frame, further leading to deviations in latency measurement results. Therefore, it is difficult to meet the accuracy requirements for latency statistics in high real-time scenarios.

[0037] To address the shortcomings in related technologies, this solution provides the following embodiments.

[0038] Figure 1 This is a schematic diagram illustrating an application scenario of a delay statistics method provided in the embodiments of this specification.

[0039] like Figure 1As shown, the scheme may include a transmitter 101 and a receiver 102. The transmitter 101 can be used to acquire media streams and send media stream data frames and frame identifiers of the data frames to the receiver 102 frame by frame through a first communication channel. It can also determine a target data frame from the media stream data frames sent frame by frame and send latency statistics information to the receiver 102 through a second communication channel. The latency statistics information includes the target frame identifier of the target data frame and first time information. The first time information represents the operation time of the transmitter for the target data frame and may include the acquisition timestamp of the target data frame. The receiver 102 can be used to receive the data frames sent by the transmitter 101 and the frame identifiers of the data frames through the first communication channel, and record the receiver time information for the data frames. The receiver time information represents the operation time of the receiver for the data frames and may include the processing completion timestamp of the receiver for the data frames. After receiving the latency statistics information through the second communication channel, the receiving end 102 can query the second time information of the target data frame from the receiving end's time information. Then, it calculates the end-to-end latency based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information. It can be understood that the second time information represents the operation time of the receiving end for the target data frame, and the second time information may include the target processing completion timestamp of the target data frame.

[0040] like Figure 1 In the scenario shown, the sender 101 and receiver 102 are two logical endpoints in the service link. The sender 101 is the source node in the service link, responsible for acquiring, encoding, and sending the media stream. In one or more embodiments of this specification, the sender 101 can be a mobile phone, personal computer, VR headset, smart camera, or smart microphone. The receiver 102 can be the last node in the service link, responsible for consuming or processing the media stream, and can also be responsible for calculating end-to-end latency. In practical applications, the specific form of the receiver 102 varies depending on the scenario. For example, in multimodal interaction products, the receiver 102 can be a large model processing node; in a remote medical consultation system, the receiver 102 can be a consultation terminal device (expert terminal); in an online education interactive classroom scenario, the receiver 102 can be a student terminal device; in a virtual reality social platform, the receiver 102 can be a VR headset for participants. In an intelligent customer service voice interaction system, the receiver 102 can be a large model processing node (e.g., a cloud server) or a final response generation node.

[0041] This application provides a delay statistics method. This application also relates to a delay statistics device and a computing device, which will be described in detail in the following embodiments.

[0042] Figure 2This is a flowchart illustrating a delay statistics method applied to a receiving end, as provided in an embodiment of this specification.

[0043] From a procedural perspective, the entity executing the process can be a receiving device or equipment. It can be understood that this method can be executed by any device, equipment, or cluster of devices with computing and processing capabilities.

[0044] like Figure 2 As shown, the process may include the following steps.

[0045] Step 202: Receive the data frame sent by the sending end and the frame identifier of the data frame through the first communication channel.

[0046] In one or more embodiments of this specification, a data frame can be a media stream data frame, such as a video frame or an audio frame; it can also be other data units that need to be transmitted and have their latency measured in the service link, such as interactive data frames, sensor data frames, or game status frames. The service link refers to the complete business processing link through which a data frame is sent from the sending end, passes through zero or one or more intermediate processing nodes, and finally reaches the receiving end.

[0047] In one or more embodiments of this specification, a data frame is the basic unit of a media stream. For video data, a data frame can refer to a single image frame, such as an H.264 or H.265 encoded frame; for audio data, a data frame can refer to a single audio block, such as a 20ms PCM or Opus encoded frame.

[0048] A frame identifier can be a unique identifier generated by the sending end for each data frame, used to uniquely identify the data frame throughout the entire service link. For example, the frame identifier can be an identifier generated based on the acquisition timestamp, or it can be a monotonically increasing sequence number assigned to each data frame by the sending end, or it can be a value obtained by hashing the data frame content or key information, or it can be a globally unique identifier, such as a UUID, or it can be a composite identifier composed of multiple forms mentioned above. It is worth noting that in one or more embodiments of this specification, the frame identifier exists in an "associated" manner with the data frame, rather than in a "carried" manner. That is, the data frame does not contain a frame identifier, and also does not contain a timestamp.

[0049] The sending end can be a source node in the service link, used to collect media stream data, generate frame identifiers, and send data frames through the first communication channel. Specifically, the sending end can be a terminal device, such as a mobile phone, personal computer, VR headset, smart camera, or smart microphone. Alternatively, the sending end can also be a server node, such as a streaming server or an AI content generation node.

[0050] In practical applications, after acquiring a frame of data, the sending end can generate a unique frame identifier for that frame and record the sending end time information corresponding to the frame identifier. The sending end time information can refer to time information related to the data frame recorded by the sending end, such as the acquisition timestamp of the data frame recorded when it was acquired. In one or more embodiments of this specification, the frame identifier and the sending end time information can be stored in the sending end's local cache.

[0051] The receiving end can be the last node in the service link. The first communication channel can be a media transport channel used to transmit media streams. The first communication channel can be based on the Real-time Transport Protocol (RTP) to transmit encoded media data, i.e., RTP packets. In real-time communication systems such as WebRTC, the first communication channel is the media transport channel.

[0052] In one or more embodiments of this specification, the data frame itself does not contain a frame identifier and a timestamp. The frame identifier exists in a way that is "associated" with the data frame rather than "carried", which realizes "zero-intrusion" transmission of the original media stream and avoids the time delay deviation introduced by each node parsing the supplementary enhancement information SEI during the transmission of the data frame.

[0053] Step 204: Record the receiving end time information of the data frame corresponding to the frame identifier. The receiving end time information includes the processing completion timestamp of the data frame at the receiving end. The sending end and the receiving end are time synchronized.

[0054] After receiving a data frame and its frame identifier, the receiving end can record the corresponding receiving end time information and frame identifier in its local cache table. This allows for querying the receiving end time information of any data frame based on the frame identifier. The receiving end time information represents the time of the receiving end's operation on the data frame. For example, the receiving end time information may include the processing completion timestamp of the data frame at the receiving end. The processing completion timestamp can refer to the moment when the data frame completes all necessary processing steps at the receiving end. The specific content reflected by the processing completion timestamp depends on the receiving end's role in the service chain. For example, if the receiving end is a large model processing node, the processing completion timestamp could be the time when the large model completes inference and generates results; if the receiving end is a recording and storage node, the processing completion timestamp could be the time when the data frame is written to the storage medium.

[0055] In one or more embodiments of this specification, time synchronization between the sending and receiving ends may mean that the system clock errors of the sending and receiving ends are within a preset range, but not necessarily exactly the same. For example, the system clock errors of the sending and receiving ends may be within milliseconds. In practical applications, clock calibration can be performed on the sending and receiving ends based on the same clock source. For example, time calibration can be performed on the sending and receiving ends using a network time protocol, or a precision time protocol can be used to further improve synchronization accuracy, or the server can uniformly send a reference time to the sending and receiving ends, thereby providing a unified time reference for subsequent delay calculations.

[0056] In practical applications, the receiving end can continuously monitor the first communication channel, receive RTP packets and frame identifiers sent by the sending end, reassemble the RTP packets to obtain complete data frames, perform business logic processing on the data frames, and read the current system clock when the data frame processing is completed to obtain the processing completion timestamp. Then, the frame identifier and the processing completion timestamp are stored in the local cache table of the receiving end to form a mapping relationship between the frame identifier and the processing completion timestamp for subsequent use.

[0057] In one or more embodiments of this specification, the end-to-end delay can be calculated based on the time information of the sending end and the time information of the receiving end.

[0058] Step 206: Receive latency statistics information through the second communication channel; the latency statistics information includes the target frame identifier of the target data frame and the first time information; the first time information may include the acquisition timestamp of the target data frame.

[0059] The second communication channel can be a control signaling channel independent of the first communication channel (media transmission channel), and can be used to transmit latency statistics. In real-time communication systems such as WebRTC, the second communication channel can be a data channel. In one or more embodiments of this specification, the independence of the second communication channel from the first communication channel means that the first and second communication channels are separated from each other at the levels of protocol stack, transmission method, data stream, processing logic, and resource usage, and do not interfere with each other.

[0060] Latency statistics can be generated on demand by the sending end and used to trigger statistical instructions for latency calculation at the receiving end. The latency statistics may include a target frame identifier and first-time information. The target frame identifier is the frame identifier of the target data frame, which can be used to determine the sending time information (i.e., first-time information) of the target data frame in the sending end's local cache table, and also to determine the receiving time information of the target data frame in the receiving end's local cache table.

[0061] In one or more embodiments of this specification, the first time information refers to the time information recorded by the sending end for the target data frame, which can represent the operation time of the sending end for the target data frame. For example, the first time information may include the acquisition timestamp of the target data frame acquired by the sending end.

[0062] In some embodiments, the target data frame may be a single frame selected by the sending end from a continuously transmitted media stream for this latency statistics. The sending end may determine the target data frame according to a preset sampling strategy. For example, the target data frame may be determined by sampling at fixed frame intervals, such as sampling one frame every 100 frames; or, the target data frame may be determined by sampling at fixed time intervals, such as sampling one frame every 1 second; or, the target data frame may be determined by random sampling. This achieves lightweight latency statistics and avoids the signaling overhead caused by frequent statistics. In one or more embodiments of this specification, the target data frame may be a data frame in the media stream already transmitted by the sending end. The target data frame in step 206 may be at least a portion of the frames selected from the data frames in step 202.

[0063] In practical applications, after determining the target data frame, the sending end can query the target frame identifier and first time information of the frame from the local cache table of the sending end based on the target frame identifier. The target frame identifier and the first time information are packaged into latency statistics and sent to the data node downstream of the service link through the second communication channel.

[0064] In one or more embodiments of this specification, latency statistics are transmitted through an independent channel without modifying the media stream, thus avoiding the intrusion of SEI injection into the media stream and the waste of bandwidth.

[0065] Step 208: Based on the target frame identifier, query the second time information of the target data frame from the receiving end time information.

[0066] The second time information is the receiving time information of the target data frame at the receiving end, indicating the operation time of the receiving end for the target data frame. It can be understood that the second time information may include the timestamp of the target processing completed at the receiving end for the target data frame.

[0067] In practical applications, after receiving the latency statistics through the second communication channel, the receiving end can parse the latency statistics and extract the target frame identifier. Then, using the target frame identifier as a key, it can retrieve the receiving end time information of the target data frame from the receiving end's time information to obtain the second time information.

[0068] Step 210: Calculate the end-to-end delay from the sending end to the receiving end based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

[0069] In one or more embodiments of this specification, end-to-end latency refers to the total time taken from the start of data frame acquisition at the sending end to the completion of data frame processing at the receiving end.

[0070] In one or more embodiments of this specification, the end-to-end delay can be obtained based on the difference between first time information and second time information. For example, the end-to-end delay can be calculated based on the difference between the acquisition timestamp in the first time information and the target processing completion timestamp in the second time information.

[0071] Figure 2 The method described involves the receiving end receiving data frames and frame identifiers sent by the sending end through a first communication channel. The receiving end records the receiving end time information of the data frames corresponding to the frame identifiers. It also receives latency statistics through a second communication channel. Based on the target frame identifier in the statistics, it queries the second time information of the target data frame from the receiving end time information. Finally, it calculates the end-to-end latency based on the acquisition timestamp of the target data frame in the latency statistics and the target processing completion timestamp of the target data frame in the second time information. This method achieves zero-intrusion latency statistics on the original media stream by separating the latency statistics from the media stream transmission, ensuring that the data frames themselves do not carry any timestamp information. It also decouples the latency statistics process from the media stream transmission process. Furthermore, the sending and receiving ends synchronize their times in advance. The receiving end directly calculates the end-to-end latency by using the difference between the locally recorded target processing completion timestamp of the target data frame and the acquisition timestamp of the target data frame carried in the statistics, thus improving the accuracy of latency statistics.

[0072] In addition, the latency statistics are transmitted through an independent communication channel, so the receiving end can obtain the first-time information required for latency statistics without parsing the media stream, eliminating the measurement bias introduced by parsing streaming media and further improving the accuracy of latency statistics.

[0073] based on Figure 2 In addition to the method described herein, this specification also provides some improved implementation methods, which will be described below.

[0074] In practical applications, the service link may also include one or more intermediate processing nodes, such as access gateways and AI inference nodes. The media stream needs to pass through each intermediate processing node sequentially before reaching the receiving end. In existing technologies, only the total end-to-end latency can usually be measured, and the segmented latency between each node cannot be obtained, making it difficult to accurately locate performance bottlenecks. Based on this, one or more embodiments of this specification provide further improvements.

[0075] Optionally, an intermediate terminal may be included between the sending end and the receiving end; the receiving of delay statistics information through the second communication channel may specifically include: The system receives latency statistics sent by the intermediate end, which further includes third time information of the target data frame; the third time information represents the operation time of the intermediate end for the target data frame.

[0076] The method may further include: Based on the first time information, the second time information, and the third time information, the segmented delay is calculated; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

[0077] In one or more embodiments of this specification, the intermediate end can be one or more processing nodes located between the sender and receiver in the service link. The intermediate end can be used to receive, process, and forward data frames, and participate in the relay recording of latency statistics. In practical applications, the intermediate end can specifically be a network node or service device such as an access gateway, a service processing server, an AI inference node, or a transcoding server. Time synchronization between the intermediate end and both the sender and receiver has been detailed above and will not be repeated here.

[0078] The third time information refers to the time information recorded by the intermediate end for the target data frame, representing the operation time of the intermediate end for the target data frame. For example, the third time information may include the time when the intermediate end receives the target data frame, the time when the target data frame is sent, and the time when the intermediate end completes the business logic processing for the target data frame.

[0079] In one or more embodiments of this specification, the segmented latency corresponding to each end can refer to the transmission latency between any two ends, such as the transmission latency between end A and end B. This can represent the time it takes for a data frame to travel from end A to end B, such as the time it takes for a data frame to be sent from end A to end B; or, for example, the time it takes for a data frame to be decoded from end A to end B. It can also include the processing latency of any end, such as the processing latency of end C, which can represent the time it takes for a data frame to complete some or all of its business processing within end C, such as the time it takes for a data frame to complete encoding at end C, or, for example, the time it takes for a data frame to complete decoding at end C.

[0080] In practical applications, when intermediate terminals exist in the service link, the data frames and frame identifiers sent by the sending end through the first communication channel pass through each intermediate terminal sequentially. The intermediate terminal continuously listens to the first communication channel, receives the RTP packets of the data frames, reassembles the RTP packets to obtain complete data frames, and simultaneously obtains the frame identifier of the frame. After receiving the data frame and its frame identifier, the intermediate terminal can record the intermediate terminal time information of the data frame, recording the intermediate terminal time information and the frame identifier in its local cache, so that the intermediate terminal time information of any data frame can be queried based on the frame identifier. The intermediate terminal time information can be relevant time information recorded by the intermediate terminal for the data frame. For example, the intermediate terminal time information may include one or more of the following: the second reception timestamp of the data frame at the intermediate terminal, the second transmission timestamp of the data frame at the intermediate terminal, and the business logic processing time of the data frame at the intermediate terminal.

[0081] After receiving the latency statistics, the intermediate end can parse the target frame identifier from the statistics and use this identifier as the key to query the pre-recorded intermediate end time information (i.e., the third time information) for that frame in its local buffer. Once the third time information is found, it is appended to the statistics, and then the updated statistics are forwarded to the next intermediate end via the second communication channel. Each subsequent intermediate end repeats the above operation. Ultimately, the statistics received by the receiving end contain the first time information from the sending end and the third time information appended by all intermediate ends along the way. The receiving end can then calculate the segmented latency corresponding to each end.

[0082] In one or more embodiments of this specification, each intermediate terminal records the corresponding time information when the data frame passes through, and during the hop-by-hop transmission of the latency statistics information, the third time information of the target data frame that was recorded in advance is sequentially added to the latency statistics information to form a complete latency chain. Based on the latency chain, segmented latency calculation can be performed to accurately locate performance bottlenecks and provide technical support for refined performance diagnosis.

[0083] For ease of understanding, this specification provides an embodiment of a specific method for calculating the segmented delay corresponding to each terminal.

[0084] Optionally, the first time information may further include a sending completion timestamp from the sending end, and the second time information may further include a receiving timestamp from the receiving end; the third time information may further include an intermediate end receiving timestamp and an intermediate end sending timestamp; the step of calculating the segmented delay corresponding to each end based on the first time information, the second time information, and the third time information may specifically include: The first transmission delay between the sending end and the intermediate end is calculated based on the difference between the receiving timestamp of the intermediate end and the sending completion timestamp of the sending end.

[0085] The second transmission delay between the intermediate end and the receiving end is calculated based on the difference between the receiving end's receiving timestamp and the intermediate end's sending timestamp.

[0086] The first transmission delay can be the transmission delay between the sending end and the intermediate end. Specifically, the first transmission delay can refer to the time elapsed from the sending end to the receiving end of the target data frame, reflecting the time consumed by network transmission and link queuing between the sending end and the intermediate end. In one or more embodiments of this specification, the first time information may further include a sending end completion timestamp, and the third time information may further include an intermediate end reception timestamp. The first transmission delay between the sending end and the intermediate end can be calculated based on the difference between the intermediate end reception timestamp and the sending end completion timestamp.

[0087] The sending completion timestamp can be the timestamp when the sending end finishes sending the first RTP packet of the target data frame, and correspondingly, the intermediate end receiving timestamp can be the timestamp when the intermediate end receives the first RTP packet of the target data frame. Alternatively, the sending completion timestamp can be the timestamp when the sending end finishes sending the last RTP packet of the target data frame, and correspondingly, the intermediate end receiving timestamp can be the timestamp when the intermediate end receives the last RTP packet of the target data frame.

[0088] The second transmission delay can be the transmission delay between the intermediate end and the receiving end. Specifically, the second transmission delay can refer to the time elapsed from when the target data frame is sent from the intermediate end to when it is received by the receiving end, reflecting the time consumed by network transmission and link queuing between the intermediate end and the receiving end. In one or more embodiments of this specification, the second time information may further include the receiving end's receiving timestamp, and the third time information may further include the intermediate end's sending timestamp. The second transmission delay between the intermediate end and the receiving end can be calculated based on the difference between the receiving end's receiving timestamp and the intermediate end's sending timestamp.

[0089] The intermediate end's sending timestamp can be the timestamp when the intermediate end finishes sending the first RTP packet of the target data frame, and correspondingly, the receiving end's receiving timestamp can be the timestamp when the receiving end receives the first RTP packet of the target data frame. Alternatively, the intermediate end's sending timestamp can be the timestamp when the intermediate end finishes sending the last RTP packet of the target data frame, and correspondingly, the receiving end's receiving timestamp can be the timestamp when the receiving end receives the last RTP packet of the target data frame.

[0090] Understandably, if the service link includes multiple intermediate ends, the transmission delay between any two intermediate ends can be calculated based on the third-party time information of each intermediate end, forming a complete link delay distribution map.

[0091] In one or more embodiments of this specification, since all nodes, namely the sending end, the intermediate end and the receiving end, are time-synchronized, the first time information, the third time information and the second time information are all in the same time coordinate system. Therefore, the accurate segmented transmission delay can be obtained by directly subtracting them.

[0092] For ease of understanding, one or more embodiments of this specification also provide another method for calculating latency.

[0093] Optionally, the third time information may include the intermediate end receiving timestamp and the intermediate end sending timestamp; the step of calculating the segmented delay corresponding to each end based on the first time information, the second time information, and the third time information may specifically include: The first processing delay of the intermediate end for the target data frame is calculated based on the difference between the sending timestamp and the receiving timestamp of the intermediate end.

[0094] In one or more embodiments of this specification, the intermediate end receiving timestamp may refer to the moment when the intermediate end receives the target data frame. The intermediate end sending timestamp may refer to the moment when the intermediate end forwards the target data frame to the next node (downstream intermediate end or receiving end).

[0095] Specifically, the intermediate end receive timestamp can be the timestamp of the first RTP packet received by the intermediate end of the target data frame, and correspondingly, the intermediate end send timestamp can be the time when the intermediate end sends the first RTP packet of the target data frame. Alternatively, the intermediate end receive timestamp can be the timestamp of the last RTP packet received by the intermediate end of the target data frame, and correspondingly, the intermediate end send timestamp can be the timestamp of the last RTP packet sent by the intermediate end of the target data frame.

[0096] In one or more embodiments of this specification, performance bottlenecks in the service link can be located based on the first processing latency of each intermediate end.

[0097] In practical applications, after receiving a data frame, the intermediate end can perform business logic processing on it. Correspondingly, the third-party time information can also include a business processing timestamp. The business processing timestamp can be the timestamp recorded by the intermediate end when it completes a specific business process for the target data frame. For example, if the intermediate end needs to decode the target data frame after receiving it, it can record the corresponding decoding completion timestamp upon completion. Understandably, if the intermediate end needs to perform other business logic processing after decoding the target data frame, it can record the corresponding timestamp upon completion of each business logic process.

[0098] In one or more embodiments of this specification, the latency of one or more business processes can be calculated based on the difference between two business processing timestamps, thereby accurately assessing the business processing efficiency of each intermediate end and precisely locating the performance bottleneck of the business logic processing link.

[0099] In one or more embodiments of this specification, the overall processing latency of the sending end and the processing latency of each business logic can also be calculated.

[0100] Optionally, the first time information further includes a timestamp indicating that the sending end has completed sending, and the method may further include: The second processing delay of the sending end for the target data frame is calculated based on the difference between the sending end's completion timestamp and the acquisition timestamp.

[0101] Optionally, the sending end's transmission completion timestamp is the timestamp of the last data packet of the target data frame being sent; the first time information may further include the encoding completion timestamp of the target data frame; the method may further include: The encoding delay of the sending end for the target data frame is calculated based on the difference between the encoding completion timestamp and the acquisition timestamp.

[0102] The whole-frame transmission delay of the sending end for the target data frame is calculated based on the difference between the sending end's transmission completion timestamp and the encoding completion timestamp.

[0103] In one or more embodiments of this specification, the sending end time information may further include an encoding completion timestamp of the data frame recorded when the data frame encoding is completed; and a sending end transmission completion timestamp recorded when the first RTP packet or the last RTP packet of the data frame is transmitted. It is understood that if the sending end performs other processing on the data frame, the sending end time information may also include other timestamps. Correspondingly, the first time information of the target data frame determined from the sending end's local cache based on the target timestamp may include an acquisition timestamp, an encoding completion timestamp, and the sending end transmission completion timestamp of the first or last data packet. Wherein, the acquisition timestamp is the timestamp at which the sending end acquires the target data; the encoding completion timestamp is the timestamp at which the sending end completes the encoding operation for the target data frame; the sending end completion timestamp of the first data packet may be the timestamp at which the sending end transmits the first RTP packet of the target data frame. The sending end transmission completion timestamp of the last data packet may be the timestamp at which the sending end transmits the last RTP packet of the target data frame.

[0104] In one or more embodiments of this specification, the processing delay of each business logic at the sending end can be calculated based on the difference between any two timestamps in the first time information. For example, the encoding delay can be calculated based on the difference between the acquisition timestamp and the encoding completion timestamp; the whole frame transmission delay can be calculated based on the encoding completion timestamp and the sending end transmission completion timestamp of the last RTP packet; and the second processing delay can be calculated based on the acquisition timestamp and the sending end transmission completion timestamp of the first or last RTP packet. The second processing delay can refer to the time taken for the target data frame to complete all business processing within the sending end.

[0105] Similarly, the second time information can also include other timestamps, and the processing latency of each business logic at the receiving end can be calculated based on the difference between any two timestamps. For example, suppose the receiving end needs to review the target data frame after receiving it, and then perform AI inference based on the target data frame. Accordingly, the second time information can include the receiving end's receiving timestamp, the review timestamp, and the AI ​​inference timestamp, where the AI ​​inference timestamp is the target processing completion timestamp. The review latency can be calculated based on the difference between the receiving end's receiving timestamp and the review timestamp; the AI ​​inference latency can be calculated based on the difference between the review timestamp and the AI ​​inference timestamp; and the third processing latency can be calculated based on the difference between the review timestamp and the receiving end's receiving timestamp. The third processing latency can refer to the time taken for the target data frame to complete all business processing within the receiving end.

[0106] In one or more embodiments of this specification, the business logic processing efficiency of each end (sender, intermediate, and receiver) can be accurately evaluated by using various processing latencies (first processing latency, second processing latency, and third processing latency), and it can be determined whether the node has a computing resource bottleneck. Specifically, when the end-to-end latency is high, if the processing latency of a certain node is significantly greater than that of other nodes, the node can be quickly identified as a performance bottleneck. Furthermore, the performance bottleneck can be accurately located based on the business logic processing latency of that node, providing technical support for performance optimization.

[0107] In one or more embodiments of this specification, the receiving end time information may be recorded in the receiving end local cache table; the step of querying the second time information of the target data frame from the receiving end time information based on the target frame identifier may specifically include: Query the time information corresponding to the target frame identifier in the local cache table.

[0108] If the local cache table of the receiving end includes the receiving end time information corresponding to the target frame identifier, then the receiving end time information corresponding to the target frame identifier shall be used as the second time information.

[0109] In one or more embodiments of this specification, the receiver's local cache table records the receiver's time information for each data frame received by the receiver. Specifically, the receiver can create a cache table locally. After receiving a data frame and its frame identifier, the receiver can store the frame identifier and receiver time information in the local cache table. In practical applications, the data in the cache can be cleaned periodically or irregularly to avoid the cache occupying too much storage space and causing the system to run slowly.

[0110] After receiving the latency statistics, the receiving end can parse the statistics to extract the target frame identifier. Then, using this identifier as the key, it can query a local cache table maintained by the receiving end. If the local cache table contains the receiving end's time information corresponding to the target frame identifier, that time information, i.e., the second time information of the target data frame, can be extracted. For example, hash lookup, binary search, or direct query can be used to quickly locate the entry matching the target frame identifier in the local cache table and extract the corresponding second time information. For instance, if the target frame identifier is an incrementing sequence number 1001, the receiving end's local cache table can be searched for an entry with key 1001. If an entry with key 1001 exists in the local cache table, the second time information of the target data frame is obtained.

[0111] In practical applications, if there is no entry matching the target frame identifier in the receiver's local cache table, the latency statistics can be temporarily suspended, and the receiver's local cache table can be queried at a preset frequency until the receiver's time information corresponding to the target frame identifier is found or the query times out.

[0112] Understandably, the intermediate end can also create a local cache table. After receiving a data frame and its frame identifier, the intermediate end can store the frame identifier and the receiving end's time information in its local cache table. In one or more embodiments of this specification, the intermediate end's local cache table may include the intermediate end's time information for each data frame received by the intermediate end.

[0113] After receiving the latency statistics, the intermediate end can parse the statistics and extract the target frame identifier. Then, using the target frame identifier as the key, it can query the local cache table maintained by the intermediate end. If the intermediate end time information corresponding to the target frame identifier is found in the intermediate end local cache table, the intermediate end time information, i.e., the third time information of the target data frame, can be extracted.

[0114] Understandably, the sending end can also create a local cache table. After the sending end collects data frames and generates frame identifiers for the data frames, it can record the sending end time information based on the frame identifiers, and then store the frame identifiers and the recorded sending end time information in the sending end's local cache table. In one or more embodiments of this specification, the sending end's local cache table may include the sending end time information of each data frame sent by the sending end.

[0115] After the sending end determines the target frame identifier, it can query the first time information of the target data frame from the local cache table of the sending end based on the target frame identifier, and write the target frame identifier and the first time information into the statistical delay statistics.

[0116] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0117] See Figure 3 , Figure 3 The diagram illustrates a flowchart of a latency statistics method applied to a transmitting end according to an embodiment of this specification. From a programming perspective, the executing entity of the process can be a transmitting end device or equipment. It is understood that this method can be executed by any device, equipment, or cluster of devices with computing and processing capabilities.

[0118] like Figure 3 As shown, the process may include the following steps.

[0119] Step 302: Send a data frame and the frame identifier of the data frame to the receiving end through the first communication channel; the sending end and the receiving end are time synchronized.

[0120] Step 304: Send latency statistics information to the receiving end through the second communication channel. The latency statistics information includes the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame. The latency statistics information is used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0121] See Figure 4 , Figure 4This diagram illustrates a latency statistics method for an intermediate end in a service processing link, according to an embodiment of this specification. From a programming perspective, the execution entity of the process can be an intermediate end device or equipment in the service processing link, where the service processing link is the service link. It is understood that this method can be executed by any device, equipment, or cluster of devices with computing and processing capabilities.

[0122] like Figure 4 As shown, the process may include the following steps.

[0123] Step 402: Receive the data frame sent by the sending end and the frame identifier of the data frame through the first communication channel.

[0124] Step 404: Send the data frame and the frame identifier to the receiving end through the first communication channel; synchronize the time of the intermediate end, the sending end and the receiving end.

[0125] Step 406: Receive the first delay statistics information sent by the sending end through the second communication channel; the first delay statistics information includes the target frame identifier of the target data frame and the first time information; the first time information includes the acquisition timestamp of the target data frame.

[0126] Step 408: Send the second delay statistics to the receiving end through the second communication channel; the second delay statistics include the first delay statistics.

[0127] Step 410: The second delay statistics information is used by the receiving end to calculate the end-to-end delay from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0128] Optionally, the method may further include: The intermediate time information of the data frame is recorded in correspondence with the frame identifier; the intermediate time information includes at least one of the receiving timestamp of the intermediate end receiving the data frame, or the sending timestamp of the intermediate end sending the data frame.

[0129] Based on the target frame identifier, the third time information of the target data frame is retrieved from the intermediate time information.

[0130] The third time information of the target data frame is appended to the first delay statistics to obtain the second delay statistics.

[0131] Optionally, the second delay statistics are further used by the receiving end to calculate segmented delay based on the first time information, the second time information, and the third time information; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

[0132] Optionally, the data frame does not contain a timestamp.

[0133] See Figure 5 , Figure 5 A schematic diagram of a delay statistics system provided according to an embodiment of this specification is shown.

[0134] like Figure 5 As shown, the system may include a transmitter 502 and a receiver 504, and the transmitter 502 and the receiver 504 are time-synchronized.

[0135] The transmitting end 502 can be configured to transmit data frames and frame identifiers of the data frames through a first communication channel; and to transmit latency statistics information through a second communication channel, wherein the latency statistics information includes the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame.

[0136] The receiving end 504 can be configured to receive data frames sent by the sending end and the frame identifier of the data frames through a first communication channel; record the receiving end time information of the data frames corresponding to the frame identifier, the receiving end time information including the processing completion timestamp of the data frames at the receiving end; receive the delay statistics information through a second communication channel; query the second time information of the target data frames according to the target frame identifier; and calculate the end-to-end delay from the sending end to the receiving end according to the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

[0137] Optionally, the system may further include: an intermediate terminal; The intermediate end can be configured to receive data frames and frame identifiers of the data frames sent by the sending end through a first communication channel; record intermediate end time information of the data frames; the intermediate end time information includes at least one of the receiving timestamp of the intermediate end receiving the data frames or the sending timestamp of the intermediate end sending the data frames; and send the data frames and frame identifiers to the receiving end through the first communication channel.

[0138] The receiver receives first delay statistics sent by the sending end through a second communication channel; the first delay statistics include the target frame identifier and the first time information; the receiver queries the third time information of the target data frame from the intermediate time information according to the target frame identifier; the receiver appends the third time information of the target data frame to the delay statistics to obtain second delay statistics; and the receiver sends the second delay statistics to the receiving end.

[0139] The receiving end can also be configured as follows: The segmented delay is calculated based on the first time information, the second time information, and the third time information; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

[0140] To more clearly illustrate the delay statistics method provided in the embodiments of this specification, Figure 6 A swimlane diagram illustrating a delay statistics method provided in an embodiment of this specification. For example... Figure 6 As shown, the entities responsible for latency statistics can include a client, a backend receiving node, a business processing node, and a large model node. The client is the sending end in one or more of the preceding embodiments; the backend receiving node and the business processing node are two intermediate ends in one or more of the preceding embodiments; and the large model node is the receiving end in one or more of the preceding embodiments. Time synchronization among the client, backend receiving node, business processing node, and large model node ensures that the local timestamps recorded by each node are based on a unified time base.

[0141] The media stream transmission phase may specifically include the following steps: Step 602: The client collects data frames, generates a frame identifier for the data frame, and records the data frame collection timestamp corresponding to the frame identifier in the client's local cache table.

[0142] Step 604: Encode the data frame. When the encoding is complete, record the client encoding completion timestamp of the data frame corresponding to the frame identifier in the client's local cache table.

[0143] Step 606: Send a data frame to the backend receiving node through the first communication channel. After sending the last RTP packet of the data frame, record the client's completion timestamp of sending the last RTP packet in the client's local cache table corresponding to the frame identifier.

[0144] The first communication channel is the RTP media transmission channel.

[0145] Step 608: The backend receiving node receives the data frame and frame identifier through the first communication channel and records the backend receiving node's timestamp of the data frame.

[0146] Step 610: Decode the data frame. When decoding is complete, record the decoding completion timestamp of the backend receiving node in the local cache table of the backend receiving node corresponding to the frame identifier.

[0147] Step 612: Perform the first analysis and processing on the data frame. When the first analysis and processing is completed, record the processing completion timestamp of the backend receiving node for the data frame corresponding to the frame identifier in the local cache table of the backend receiving node.

[0148] Step 614: Send a data frame to the service processing node through the first communication channel. After sending the data frame, record the time stamp of the backend receiving node's sending completion in the local cache table of the backend receiving node corresponding to the frame identifier.

[0149] In one or more embodiments of this specification, it is assumed that after receiving a data frame, the backend receiving node needs to sequentially decode, perform a first analysis process, and forward the data frame to the next node. When the backend receiving node receives the data frame and its frame identifier, it records the backend receiving node's reception timestamp corresponding to the frame identifier in its local cache table; when the data frame is decoded, it records the backend receiving node's decoding completion timestamp corresponding to the frame identifier in its local cache table; when the first analysis process of the data frame is completed, it records the backend receiving node's processing completion timestamp corresponding to the frame identifier in its local cache table; and when the data frame is forwarded, it records the backend receiving node's transmission completion timestamp corresponding to the frame identifier in its local cache table.

[0150] In one or more embodiments of this specification, the first analysis and processing procedure may include one or more analysis and processing contents executed sequentially. In practical applications, the method of recording timestamps can be selected as needed: only the timestamp of the completion of the last analysis and processing content can be recorded, or the timestamp of the completion of each analysis and processing content can be recorded separately. For example, assuming that the analysis and processing procedure may include speech recognition, emotion analysis, keyword detection, and volume normalization sequentially, an analysis and processing timestamp can be recorded according to the system time when volume normalization is completed.

[0151] Alternatively, the corresponding analysis and processing timestamps can be recorded when speech recognition, emotion analysis, keyword detection, and volume normalization are completed. Specifically, the timestamp of each sub-step can be recorded separately, and the processing delay of each sub-step can be calculated.

[0152] Step 616: The service processing node receives the data frame and frame identifier through the first communication channel, and records the service processing node's receive timestamp of the data frame corresponding to the frame identifier in the local cache table of the service processing node.

[0153] Step 618: Perform a second analysis and processing on the data frame. When the second analysis and processing is completed, record the processing completion timestamp of the data frame corresponding to the frame identifier in the local cache table of the business processing node.

[0154] Step 620: Send a data frame to the large model node through the first communication channel. After sending the data frame, record the time stamp of the service processing node's sending completion in the local cache table corresponding to the frame identifier.

[0155] In one or more embodiments of this specification, it is assumed that after a service processing node receives a data frame, it needs to perform a second analysis process on the data frame and forward the data frame to the next node. When the service processing node receives the data frame and its frame identifier, it records the data frame reception timestamp corresponding to the frame identifier in its local cache table; when the second analysis process of the data frame is completed, it records the service processing node's processing completion timestamp corresponding to the frame identifier in its local cache table; and when the data frame forwarding is completed, it records the service processing node's transmission completion timestamp corresponding to the frame identifier in its local cache table.

[0156] Step 622: The large model node receives the data frame and frame identifier through the first communication channel, and records the large model node receiving timestamp of the data frame corresponding to the frame identifier in the local cache table of the large model node.

[0157] Step 624: Perform third analysis processing on the data frame. When the third analysis processing is completed, record the processing completion timestamp of the large model node for the data frame corresponding to the frame identifier in the local cache table of the large model node.

[0158] Assuming that after receiving a data frame, the large model node needs to perform a third analysis process on the data frame in sequence; when the large model node receives the data frame and the frame identifier, it can record the data frame reception timestamp corresponding to the frame identifier in the local cache table of the large model node; when the third analysis process of the data frame is completed, it can record the large model node analysis processing timestamp corresponding to the frame identifier in the local cache table of the large model node.

[0159] The latency statistics phase may include the following steps: Step 640: Determine the target data frame. For example, the client determines a target data frame from the sent data frames according to a preset sampling rule (e.g., one frame is sampled every 100 frames, or one frame is sampled every 1 second).

[0160] Step 642: Determine the acquisition timestamp, client encoding completion timestamp, and client transmission completion timestamp of the target data frame based on the target data frame identifier.

[0161] Step 644: Generate latency statistics, which include the acquisition timestamp of the target data frame, the client encoding completion timestamp, and the client transmission completion timestamp.

[0162] Step 646: Send latency statistics information to the backend receiving node through the second communication channel.

[0163] The second communication channel is the DataChannel control signaling channel. In practical applications, delay statistics can be triggered periodically or randomly, meaning delay statistics information can be sent periodically or randomly.

[0164] Step 648: After receiving the latency statistics information, the backend receiving node determines the backend receiving node's reception timestamp, decoding completion timestamp, processing completion timestamp, and transmission completion timestamp of the target data frame from the backend receiving node's local cache table based on the target frame identifier in the latency statistics information.

[0165] Step 650: Append the backend receiving node's reception timestamp, backend receiving node's decoding completion timestamp, backend receiving node's processing completion timestamp, and backend receiving node's transmission completion timestamp of the target data frame to the latency statistics information to obtain the first appended latency statistics information.

[0166] Step 652: Send the first additional delay statistics to the service processing node through the second communication channel.

[0167] Step 654: After receiving the latency statistics information, the service processing node determines the service processing node's reception timestamp, processing completion timestamp, and transmission completion timestamp of the target data frame from the local cache table of the service processing node based on the target frame identifier in the first appended latency statistics information.

[0168] Step 656: Append the service processing node's reception timestamp, service processing node's processing completion timestamp, and service processing node's transmission completion timestamp of the target data frame to the first appended delay statistics information to obtain the second appended delay statistics information.

[0169] Step 658: Send the second additional delay statistics to the large model node through the second communication channel.

[0170] Step 660: After receiving the delay statistics information, the large model node determines the large model node's reception timestamp and processing completion timestamp of the target data frame from the large model node's local cache table based on the target frame identifier in the second appended delay statistics information.

[0171] Step 662: Append the large model node receiving timestamp and the large model node processing completion timestamp of the target data frame to the latency statistics information to obtain the third appended latency statistics information.

[0172] Step 664: Calculate the end-to-end delay or segmented delay based on the time information in the third additional delay statistics.

[0173] In such Figure 6 In the example, the client is the acquisition and transmission end of the audio and video streams, typically deployed on terminal devices such as mobile phones, PCs, and VR headsets. The client is responsible for acquiring raw audio and video frames, generating a unique frame identifier for each frame, and recording the acquisition timestamp, encoding completion timestamp, and the transmission completion timestamp of the last RTP packet; it sends an RTP stream without SEI through the first communication channel (media channel); and it sends latency statistics information according to sampling rules through the second communication channel (such as DataChannel).

[0174] The audio / video backend receiving node (backend receiving node) is the first intermediate node in the multimodal backend link, typically an access gateway or media server. This node is responsible for receiving RTP streams sent by clients and recording the reception timestamp; decoding audio and video frames and recording the decoding completion timestamp; optionally performing first analysis processing (such as speech recognition, sentiment analysis, keyword detection, and volume normalization); and finally forwarding the audio and video frames to the service processing node and recording the transmission completion timestamp. Simultaneously, this node participates in the relay of latency statistics: it queries the locally cached timestamp based on the frame identifier in the statistics, appends it, and then forwards it.

[0175] The business processing node is the second intermediate node in the multimodal backend chain, responsible for processing related to specific business logic. This node also records timestamps for each stage, such as receiving, analyzing and processing, and sending, and appends its own timestamp after receiving statistical information before forwarding it.

[0176] The large model node is the final receiving node in the multimodal backend chain, typically an AI large model inference engine (such as a large language model or a multimodal understanding model). This node is responsible for: receiving audio and video frames forwarded by the business processing node and recording the receiving timestamp; performing third-party analysis and processing (such as large model inference and content generation) and recording the processing completion timestamp. Finally, based on the collection timestamp in the statistical information and its own processing completion timestamp, this node calculates the end-to-end latency and can further calculate the latency of each segment.

[0177] Figure 6 The provided link is a typical application of a multimodal backend link in an ultra-low latency multimodal interaction system. In practical applications, this solution can be applied to various products such as multimodal interaction products, remote medical consultation systems, online education interactive classrooms, virtual reality social platforms, intelligent customer service voice interaction systems, and metaverse real-time interaction engines. Therefore, the specific composition of the link can be flexibly adjusted according to actual business needs. For example, the link nodes may include a client and a single backend server, suitable for direct connection scenarios without intermediate processing nodes, such as ordinary video calls, where the server also acts as the receiving end, and end-to-end latency can be calculated. Alternatively, the link nodes may include a client, edge gateway, AI inference cluster, storage nodes, and a client, suitable for complex scenarios requiring edge computing, distributed storage, and final backhaul. The number of intermediate nodes can be arbitrarily increased, and the relay-style latency calculation method provided in the embodiments of this specification remains applicable.

[0178] It should be noted that the embodiments in this specification provide a non-intrusive, dual-channel separated, frame ID-based relay latency statistics scheme, which does not rely on a specific number of nodes or a specific type of processing logic. As long as there is at least one sender and one receiver (with any number of intermediate nodes), and the time of each node is synchronized, the scheme of the embodiments in this specification can be applied. Figure 6 The embodiments described herein should not be construed as limiting the scope of protection.

[0179] To more clearly illustrate the delay statistics method provided in the embodiments of this specification, Figure 7 This is a flowchart illustrating a delay statistics method provided in an embodiment of this specification. Figure 7 As shown, the entity responsible for latency statistics can include a client, an audio / video backend receiving node, a service processing node, and a large model node. The client is the sending end in one or more of the preceding embodiments; the audio / video backend receiving node and the service processing node are two intermediate ends in one or more of the preceding embodiments; and the large model node is the time synchronization mechanism for the receiving end, client, audio / video backend receiving node, service processing node, and large model node in one or more of the preceding embodiments.

[0180] All nodes (client, audio / video backend receiving node, service processing node, and large model node) synchronize time via NTP. The client acquires audio / video frames and generates frame IDs, recording the acquisition time Ts1. After encoding, it records the encoding end time Ts2, and then sends the RTP stream without SEI to the audio / video backend receiving node. When sending the last RTP packet, it records the sending time Ts3. After receiving the RTP stream, the audio / video backend receiving node records the reception time Tr1, decodes the RTP stream and records the decoding time Tr2, and then forwards the audio / video frames without SEI to the service processing node. After receiving the frames, the service processing node records the reception time Tr3, performs service processing, and then forwards the audio / video frames without SEI to the large model node. After receiving the frames, the large model node records the reception time Tr4 and the processing completion time Tr5. Meanwhile, on an independent latency statistics transmission path (second communication channel, such as DataChannel), the client selects a target data frame every N frames or randomly samples it, and sends a statistics request through DataChannel. The request contains the frame ID of the target data frame, the acquisition time, encoding time, and the completion time of the last RTP packet transmission, i.e., {F_id, Ts1, Ts2, Ts3}. Upon receiving the statistics request, the audio / video backend receiving node looks up the frame ID in its local cache to obtain the reception time Tr1 and decoding time Tr2 for that frame. It appends these two timestamps to the statistics and forwards the statistics {F_id, Ts1, Ts2, Ts3, Tr1, Tr2}. Upon receiving the statistics, the service processing node looks up the frame's reception time Tr3 in its local cache, appends it, and forwards the statistics {F_id, Ts1, Ts2, Ts3, Tr1, Tr2, Tr3}. Upon receiving the statistics, the large model node looks up the frame's processing completion time Tr5 in its local cache. Finally, the end-to-end latency is calculated as: End-to-end latency = Tr5 - Tr2. Ts1. Simultaneously, utilizing all timestamps (Ts1, Ts2, Ts3, Tr1, Tr2, Tr3, Tr4, Tr5) carried in the statistical information, the transmission delay between any two nodes or the delay of each sub-stage within a single node can be calculated in reverse. Examples include client encoding delay Ts2-Ts1, client transmission delay Ts3-Ts2, transmission delay from the client to the audio / video backend receiving node Tr1-Ts3, internal processing delay of the audio / video backend receiving node Tr2-Tr1, transmission delay from the audio / video backend receiving node to the service processing node Tr3-Tr2, and transmission delay from the service processing node to the large model node Tr4-Tr3. The entire process achieves zero intrusion into the media stream (data frames do not contain SEI timestamps) and decouples service transmission from delay measurement through dual-channel separation, enabling accurate statistics of end-to-end delay and the delay of each segment.

[0181] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0182] Figure 8 This is a schematic diagram of a delay statistics device applied to a receiving end, provided as an embodiment of this specification.

[0183] like Figure 8 As shown, the device may include: The first receiving module 802 is used to receive data frames sent by the sending end and the frame identifier of the data frames through the first communication channel.

[0184] The recording module 804 is used to record the receiving end time information of the data frame corresponding to the frame identifier. The receiving end time information includes the processing completion timestamp of the data frame at the receiving end. The sending end and the receiving end are time synchronized.

[0185] The second receiving module 806 is used to receive latency statistics information through the second communication channel; the latency statistics information includes the target frame identifier of the target data frame and the first time information; the first time information includes the acquisition timestamp of the target data frame.

[0186] The query module 808 is used to query the second time information of the target data frame from the receiving end time information based on the target frame identifier.

[0187] The first calculation module 810 is used to calculate the end-to-end delay from the sending end to the receiving end based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

[0188] Optional, Figure 8 The device shown may further include an intermediate end between the transmitting end and the receiving end; the second receiving module 806 may specifically be used for: The system receives latency statistics sent by the intermediate end, which further includes third time information of the target data frame; the third time information represents the operation time of the intermediate end for the target data frame.

[0189] Optional, Figure 8 The apparatus shown may further include: The second calculation module is used to calculate the segmented delay based on the first time information, the second time information, and the third time information; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

[0190] Optionally, the first time information also includes a sending completion timestamp from the sending end, and the second time information also includes a receiving timestamp from the receiving end; the third time information also includes an intermediate receiving timestamp and an intermediate sending timestamp; the second calculation module can specifically be used for: The first transmission delay between the sending end and the intermediate end is calculated based on the difference between the receiving timestamp of the intermediate end and the sending completion timestamp of the sending end.

[0191] The second transmission delay between the intermediate end and the receiving end is calculated based on the difference between the receiving end's receiving timestamp and the intermediate end's sending timestamp.

[0192] Optionally, the third time information includes the intermediate end receiving timestamp and the intermediate end sending timestamp; the second calculation module can specifically be used for: The first processing delay of the intermediate end for the target data frame is calculated based on the difference between the sending timestamp and the receiving timestamp of the intermediate end.

[0193] Optionally, the first time information may also include a timestamp indicating that the sending end has completed sending, such as... Figure 8 The apparatus shown may also include: The second processing delay calculation module is used to calculate the second processing delay of the sending end for the target data frame based on the difference between the sending end's completion timestamp and the acquisition timestamp.

[0194] Optionally, the sending end's transmission completion timestamp is the timestamp of the last data packet of the target data frame being sent; the first time information also includes the encoding completion timestamp of the target data frame; such as Figure 8 The apparatus shown may also include: The encoding delay calculation module is used to calculate the encoding delay of the sending end for the target data frame based on the difference between the encoding completion timestamp and the acquisition timestamp; The whole frame transmission delay calculation module is used to calculate the whole frame transmission delay of the sending end for the target data frame based on the difference between the sending end's transmission completion timestamp and the encoding completion timestamp.

[0195] Optionally, the receiving end time information is recorded in the receiving end local cache table; the query module 808 can specifically be used for: Query the time information corresponding to the target frame identifier in the local cache table.

[0196] If the local cache table of the receiving end includes the receiving end time information corresponding to the target frame identifier, then the receiving end time information corresponding to the target frame identifier shall be used as the second time information.

[0197] Optionally, the first communication channel is a media transmission channel, and the second communication channel is a control signaling channel independent of the first communication channel.

[0198] Optionally, the target data frame is a data frame selected by the sending end from the continuously transmitted data frames according to a preset sampling rule.

[0199] Optionally, the data frame does not contain a timestamp.

[0200] Optionally, the data frame is a media stream data frame.

[0201] The above is a schematic scheme of a delay statistics device according to this embodiment. It should be noted that the technical solution of this delay statistics device and the technical solution of the delay statistics method described above belong to the same concept. For details not described in detail in the technical solution of the delay statistics device, please refer to the description of the technical solution of the delay statistics method described above.

[0202] Figure 9 This is a schematic diagram of a delay statistics device applied to a transmitting end, provided as an embodiment of this specification.

[0203] like Figure 9 As shown, the device may include: A delay statistics device, applied at the transmitting end, includes: The first transmitting module 902 is used to transmit a data frame and the frame identifier of the data frame to the receiving end through a first communication channel; the transmitting end and the receiving end are time-synchronized.

[0204] The second sending module 904 is used to send delay statistics information to the receiving end through the second communication channel. The delay statistics information includes the target frame identifier of the target data frame and the first time information; the first time information includes the acquisition timestamp of the target data frame.

[0205] The latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0206] Optional, Figure 9 The apparatus shown may also include: The acquisition module is used to acquire data frames.

[0207] The frame identifier generation module is used to generate the frame identifier of the data frame.

[0208] The time recording module is used to record the acquisition timestamp and the sending completion timestamp of the data frame corresponding to the frame identifier.

[0209] The above is a schematic scheme of a delay statistics device according to this embodiment. It should be noted that the technical solution of this delay statistics device and the technical solution of the delay statistics method described above belong to the same concept. For details not described in detail in the technical solution of the delay statistics device, please refer to the description of the technical solution of the delay statistics method described above.

[0210] Figure 10 This is a schematic diagram of a latency statistics device applied to the middle end of a service processing link, provided as an embodiment of this specification.

[0211] like Figure 10 As shown, the device may include: A latency statistics device, applied at the middle end of a service processing link, includes: The first receiving module 1002 is used to receive data frames sent by the sending end and the frame identifier of the data frames through the first communication channel.

[0212] The first sending module 1004 is used to send the data frame and the frame identifier to the receiving end through the first communication channel; the intermediate end, the sending end and the receiving end are time synchronized.

[0213] The second receiving module 1006 is used to receive first delay statistics information sent by the sending end through the second communication channel; the first delay statistics information includes the target frame identifier of the target data frame and first time information; the first time information includes the acquisition timestamp of the target data frame.

[0214] The second sending module 1008 is used to send the second delay statistics to the receiving end through the second communication channel; the second delay statistics include the first delay statistics.

[0215] The second latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

[0216] Optional, Figure 10 The apparatus shown may also include: A time recording module is used to record the intermediate time information of the data frame corresponding to the frame identifier; the intermediate time information includes at least one of the receiving timestamp of the intermediate end receiving the data frame, or the sending timestamp of the intermediate end sending the data frame.

[0217] The query module is used to query the third time information of the target data frame from the intermediate time information based on the target frame identifier.

[0218] The information appending module is used to append the third time information of the target data frame to the first delay statistics information to obtain the second delay statistics information.

[0219] Optionally, the second delay statistics are further used by the receiving end to calculate segmented delay based on the first time information, the second time information, and the third time information; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

[0220] Optionally, the data frame does not contain a timestamp.

[0221] The above is a schematic scheme of a delay statistics device according to this embodiment. It should be noted that the technical solution of this delay statistics device and the technical solution of the delay statistics method described above belong to the same concept. For details not described in detail in the technical solution of the delay statistics device, please refer to the description of the technical solution of the delay statistics method described above.

[0222] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0223] Figure 11 A schematic diagram of the structure of a computing device 1100 provided according to an embodiment of this specification is shown.

[0224] The computing device 1100 includes: Memory 1110 and processor 1120; The memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions, which, when executed by the processor 1120, implement the steps of the delay statistics method.

[0225] Specifically, the components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and the database 1150 is used to store data.

[0226] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0227] In one embodiment of this specification, the above-described components of the computing device 1100 and Figure 11 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 11 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0228] The computing device 1100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1100 can also be a mobile or stationary server.

[0229] The processor 1120 implements the delay statistics method when executing the computer instructions.

[0230] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described latency statistics method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described latency statistics method.

[0231] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatuses, devices, and embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The apparatuses, devices, and methods provided in the embodiments of this specification are corresponding to each other, and therefore the apparatuses and devices also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatuses and devices will not be repeated here.

[0232] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0233] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0234] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0235] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0236] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0237] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, the invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0238] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0239] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0240] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0241] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0242] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0243] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0244] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0245] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A delay statistics method, applied at a receiving end, the method comprising: The system receives data frames sent by the sending end and the frame identifier of the data frames through the first communication channel. The receiving end time information of the data frame is recorded corresponding to the frame identifier, and the receiving end time information includes the processing completion timestamp of the data frame at the receiving end; The time of the transmitting end and the receiving end are synchronized; The delay statistics information is received through the second communication channel; the delay statistics information includes the target frame identifier of the target data frame and the first time information. The first time information includes the acquisition timestamp of the target data frame; Based on the target frame identifier, query the second time information of the target data frame from the time information of the receiving end; The end-to-end delay from the sending end to the receiving end is calculated based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

2. The method as described in claim 1, wherein an intermediate terminal is further included between the sending end and the receiving end; the step of receiving delay statistics information through the second communication channel specifically includes: The system receives latency statistics sent by the intermediate terminal, and the latency statistics further include third time information of the target data frame. The third time information indicates the operation time of the intermediate end for the target data frame; The method further includes: Calculate the segmented time delay based on the first time information, the second time information, and the third time information; The segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

3. The method as described in claim 2, wherein the first time information further includes a sending completion timestamp, the second time information further includes a receiving timestamp, and the third time information further includes an intermediate receiving timestamp and an intermediate sending timestamp; the step of calculating segmented delay based on the first time information, the second time information, and the third time information specifically includes: The first transmission delay between the sending end and the intermediate end is calculated based on the difference between the receiving timestamp of the intermediate end and the sending completion timestamp of the sending end. The second transmission delay between the intermediate end and the receiving end is calculated based on the difference between the receiving end's receiving timestamp and the intermediate end's sending timestamp.

4. The method as described in claim 2, wherein the third time information includes an intermediate end receiving timestamp and an intermediate end sending timestamp; the step of calculating the segmented delay based on the first time information, the second time information, and the third time information specifically includes: The first processing delay of the intermediate end for the target data frame is calculated based on the difference between the sending timestamp and the receiving timestamp of the intermediate end.

5. The method as described in claim 1, wherein the first time information further includes a sending completion timestamp from the sending end, and the method further includes: The second processing delay of the sending end for the target data frame is calculated based on the difference between the sending end's completion timestamp and the acquisition timestamp.

6. The method as described in claim 5, wherein the sending end's transmission completion timestamp is the timestamp of the completion of transmission of the last data packet of the target data frame; the first time information further includes the encoding completion timestamp of the target data frame; the method further includes: The encoding delay of the sending end for the target data frame is calculated based on the difference between the encoding completion timestamp and the acquisition timestamp. The whole-frame transmission delay of the sending end for the target data frame is calculated based on the difference between the sending end's transmission completion timestamp and the encoding completion timestamp.

7. The method as described in claim 1, wherein the receiving end time information is recorded in the receiving end local cache table; the step of querying the second time information of the target data frame from the receiving end time information based on the target frame identifier specifically includes: Query the local cache table for the time information corresponding to the target frame identifier; If the local cache table of the receiving end includes the receiving end time information corresponding to the target frame identifier, then the receiving end time information corresponding to the target frame identifier shall be used as the second time information.

8. The method as described in claim 1, wherein the first communication channel is a media transmission channel, and the second communication channel is a control signaling channel independent of the first communication channel.

9. The method as described in claim 1, wherein the target data frame is a data frame selected by the sending end from the continuously transmitted data frames according to a preset sampling rule.

10. The method of claim 1, wherein the data frame does not contain a timestamp.

11. The method of claim 1, wherein the data frame is a media stream data frame.

12. A delay statistics method, applied at the transmitting end, the method comprising: The data frame and its frame identifier are sent to the receiving end through the first communication channel. The time of the transmitting end and the receiving end are synchronized; The delay statistics information is sent to the receiving end through the second communication channel. The delay statistics information includes the target frame identifier of the target data frame and the first time information. The first time information includes the acquisition timestamp of the target data frame. The latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

13. The method of claim 12, further comprising: Acquire data frames; Generate the frame identifier for the data frame; The data frame acquisition timestamp and the sending end completion timestamp are recorded corresponding to the frame identifier.

14. The method of claim 12, wherein the data frame does not contain a timestamp.

15. A latency statistics method, applied to the middle end of a service processing link; the method includes: The system receives data frames sent by the sending end and the frame identifier of the data frames through the first communication channel. The data frame and the frame identifier are sent to the receiving end through the first communication channel; The intermediate end, the transmitting end, and the receiving end are time synchronized; The first delay statistics information sent by the sending end is received through the second communication channel; The first latency statistics include the target frame identifier of the target data frame and the first time information; the first time information includes the acquisition timestamp of the target data frame; The second delay statistics are sent to the receiving end through the second communication channel; the second delay statistics include the first delay statistics. The second latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

16. The method of claim 15, further comprising: The intermediate time information of the data frame is recorded corresponding to the frame identifier; The intermediate end time information includes at least one of the following: the receiving timestamp of the intermediate end receiving the data frame, or the sending timestamp of the intermediate end sending the data frame. Based on the target frame identifier, query the third time information of the target data frame from the intermediate time information; The third time information of the target data frame is appended to the first delay statistics to obtain the second delay statistics.

17. The method of claim 16, wherein the second delay statistics are further used by the receiving end to calculate segmented delay based on the first time information, the second time information, and the third time information; the second time information is the receiving end time information of the target data frame at the receiving end, representing the operation time of the receiving end for the target data frame; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two ends of the sending end, the receiving end, and the intermediate end.

18. The method of claim 15, wherein the data frame does not contain a timestamp.

19. A time delay statistics system, comprising: The sending end and the receiving end are time-synchronized; The sending end is configured to send data frames and frame identifiers of the data frames through a first communication channel; Delay statistics are transmitted through a second communication channel. The delay statistics include the target frame identifier of the target data frame and first time information. The first time information includes the acquisition timestamp of the target data frame. The receiving end is configured to receive data frames sent by the sending end and the frame identifier of the data frames through a first communication channel; The receiving end time information of the data frame is recorded corresponding to the frame identifier, and the receiving end time information includes the processing completion timestamp of the data frame at the receiving end; The delay statistics information is received through the second communication channel; Query the second time information of the target data frame based on the target frame identifier; The end-to-end delay from the sending end to the receiving end is calculated based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

20. The system of claim 19, further comprising an intermediate terminal; The intermediate end is configured to receive data frames and frame identifiers of the data frames sent by the sending end through a first communication channel; and to record the intermediate end time information of the data frames. The intermediate end time information includes at least one of the following: the receiving timestamp of the intermediate end receiving the data frame, or the sending timestamp of the intermediate end sending the data frame; the data frame and the frame identifier are sent to the receiving end through the first communication channel. The first delay statistics information sent by the sending end is received through the second communication channel; The first latency statistics include the target frame identifier and the first time information; Based on the target frame identifier, the third time information of the target data frame is queried from the intermediate time information; the third time information of the target data frame is appended to the latency statistics to obtain the second latency statistics; the second latency statistics are sent to the receiving end. The receiver is also configured to: The segmented delay is calculated based on the first time information, the second time information, and the third time information; the segmented delay includes the processing delay of any one of the sending end, the receiving end, and the intermediate end, or the transmission delay between any two of the sending end, the receiving end, and the intermediate end.

21. A delay statistics device, applied at a receiving end, comprising: The first receiving module is used to receive data frames sent by the sending end and the frame identifier of the data frames through the first communication channel; A recording module is used to record the receiving end time information of the data frame corresponding to the frame identifier, wherein the receiving end time information includes the processing completion timestamp of the data frame at the receiving end; The time of the transmitting end and the receiving end are synchronized; The second receiving module is used to receive latency statistics information through the second communication channel; the latency statistics information includes the target frame identifier of the target data frame and the first time information. The first time information includes the acquisition timestamp of the target data frame; The query module is used to query the second time information of the target data frame from the receiving end time information based on the target frame identifier; the second time information includes the target processing completion timestamp of the target data frame at the receiving end; The first calculation module is used to calculate the end-to-end delay from the sending end to the receiving end based on the target processing completion timestamp in the second time information and the acquisition timestamp in the first time information.

22. A delay statistics device, applied at a transmitting end, comprising: The first sending module is used to send a data frame and the frame identifier of the data frame to the receiving end through the first communication channel; The time of the transmitting end and the receiving end are synchronized; The second sending module is used to send delay statistics information to the receiving end through the second communication channel. The delay statistics information includes the target frame identifier of the target data frame and the first time information. The first time information includes the acquisition timestamp of the target data frame; The latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

23. A latency statistics device, applied at the middle end of a service processing link, comprising: The first receiving module is used to receive data frames sent by the sending end and the frame identifier of the data frames through the first communication channel; A first transmitting module is configured to transmit the data frame and the frame identifier to a receiving end through the first communication channel; the intermediate end, the transmitting end, and the receiving end are time-synchronized; The second receiving module is used to receive the first delay statistics information sent by the sending end through the second communication channel; The first latency statistics include the target frame identifier of the target data frame and the first time information; the first time information includes the acquisition timestamp of the target data frame; The second sending module is used to send second delay statistics to the receiving end through the second communication channel; the second delay statistics include the first delay statistics. The second latency statistics are used by the receiving end to calculate the end-to-end latency from the sending end to the receiving end based on the target processing completion timestamp of the target data frame and the acquisition timestamp; the target processing completion timestamp is the timestamp when the target data frame is processed at the receiving end.

24. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 18.

Images