Real-time transmission (RTP) header extension binding and RTP header extension for in-band delay measurement on any terminal device

Abstract

An example method includes transmitting or receiving a session description protocol (SDP) message, the session description protocol (SDP) message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, where the binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicate a time to transmit a first RTP packet including the first RTP header extension. The method comprises: transmitting, by a first device, the first RTP packet; and receiving, by the first device, a second RTP packet, the second RTP packet comprising the second RTP header extension, the second RTP header extension comprising the first timestamp, a second timestamp, and a third timestamp. The method includes determining a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

Description

Real-time Transport (RTP) header extension binding and RTP header extension for in-band delay measurement on any terminal device.

[0001] This application claims priority to U.S. Application No. 18 / 904,548, filed October 2, 2024, and U.S. Provisional Application No. 63 / 587,905, filed October 4, 2023, the entire contents of which are incorporated herein by reference. U.S. Application No. 18 / 904,548, filed October 2, 2024, claims the benefit of U.S. Provisional Application No. 63 / 587,905, filed October 4, 2023. Technical Field

[0002] This disclosure relates to the transmission of data, such as Real-Time Transport Protocol (RTP) packets. Background Technology

[0003] Applications such as extended reality (XR) can be accessed by one device from another via one or more networks. These networks can include wireless wide area networks (such as 5G networks), wireless local area networks (such as Wi-Fi networks), the Internet, etc. Therefore, an end-to-end connection between two devices can traverse different types of networks. Data packet traversal across networks can result in data packet latency. Summary of the Invention

[0004] Generally, this disclosure describes techniques for improving the accuracy of delay measurements. More specifically, this disclosure describes techniques for more accurately determining the end-to-end transmission delay and / or processing delay of data packets, such as RTP and / or Secure RTP (SRTP) packets. RTP / SRTP packets may include RTP packets and / or SRTP packets. Information included in the RTP header extension can be used to determine this delay. This disclosure describes techniques for using RTP header extensions to determine delay. This determination of delay may be important for Quality of Experience (QoE) purposes.

[0005] In one example, a method includes: a first device sending or receiving a Session Description Protocol (SDP) message, the SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; the first device sending the first RTP packet; the first device receiving a second RTP packet, the second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time the second device received the first RTP packet, and a third timestamp indicating the time the second device sent the second RTP packet; and determining a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0006] In another example, a method includes: a second device sending or receiving a Session Description Protocol (SDP) message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; the second device receiving the first RTP packet; the second device sending a second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time the second device received the first RTP packet, and a third timestamp indicating the time the second device sent the second RTP packet; and determining a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0007] In another example, a computing device includes: one or more memories for storing media data; and one or more processors communicatively coupled to the memories, the processors being configured to: send or receive Session Description Protocol (SDP) messages including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; send the first RTP packet; receive a second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time when a second device receives the first RTP packet, and a third timestamp indicating the time when the second device sends the second RTP packet; and determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0008] In another example, a computing device includes: one or more memories for storing media data; and one or more processors communicatively coupled to the memories, the processors being configured to: send or receive Session Description Protocol (SDP) messages including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; receive the first RTP packet; transmit a second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time the device received the first RTP packet, and a third timestamp indicating the time the device transmitted the second RTP packet; and determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0009] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from these descriptions and drawings, and from the claims. Attached Figure Description

[0010] Figure 1A is a block diagram illustrating an example system for implementing a technology for streaming media data over a network.

[0011] Figure 1B is a block diagram illustrating another example system for implementing a technology for streaming media data over a network.

[0012] Figure 2 is a block diagram illustrating an end-to-end XR system in which data packets traverse more than one network.

[0013] Figure 3 is a conceptual diagram illustrating an example latency in an end-to-end connection for an XR application according to one or more aspects of this disclosure.

[0014] Figure 4 is a conceptual diagram illustrating an example RTP header extension for delay measurement according to one or more aspects of this disclosure.

[0015] Figure 5 is a conceptual diagram illustrating an example of the use of a single type of RTP header extension including three timestamps according to one or more aspects of this disclosure.

[0016] Figures 6A and 6B are conceptual diagrams illustrating examples of the use of timestamps in RTP header extensions according to one or more aspects of this disclosure.

[0017] Figures 7A and 7B are conceptual diagrams illustrating examples of timestamps in the payload of RTP packets according to one or more aspects of this disclosure.

[0018] Figure 8 is a flowchart illustrating an example delay measurement technique according to one or more aspects of this disclosure.

[0019] Figure 9 is a flowchart illustrating example other delay measurement techniques according to one or more aspects of this disclosure. Detailed Implementation

[0020] Generally, this disclosure describes techniques for determining latency measurements using RTP header extensions. More specifically, this disclosure describes techniques for negotiating the binding of, for example, two RTP header extensions of different types. The header extensions may include one or more timestamps for determining latency measurements, such as round-trip time (RTT), one-way latency, and / or processing latency. Determining such latency can be important for the Quality of Experience (QoE) of an application, as navigating packets through more than one type of network can make controlling end-to-end latency more difficult. The techniques of this disclosure address this problem by informing network devices of the latency, allowing them to negotiate and implement appropriate latency on at least one of these networks to meet overall end-to-end latency requirements for QoE purposes. For example, network devices can use the determined latency to change the prioritization of application packets to meet such latency requirements.

[0021] In extended reality (XR) applications, end-to-end connectivity can include both wireless wide area networks (WANs) such as 5G networks and non-WANs (e.g., non-5G networks). Non-5G networks can include the Internet, wireless local area networks (e.g., Wi-Fi networks), etc. While the techniques disclosed herein are applicable to WANs other than 5G networks, for ease of description, 5G networks are used below as a representative example of WANs.

[0022] Figure 1A is a block diagram illustrating an example system 10 for implementing a technique for streaming media data over a network. In this example, system 10 includes a content preparation device 20, a server device 60, and a client device 40. The server device 60 may be an XR application server. The client device 40 and server device 60 are communicatively connected via a network 74, which may include a wireless wide area network, a wireless local area network, the Internet, etc. In some examples, the content preparation device 20 and server device 60 may also be connected via network 74 or another network, or may be directly communicatively connected. In some examples, the content preparation device 20 and server device 60 may include the same device.

[0023] In the example of Figure 1A, the content preparation device 20 includes an audio source 22 and a video source 24. The audio source 22 may include, for example, a microphone that generates electrical signals representing captured audio data to be encoded by the audio encoder 26. Alternatively, the audio source 22 may include: a storage medium storing previously recorded audio data; an audio data generator, such as a computerized synthesizer; or any other audio data source. The video source 24 may include: a video camera that generates video data to be encoded by the video encoder 28; a storage medium encoding previously recorded video data; a video data generation unit, such as a computer graphics source; or any other video data source. The content preparation device 20 is not necessarily communicatively connected to the server device 60 in all examples, but multimedia content may be stored on a separate medium that is read by the server device 60.

[0024] The raw audio and video data may include analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and / or video encoder 28. While a speaker is speaking, audio source 22 may obtain audio data from that speaker, and video source 24 may simultaneously obtain video data of that speaker. In other examples, audio source 22 may include a computer-readable storage medium containing stored audio data, and video source 24 may include a computer-readable storage medium containing stored video data. Thus, the techniques described in this disclosure can be applied to live, streaming, real-time audio and video data, or to archived, pre-recorded audio and video data.

[0025] An audio frame corresponding to a video frame is typically an audio frame containing audio data, which is simultaneously captured (or generated) by audio source 22 and video data captured (or generated) by video source 24 and contained within the video frame. For example, when a speaker typically generates audio data by speaking, audio source 22 captures the audio data, and video source 24 simultaneously (i.e., while audio source 22 is capturing audio data) captures the speaker's video data. Therefore, an audio frame can temporally correspond to one or more specific video frames. Thus, an audio frame corresponding to a video frame typically corresponds to the following situation: in which audio data and video data are captured simultaneously, and in this situation, the audio frame and video frame respectively include the simultaneously captured audio data and video data.

[0026] In some examples, audio encoder 26 may encode a timestamp into each encoded audio frame, where the timestamp indicates the time when the audio data for the encoded audio frame was recorded; similarly, video encoder 28 may encode a timestamp into each encoded video frame, where the timestamp indicates the time when the video data for the encoded video frame was recorded. In such examples, the audio frame corresponding to the video frame may include: an audio frame including a timestamp, and a video frame including the same timestamp. Content preparation device 20 may include an internal clock, which audio encoder 26 and / or video encoder 28 may use to generate timestamps, or audio source 22 and video source 24 may use the internal clock to associate audio and video data with timestamps, respectively. It should be noted that these timestamps may differ from the timestamps discussed herein for determining data packet delays.

[0027] In some examples, audio source 22 may transmit data corresponding to the time when the audio data was recorded to audio encoder 26, and video source 24 may transmit data corresponding to the time when the video data was recorded to video encoder 28. In some examples, audio encoder 26 may encode sequence identifiers into the encoded audio data to indicate the relative time order of the encoded audio data, rather than the absolute time when the audio data was recorded; similarly, video encoder 28 may use sequence identifiers to indicate the relative time order of the encoded video data. Similarly, in some examples, sequence identifiers may be mapped or otherwise associated with timestamps.

[0028] Audio encoder 26 typically produces encoded audio data streams, while video encoder 28 produces encoded video data streams. Each individual data stream (whether audio or video) can be referred to as an elementary stream. An elementary stream is a single, digitally decoded (possibly compressed) component. For example, a decoded video or audio portion can be an elementary stream. Elementary streams can be converted into Packed Elementary Streams (PES) before being encapsulated into a video file. Within the same representation, stream IDs can be used to distinguish PES packets belonging to one elementary stream from PES packets belonging to other elementary streams. The basic unit of data in an elementary stream is a Packed Elementary Stream (PES) packet. Therefore, decoded video data typically corresponds to an elementary video stream. Similarly, audio data corresponds to one or more corresponding elementary streams.

[0029] Many video decoding standards, such as ITU-T H.264 / AVC and High-Efficiency Video Decoding (HEVC), define the syntax, semantics, and decoding process for error-free bitstreams, each conforming to a profile or level. Video decoding standards typically do not specify an encoder, but the encoder's task is to ensure that the generated bitstream conforms to the standard for the decoder. In the context of video decoding standards, a "profile" corresponds to an algorithm, feature, or tool, and a subset of constraints imposed on that algorithm, feature, or tool. For example, as defined by the H.264 standard, a "profile" is, for instance, a subset of the entire bitstream syntax specified by the H.264 standard. A "level" corresponds to limitations on decoder resource consumption, such as decoder memory and computation, which are related to image resolution, bit rate, and block processing rate. Profiles can be signaled using the `profile_idc` (profile indicator) value, and levels can be signaled using the `level_idc` (level indicator) value.

[0030] For example, the H.264 standard recognizes that, within the limits imposed by the syntax of a given profile, the performance requirements for encoders and decoders can still vary significantly depending on the values ​​taken by the syntax elements in the bitstream (such as the specified size of the decoded image). The H.264 standard further recognizes that, in many applications, implementing a decoder capable of handling all assumptions about the syntax within a particular profile is neither practical nor economical. Therefore, the H.264 standard defines a “hierarchy” as a specified set of constraints imposed on the values ​​of syntax elements in the bitstream. These constraints can be simple restrictions on the values. Alternatively, these constraints can take the form of constraints on arithmetic combinations of values ​​(e.g., image width multiplied by image height multiplied by the number of images decoded per second). The H.264 standard also provides that individual implementations can support different hierarchies for each supported profile.

[0031] A decoder conforming to a profile typically supports all features defined in that profile. For example, B-picture decoding, as a decoding feature, is not supported in the baseline H.264 / AVC profile but is supported in other H.264 / AVC profiles. A decoder conforming to a hierarchy should be able to decode any bitstream that does not require exceeding the limits defined in that hierarchy. The definitions of profiles and hierarchies can contribute to interpretability. For example, during video transmission, a pair of profile and hierarchy definitions can be negotiated and agreed upon for the entire transmission session. More specifically, in H.264 / AVC, a hierarchy can define constraints on the following: the number of macroblocks to be processed, the size of the post-decoded picture buffer (DPB), the size of the post-decoded picture buffer (CPB), the range of vertical motion vectors, the maximum number of motion vectors per two consecutive MBs, and whether a B-block can have sub-macroblock partitions smaller than 8×8 pixels. In this way, the decoder can determine whether it is able to properly decode the bitstream.

[0032] In the example of Figure 1A, the encapsulation unit 30 of the content preparation device 20 receives a base stream of video data including decoded data from the video encoder 28 and a base stream of audio data including decoded data from the audio encoder 26. In some examples, both the video encoder 28 and the audio encoder 26 may include packers for forming PES packets based on the encoded data. In other examples, both the video encoder 28 and the audio encoder 26 may interface with corresponding packers for forming PES packets based on the encoded data. In other examples, the encapsulation unit 30 may include packers for forming PES packets based on the encoded audio and video data.

[0033] Video encoder 28 can encode video data of multimedia content in various ways to produce different representations of the multimedia content at various bit rates and utilizing various characteristics such as pixel resolution, frame rate, compliance with various decoding standards, compliance with various profiles and / or profile levels used for various decoding standards, representations with one or more views (e.g., for two-dimensional or three-dimensional playback), or other such characteristics. As used in this disclosure, a representation may include one of the following: audio data, video data, text data (e.g., for closed captions), or other such data. A representation may include a primary stream, such as an audio primary stream or a video primary stream. Each PES packet may include a stream_id, which identifies the primary stream to which the PES packet belongs. Encapsulation unit 30 is responsible for assembling the primary streams into video files (e.g., segments) of various representations.

[0034] Encapsulation unit 30 receives PES packets for representing the basic stream from audio encoder 26 and video encoder 28, and forms corresponding Network Abstraction Layer (NAL) units based on the PES packets. Decoded video segments can be organized into NAL units that provide a “network-friendly” video representation for addressing applications such as video telephony, storage, broadcasting, or streaming. NAL units can be classified as Video Decoding Layer (VCL) NAL units and non-VCL NAL units. VCL units may contain the core compression engine and may include block, macroblock, and / or slice-level data. Other NAL units may be non-VCL NAL units. In some examples, a picture of the decoded data in a time instance (typically presented as a picture of the main decoded data) may be included in an access unit, which may include one or more NAL units.

[0035] Non-VCL NAL units can include parameter set NAL units and SEI NAL units, etc. Parameter sets can contain sequence-level header information (in the Sequence Parameter Set (SPS)) and picture-level header information that changes very little (in the Picture Parameter Set (PPS)). Using parameter sets (e.g., PPS and SPS), it is not necessary to repeat information that changes very little for each sequence or picture; therefore, decoding efficiency can be improved. Furthermore, the use of parameter sets allows for out-of-band transmission of important header information, thus avoiding redundant transmissions required for error recovery. In an example of out-of-band transmission, parameter set NAL units can be transmitted on a different channel than other NAL units (such as SEI NAL units).

[0036] Supplemental Enhancement Information (SEI) can contain information that is not essential for decoding image samples from VCL NAL units but can assist in processes related to decoding, display, error recovery, and other purposes. SEI messages can be included in non-VCL NAL units. SEI messages are a specification part of some standards and are therefore not always mandatory for specific implementations of standards-compliant decoders. SEI messages can be sequence-level SEI messages or image-level SEI messages. Some sequence-level information can be included in SEI messages, such as the scalability information SEI message in examples of Scalable Video Decoding (SVC) and the view scalability information SEI message in Multi-View Video Decoding (MVC). These example SEI messages can convey information about, for example, the extraction and characteristics of operation points. Additionally, encapsulation unit 30 can form manifest files, such as Media Presentation Descriptors (MPDs) that describe the characteristics of the representation. Encapsulation unit 30 can format MPDs according to Extensible Markup Language (XML).

[0037] Encapsulation unit 30 can provide data for one or more representations of multimedia content, along with a manifest file (e.g., MPD), to output interface 32. Output interface 32 may include a network interface or an interface for writing to storage media, such as a Universal Serial Bus (USB) interface, a CD or DVD writer or burner, an interface to magnetic or flash storage media, or other interfaces for storing or transmitting media data. Encapsulation unit 30 can provide data for each representation of the multimedia content to output interface 32, wherein the output interface can transmit data to server device 60 via a network or storage medium. In the example of FIG. 1A, server device 60 includes storage medium 62 storing various multimedia content 64, wherein each multimedia content includes a corresponding manifest file 66 and one or more representations 68A to 68N (representation 68). In some examples, output interface 32 can also transmit data directly to network 74.

[0038] In some examples, representation 68 can be separated into adaptation sets. That is, each subset of representation 68 may include a corresponding set of common characteristics, such as codecs, profiles and hierarchies, resolution, number of views, file format of segments, text type information (which may identify the language or other characteristics of the text to be displayed using the representation and / or the audio data to be decoded and presented by, for example, a speaker), camera angle information (which may describe the camera angle or real-world camera view of the scene for the representation in the adaptation set), and hierarchical information describing the suitability of the content for a particular audience, etc.

[0039] Manifest file 66 may include a subset of representations 68 corresponding to a particular adaptation set, as well as data indicating common characteristics of the adaptation set. Manifest file 66 may also include data indicating individual characteristics (such as bit rate) of individual representations for each adaptation set. In this way, the adaptation set can provide simplified network bandwidth adaptation. Sub-elements of the adaptation set element in manifest file 66 can be used to indicate representations within the adaptation set.

[0040] Server device 60 includes request processing unit 70 and network interface 72. In some examples, server device 60 may include multiple network interfaces. Furthermore, any or all of the features of server device 60 may be implemented on other devices in the content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, intermediate devices in the content delivery network may cache data of multimedia content 64 and include components that generally conform to the components of server device 60. Generally, network interface 72 is configured to transmit and receive data via network 74.

[0041] The request processing unit 70 is configured to receive network requests for data for the storage medium 62 from a client device (such as client device 40). In some examples, the request processing unit 70 may receive network requests from client device 40 in the form of RTP / SRTP packets, and may deliver content such as XR application content to client device 40 in the form of RTP / SRTP packets.

[0042] Additionally or alternatively, request processing unit 70 may implement Hypertext Transfer Protocol (HTTP) version 1.1, as described by R. Fielding et al. in RFC 2616 “Hypertext Transfer Protocol – HTTP / 1.1” (Network Working Group, Internet Engineering Task Force (IETF), June 1999). That is, request processing unit 70 may be configured to receive HTTP GET or partial GET requests and, in response to such requests, provide data for multimedia content 64. The request may, for example, use a Uniform Resource Locator (URL) for a segment to specify a segment representing one of 68. In some examples, the request may also specify one or more byte ranges of a segment, thus including partial GET requests. Request processing unit 70 may also be configured to service HTTP HEAD requests to provide header data for a segment representing one of 68. In any case, request processing unit 70 may be configured to process a request to provide the requested data to a requesting device (such as client device 40).

[0043] Additionally or alternatively, request processing unit 70 may be configured to deliver media data via a broadcast or multicast protocol, such as Evolved Multimedia Broadcast Multicast Service (eMBMS). Content preparation device 20 may create DASH segments and / or subsegments in a manner substantially the same as described, but server device 60 may use eMBMS or another broadcast or multicast network transport protocol to deliver these segments or subsegments. For example, request processing unit 70 may be configured to receive multicast group join requests from client device 40. That is, server device 60 may advertise the Internet Protocol (IP) address associated with the multicast group to client devices (including client device 40) associated with specific media content (e.g., broadcast of a live event). Client device 40 may then submit a request to join the multicast group. The request may be propagated throughout network 74 (e.g., routers constituting network 74), causing routers to direct traffic destined for the IP address associated with the multicast group to subscribing client devices (such as client device 40).

[0044] As illustrated in the example of Figure 1A, multimedia content 64 includes a manifest file 66, which may correspond to a Media Presentation Description (MPD). The manifest file 66 may contain descriptions of different alternative representations 68 (e.g., video services with different qualities), and the descriptions may include, for example, codec information, profile values, layer values, bitrates, and other descriptive characteristics of representation 68. Client device 40 may obtain the MPD of the media presentation to determine how to access segments of representation 68.

[0045] Specifically, the acquisition unit 52 can acquire configuration data (not shown) of the client device 40 to determine the decoding capability of the video decoder 48 and the rendering capability of the video output 44. The configuration data may also include any or all of the following: the language preference selected by the user of the client device 40, one or more camera angles corresponding to the depth preference set by the user of the client device 40, and / or the grading preference selected by the user of the client device 40. The retrieval unit 52 may include, for example, a web browser or media client configured to submit HTTP GET and partial GET requests. The retrieval unit 52 may correspond to software instructions executed by one or more processors or processing units (not shown) of the client device 40. In some examples, all or part of the functionality described for the retrieval unit 52 may be implemented in hardware, or a combination of hardware, software, and / or firmware, wherein the necessary hardware may be provided to execute instructions for the software or firmware.

[0046] The retrieval unit 52 can compare the decoding and rendering capabilities of the client device 40 with the characteristics of the representation 68 indicated by the information in the manifest file 66. The acquisition unit 52 can initially acquire at least a portion of the manifest file 66 to determine the characteristics of the representation 68. For example, the retrieval unit 52 can request a portion of the manifest file 66 describing the characteristics of one or more adapter sets. The retrieval unit 52 can select a subset (e.g., an adapter set) of representations 68 having characteristics that can be satisfied by the decoding and rendering capabilities of the client device 40. The acquisition unit 52 can then determine the bit rate for the representations in the adapter set, determine the amount of currently available network bandwidth, and acquire a segment from one of the representations having a bit rate that the network bandwidth can satisfy.

[0047] Generally speaking, a higher bit rate representation can produce higher quality video playback, while a lower bit rate representation can provide sufficient quality video playback when available network bandwidth decreases. Therefore, when available network bandwidth is relatively high, the acquisition unit 52 can acquire data from a relatively high bit rate representation, and when available network bandwidth is low, the acquisition unit 52 can acquire data from a relatively low bit rate representation. In this way, the client device 40 can stream multimedia data through the network 74, while also adapting to the changing network bandwidth availability of the network 74.

[0048] Additionally or alternatively, the retrieval unit 52 may be configured to receive data according to broadcast or multicast network protocols (such as eMBMS or IP multicast). In such an example, the retrieval unit 52 may submit a request to join a multicast network group associated with specific media content. After joining the multicast group, the retrieval unit 52 may receive data from the multicast group without issuing further requests to the server device 60 or the content preparation device 20. When the multicast group data is no longer needed, the retrieval unit 52 may submit a request to leave the multicast group, for example, to stop playback or change the channel to a different multicast group.

[0049] Network interface 54 can receive data from the selected represented segment and provide that data to retrieval unit 52, which in turn can provide the segment to decapsulation unit 50. Decapsulation unit 50 can decapsulate the elements of the video file into a PES stream, unpack the PES stream to retrieve encoded data, and transmit the encoded data to audio decoder 46 or video decoder 48, depending on whether the encoded data is part of an audio stream or a video stream (e.g., as indicated by the PES packet header of the stream). Audio decoder 46 decodes the encoded audio data and transmits the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and transmits the decoded video data (which may include multiple views of the stream) to video output 44.

[0050] The video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, retrieval unit 52, and decapsulation unit 50 can all be implemented as any of a variety of suitable processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of the video encoder 28 and video decoder 48 can be included in one or more encoders or decoders, and either the video encoder or the video decoder can be integrated as part of a combined video encoder / decoder (CODEC). Similarly, each of the audio encoder 26 and audio decoder 46 can be included in one or more encoders or decoders, and either the audio encoder or the audio decoder can be integrated as part of a combined CODEC. The apparatus including the video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, retrieval unit 52, and / or decapsulation unit 50 can include integrated circuits, microprocessors, and / or wireless communication devices, such as cellular phones.

[0051] Client device 40, server device 60, and / or content preparation device 20 may be configured to operate according to the techniques of this disclosure. For illustrative purposes, these techniques are described with respect to client device 40 and server device 60. However, it should be understood that content preparation device 20 may also be configured to perform these techniques as an alternative to (or other than) server device 60.

[0052] The encapsulation unit 30 can form NAL units, which include a header identifying the program to which the NAL unit belongs and a payload, such as audio data, video data, or data describing the transport or program stream corresponding to the NAL unit. For example, in H.264 / AVC, a NAL unit includes a 1-byte header and a payload of varying size. NAL units whose payloads include video data can include video data at various granular levels. For example, a NAL unit can include video data blocks, multiple blocks, video data slices, or entire frames of video data. The encapsulation unit 30 can receive encoded video data in PES packet format with elementary streams from the video encoder 28. The encapsulation unit 30 can associate each elementary stream with a corresponding program.

[0053] The encapsulation unit 30 can also assemble access units based on multiple NAL units. Generally, an access unit may include one or more NAL units representing a video data frame, and the corresponding audio data (when the audio data is available). Access units typically include all NAL units for a single output time instance, such as all audio and video data for a single time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, specific frames for all views with the same access unit (same time instance) can be rendered simultaneously. In one example, an access unit may include a decoded image within a time instance, which can be rendered as the primary decoded image.

[0054] Therefore, an access unit can include all audio and video frames of a common time instance, such as all views corresponding to time X. This disclosure also refers to the encoded picture of a particular view as a "view component." That is, a view component can include pictures (or frames) encoded for a particular view at a particular time. Therefore, an access unit can be defined as including all view components of a common time instance. The decoding order of the access units need not be the same as the output or display order.

[0055] Media presentation may include a Media Presentation Description (MPD), which may contain descriptions of different alternative representations (e.g., video services with different qualities), and this description may include, for example, codec information, profile values, and hierarchical values. An MPD is an example of a manifest file (such as manifest file 66). Client device 40 may acquire the MPD of the media presentation to determine how to access movie clips in various representations. Movie clips may reside in a movie clip box (moof box) of the video file.

[0056] The manifest file 66 (which may include, for example, an MPD) can announce the availability of segments in representation 68. Specifically, the MPD may include information indicating the clock time when a first segment of one of the representations in 68 becomes available, and information indicating the duration of segments within representation 68. Thus, the retrieval unit 52 of the client device 40 can determine when each segment is available based on the start time and duration of segments preceding a specific segment.

[0057] After the encapsulation unit 30 has assembled the NAL units and / or access units into a video file based on the received data, the encapsulation unit 30 passes the video file to the output interface 32 for output. In some examples, the encapsulation unit 30 may store the video file locally or transmit the video file to a remote server via the output interface 32, instead of transmitting the video file directly to the client device 40. The output interface 32 may include, for example, a transmitter, a transceiver, a device for writing data to a computer-readable medium (such as, for example, an optical drive, a magnetic media drive (e.g., a floppy disk drive)), a universal serial bus (USB) port, a network interface, or other output interface. The output interface 32 outputs the video file to a computer-readable medium, such as, for example, a transmitting signal, a magnetic medium, an optical medium, a memory, a flash drive, or other computer-readable medium.

[0058] Network interface 54 can receive NAL units or access units via network 74 and provide NAL units or access units to decapsulation unit 50 via retrieval unit 52. Decapsulation unit 50 can decapsulate the elements of the video file into a PES stream, unpack the PES stream to retrieve encoded data, and transmit the encoded data to audio decoder 46 or video decoder 48, depending on whether the encoded data is part of an audio stream or a video stream (e.g., as indicated by the PES packet header of the stream). Audio decoder 46 decodes the encoded audio data and transmits the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and transmits the decoded video data (which may include multiple views of the stream) to video output 44.

[0059] For illustrative purposes, the example in Figure 1A illustrates the use of streaming based on RTP, DASH, and HTTP. However, it should be understood that other types of protocols can be used to transmit media data. For example, the request processing unit 70 and the retrieval unit 52 can be configured to operate according to protocols such as Real-time Streaming Protocol (RTSP) and use supported protocols such as Session Description Protocol (SDP) or Session Initiation Protocol (SIP).

[0060] Figure 1B is a block diagram illustrating another example system for implementing techniques for streaming media data over a network. Figure 1B is similar to the example in Figure 1A, but instead of a client device and a server device, Figure 1B includes two terminal devices. For example, each terminal device 80A and 80B in system 10B can be configured to consume content from the other terminal device 80A and 80B and provide content to the other terminal device 80A and 80B. The system of Figure 1B can implement the techniques disclosed herein.

[0061] Figure 2 is a block diagram illustrating an end-to-end XR system in which data packets traverse more than one network. XR device 102, which may be an example of client device 40 or a portion thereof, may be coupled to network 104. In some examples, XR device 102 may include XR glasses or XR headsets configured to deliver XR content to a user. In some examples, network 104 may include a wireless local area network, such as a Wi-Fi network. In some examples, network 104 may include multiple networks, such as a Wi-Fi network cascaded with Ethernet.

[0062] Mobile device 106, which may be an example of client device 40 or a portion thereof, may be coupled to network 104 and network 108. Network 108 may include a wireless wide area network, such as a 5G network.

[0063] User Plane Function (UPF) 110 can be coupled to network 108 and network 112. Network 112 may include the Internet. In some examples, network 112 may also include one or more local networks. Edge Application Server (EAS) 114 (which may be an example of server device 60) can be coupled to network 112 and content preparation device 20 (not shown in Figure 2). Therefore, the end-to-end connection between XR device 102 and EAS 114 can traverse multiple networks, such as network 104, network 108, and network 112. These networks may be of different types, with different latency attributes and different mechanisms for determining latency, for example, for Quality of Service (QoS) or QoE purposes. Therefore, techniques for determining end-to-end latency may be desired.

[0064] In some examples, network 108 may be coupled to QoS device 116. QoS device can be any computing device capable of calculating latency or comparing latency to latency thresholds for QoS or QoE purposes.

[0065] Computing devices (such as XR device 102 and EAS 114) may exchange one or more SDP messages 120 to negotiate the binding between RTP packets such that a first timestamp transmitted in a first type of RTP header extension in a first RTP packet transmitted by one of these computing devices is the same as a first timestamp transmitted in a second type of RTP header extension in a second RTP packet transmitted by another of these computing devices.

[0066] For example, XR device 102 may send or receive (in one or more SDP messages 120) an SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension. XR device 102 may send the first RTP packet. XR device 102 may receive a second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time the second device received the first RTP packet, and a third timestamp indicating the time the second device sent the second RTP packet. XR device 102 may determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0067] For example, EAS 114 can send or receive an SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session. This binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time when a first RTP packet including the first RTP header extension was transmitted. EAS 114 can receive the first RTP packet. EAS 114 can send a second RTP packet including the second RTP header extension, which includes the first timestamp, a second timestamp indicating the time when a second device received the first RTP packet, and a third timestamp indicating the time when the second device transmitted the second RTP packet. EAS 114 can determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0068] Figure 3 is a conceptual diagram illustrating an example latency in an end-to-end connection for an XR application according to one or more aspects of this disclosure. In the example of Figure 3, the wireless wide area network (such as network 108) can be a 3GPP network (e.g., a 5G network), represented in Figure 3 as gNodeB (gNB) 208. As can be seen in Figure 3, the end-to-end latency... This can be equivalent to the latency in a 5G network between a phone 206 (which could be an example of a mobile device 106) and a UPF 210 (which could be an example of a UPF 110). In addition, there is a delay from the augmented reality (AR) glasses 202 (which is an example of XR device 102) to the phone 206. And the latency from UPF 210 to EAS 214 Therefore, the latency D in non-3GPP networks n (This may include latency in 5G networks) Other delays) can be equal to delays + .

[0069] For example, latency measurement can be important for XR applications. Latency matching by 3GPP networks (e.g., 5G networks) can include matching end-to-end connections involving both 3GPP and non-3GPP networks (such as the Internet, Wi-Fi, etc.). End-to-end latency measurement allows for the derivation of latency D within non-3GPP networks. n .

[0070] For example, 3GPP networks (e.g., gNB 208) can be based on Dc = Adjusting the delay DC to achieve the target end-to-end delay When performing latency monitoring, such as for XR applications, the measured latency should represent the latency experienced by data packets. For example, data packets undergoing the same QoS or QoE processing, having similar packet sizes, etc. Therefore, it may be desirable to piggyback latency measurements within the data packets (e.g., via RTP header extension), such as using in-band latency measurements.

[0071] Non-5G network latency can be estimated via end-to-end latency measurement. Measurement of 5G network latency is also possible. Existing solutions or technologies, such as those specified in Clause 5.33.3 of TS23.501. Measurement packets (on both 5G and non-5G networks) should represent XR data packets: for example, having the same QoS or QoE processing, similar packet sizes, etc.

[0072] However, the QoS or QoE processing received from measurement packets may differ from that received from the data packets themselves, for example, when using out-of-band measurement techniques. For instance, in 5G networks, measurement messages and data packets may use different protocols (e.g., Internet Control Message Protocol (ICMP) messages (echo or echo acknowledgment)). For example, a measurement message may have protocol number 1, while a data packet (RTP / UDP) may have protocol number 17. Measurement messages and data packets may have different IP 5-tuples (IP source address, IP destination address, source port number, destination port number, protocol number). Measurement messages and data packets may be mapped to different QoS flows and may receive different QoS processing in 5G networks. On the Internet, the Differentiated Service Code Point (DSCP) values ​​in the IP packet header may differ between measurement packets and data packets. In Wi-Fi networks (e.g., Network 104), measurement packets and data packets may be mapped to different access classes. Furthermore, differences in packet size between measurement packets and data packets can affect the accuracy of latency measurements, especially for low-rate links.

[0073] Figure 4 is a conceptual diagram illustrating an example RTP header extension for delay measurement according to one or more aspects of this disclosure. T1 may be a timestamp indicating the time when AR glasses 202 transmits RTP packet 320. T2 may be a timestamp indicating the time when EAS 214 receives RTP packet 320. T3 may be a timestamp indicating the time when EAS 214 transmits RTP packet 322. T4 may be the time when AR glasses 202 receives RTP packet 322. By using RTP header extensions, such as those in Figure 4, delay measurement can be performed as follows. The one-way delay from AR glasses 202 to EAS 214 can be determined as T2–T1. The one-way delay in the opposite direction can be determined as T4–T3. The round-trip time (RTT) can be determined as T4–T1–(T3–T2). In this example, AR glasses 202 can determine the one-way delay from AR glasses 202 to EAS 214, the one-way delay in the opposite direction, and the RTT based on the time and / or timestamps available to AR glasses 202. EAS 214 can determine the one-way latency from AR glasses 202 to EAS 214 and / or the processing latency of EAS 214 based on the time and / or timestamp available in EAS 214.

[0074] The RTP header extension 330 of RTP packet 320 may be different in size from the RTP header extension 332 of RTP packet 322, because RTP header extension 330 includes one timestamp (e.g., T1), while RTP header extension 332 includes three timestamps (e.g., T1, T2, and T3).

[0075] Fields in RTP header extension 332 (e.g., for T1 and T2) may depend on RTP header extension 330. For example, T1 in RTP header extension 332 may be the same as T1 in RTP header extension 330 (e.g., the time when AR glasses 202 transmits RTP packet 320), and T2 may represent the time when EAS 214 receives RTP packet 320. Therefore, this dependency (or binding) between RTP header extension 330 and RTP header extension 332 should be agreed upon between the transmitter (e.g., AR glasses 202) and the receiver (e.g., EAS 214).

[0076] For example, EAS 214 should know the meaning of T1 and T2 because EAS 214 can insert T1 and T2 into the RTP header extension 332. For example, if EAS 214 does not know the meaning of T1 and T2, EAS 214 may not include the appropriate values ​​of T1 and T2 in the RTP header extension 332 of packet 322, which may lead to, for example, an improper determination of latency by AR glasses 202. In some examples, AR glasses 202, EAS 214, or one or more other devices in the network (e.g., gNB 208) may use Session Data Protocol (SDP) to bind the two header extensions (HE 1 and HE 2) in such a way that T1 is the timestamp carried in the RTP HE of type 1 (carrying only one timestamp) (e.g., RTP header extension 330) of RTP packet (RTP packet 320) and / or T2 is the arrival time of the RTP HE of type 1 (e.g., RTP header extension 330) of RTP packet 320.

[0077] For example, AR glasses 202 can send or receive SDP messages including binding information that associates a first type of RTP header extension and a second type of RTP header extension in an RTP session. This binding information may indicate a first timestamp in the first type of RTP header extension and a first timestamp in the second type of RTP header extension, both indicating the time when a first RTP packet including the first RTP header extension was sent. AR glasses 202 can send the first RTP packet. AR glasses 202 can receive a second RTP packet including the second type of RTP header extension, which includes the first timestamp, a second timestamp indicating the time when EAS 214 received the first RTP packet, and a third timestamp indicating the time when the EAS 214 device sent the second RTP packet. AR glasses 202 can determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0078] For example, EAS 214 may send or receive an SDP message including binding information that associates a first type of RTP header extension and a second type of RTP header extension of an RTP session. This binding information may indicate a first timestamp in the first type of RTP header extension and a first timestamp in the second type of RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension. EAS 214 may receive the first RTP packet. EAS 214 may send a second RTP packet including the second type of RTP header extension, which includes the first timestamp, a second timestamp indicating the time EAS 214 received the first RTP packet, and a third timestamp indicating the time EAS 214 transmitted the second RTP packet. EAS 214 may determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0079] For example, terminal devices in an RTP session (e.g., AR glasses 202 and EAS 214) can (via SDP) negotiate the binding of timestamps between two types of RTP header extensions. For instance, an RTP header extension with a single timestamp T1 (such as RTP header extension 330) can be referred to as a first-type RTP header extension, and an RTP header extension with three timestamps T1, T2, and T3 (such as RTP header extension 332) can be referred to as a second-type RTP header extension. Each RTP header extension can have its own identifier (ID). In some examples, RTP header extensions can be defined in single-byte format (e.g., short) or double-byte format (e.g., long). In some examples, multiple instances of one type of RTP header extension may exist during an RTP session, and this identifier helps computing devices distinguish different RTP packets for binding purposes.

[0080] In some examples, fields in a Type 1 RTP header extension can be mapped to fields in a Type 2 RTP header extension. For example, for the latency measurements discussed herein, a T1 in a Type 1 RTP header extension can be mapped to a T1 in a Type 2 RTP header extension. Therefore, devices transmitting or receiving specific RTP packets with either a Type 1 or Type 2 RTP header extension can associate specific T1s with each other. In this way, devices can use the associated T1 as part of determining one or more latency measurements.

[0081] In some examples, the definition of fields in the second type of RTP header extension may be based on events associated with the first type of RTP header extension. For example, the definition of T2 in the second type of RTP header extension may be the arrival time of an RTP packet (e.g., RTP packet 320) that includes the first type of RTP header extension.

[0082] In some examples, the association between RTP packets carrying RTP header extensions can indicate cause and effect. For example, in latency measurement, this indication could be processing latency: for instance, the last RTP packet carrying an image transmitted by the user equipment (UE) to a cloud server for image segmentation is associated with the first RTP packet carrying the segmentation result (e.g., RTP packet 320).

[0083] For example, the Enhanced Backsnorr Normal Form (ABNF) syntax in SDP may include:

[0084] extmap-attr="a=extmap:" header-ext-ID-1[" / " direction] SPwww.webrtc.org / experiments / rtp-hdrext / abs-send-time

[0085] extmap-attr="a=extmap:" header-ext-ID-2[" / " direction] SP urn:3gpp:delay-measurement-1-timestamps:rel-18 SP (format SP dependent-extmap-ID)

[0086] header-ext-ID-1 = 1 3DIGIT

[0087] header-ext-ID-2 = 1 3DIGIT

[0088] direction = "sendonly" / "recvonly" / "sendrecv" / "inactive"

[0089] format = "short" / "long"

[0090] dependent-extmap-ID = header-ext-ID-1

[0091] In some examples, a single type of RTP header extension can be used for latency measurement. For example, an RTP header extension may include three timestamps, such as those discussed in U.S. Patent Application No. 18 / 782,823, filed July 24, 2024, which is incorporated herein by reference. For example, an RTP header extension may include timestamps T1, T2, and T3. In another example, an RTP header extension may include two timestamps and a time difference, such as T1, T2, and ΔT = T3 – T1. In yet another example, an RTP header extension may include one timestamp and two time differences, such as T1, T2 - T1, and T3 - T2. In some examples, the timestamp may occupy 24 bits of the 32-bit Network Time Protocol (NTP) short format, such as X least significant bits (LSBs) for the "seconds" portion and 24 - X (24 minus X) most significant bits (MSBs) for the "fractions" portion, where X may be equal to 12, 8, 6, 4, 3, 2, or 1. In some examples, the time difference can take up fewer bits.

[0092] In some examples, the interpretation of the timestamps may differ from examples where there is more than one type of RTP header extension. For example, T1 may be interpreted as the time when an RTP packet carrying the same type of RTP header extension in the opposite direction is sent. T2 may be interpreted as the time when an RTP packet carrying the same type of RTP header extension in the opposite direction is received. T3 may be interpreted similarly as the time when an RTP packet carrying that RTP header extension is sent. In some examples, if there is no previous packet received in the opposite direction, the device may set T1 and T2 to special (e.g., predetermined) values, such as 0.

[0093] In some examples, the T1 and T2 fields in the RTP header extension of the second RTP packet (e.g., RTP packet 322) represent the transmission and reception times of the first RTP packet (e.g., RTP packet 320) that carries the RTP header extension and causes the generation of the second RTP data packet. For example, the first RTP packet may carry an image transmitted by the UE to a cloud server for image segmentation, and the second RTP packet may carry the image segmentation result.

[0094] Using the single RTP header extension described above, only a single RTP header extension needs to be defined, and the binding described in this paper using the two types of RTP header extensions is not required. Additionally, by using a single RTP header extension, both end devices can derive two one-way delays, round-trip time (RTT), and processing delays (T3-T2).

[0095] Figure 5 is a conceptual diagram illustrating an example of the use of a single type of RTP header extension including three timestamps according to one or more aspects of this disclosure. As can be seen, both the AR glasses 202 and the EAS 214 can determine the one-way latency, RTT, and processing latency.

[0096] In the example of Figure 5, t1 represents the time when EAS 214 transmits RTP packet 1 350, t2 represents the time when AR glasses 202 receives RTP packet 1 350, t3 represents the time when AR glasses 202 transmits RTP packet 2 352, t4 represents the time when EAS 214 receives RTP packet 2 352, t5 represents the time when EAS 214 transmits RTP packet 3 354, and t6 represents the time when AR glasses 202 receives RTP packet 3 354. AR glasses 202 can transmit RTP packet 2 352 in response to receiving RTP packet 1 350, and EAS 214 can transmit RTP packet 3 354 in response to receiving RTP packet 2 352.

[0097] For example, EAS 214 may include the value of T1 as 0, the value of T2 as 0, and the value of T3 as t1 (the time EAS 214 sends RTP packet 1 350) in the RTP header extension of RTP packet 1 350. AR glasses 202 may include the value of T1 as t1 (e.g., from the RTP header extension of RTP packet 1 350), the value of T2 as t2 (the time AR glasses 202 receives RTP packet 1 350), and the value of T3 as t3 (the time AR glasses 202 sends RTP packet 2 352) in the RTP header extension of RTP packet 2 352. EAS 214 may include the value of T1 as t3 (e.g., the RTP header extension from RTP packet 2352) in the RTP header extension of RTP packet 3354, include the value of T2 as t4 (the time when EAS 214 receives RTP packet 1350), and include the value of T3 as t5 (the time when EAS 214 sends RTP packet 3354).

[0098] In this manner, in the example of Figure 5, the delay of AR glasses 202 can be determined or measured as follows: the delay from AR glasses 202 to EAS 214 is equal to t4-t3, the delay from EAS 214 to AR glasses 202 is equal to t2-t1 or t6-t5, RTT is equal to t6-t3-(t5-t4), and / or the processing delay on EAS 214 is equal to t5-t4. The delay of EAS 214 can be determined or measured as follows: the delay from AR glasses 202 to EAS 214 is equal to t4-t3, the delay from EAS 214 to AR glasses 202 is equal to or t2-t1, RTT is equal to t4-t1-(t3-t2), and the processing delay on AR glasses 202 is equal to t3-t2.

[0099] Figures 6A and 6B are conceptual diagrams illustrating examples of using timestamps in RTP header extensions according to one or more aspects of this disclosure. In Figure 6A, packet 400 may be an RTP packet. Packet 400 may include: a payload 410, which may include, for example, video data; a header extension 420, which may include one or more header extension elements carrying a first time indicator T1; and a header 430, which may be an RTP header. For example, AR glasses 202 may generate packet 400 for transmission to EAS 214.

[0100] Packet 402 may also be an RTP packet. Packet 402 may include: a payload 412, which in this example may include video data; a header extension 422, which may include one or more header extension elements carrying a first time indicator T1, a second time indicator T2, and a third time indicator T3; and a header 432, which may be an RTP header. EAS 214 may determine the second time indicator T2 upon receiving packet 400, and may determine the third time indicator T3 after processing packet 400 and before sending packet 402. In some examples, the difference between the second time indicator T2 and the third time indicator T3 may represent the processing delay associated with EAS 214 processing packet 400.

[0101] AR glasses 202 can determine a fourth time indicator T4 upon receiving packet 402. When determining a data packet delay, AR glasses 202 can use a first time indicator T1, a second time indicator T2, a third time indicator T3, and / or a time indicator T4.

[0102] Figure 6B illustrates example formats for RTP header extension 420 and RTP header extension 422. For example, RTP header extension 420 includes T1, while RTP header extension 422 includes T1, T2, and T3. RTP header extension 420 may include an identifier (ID), which may have a value different from the ID of RTP header extension 422. The value of the ID identifies the RTP header extension, including whether the RTP header extension is a first type of RTP header extension (e.g., including a timestamp T1) or a second type of RTP header extension (e.g., including timestamps T1, T2, and T3).

[0103] Figures 7A and 7B are conceptual diagrams illustrating examples of timestamps in the payload of an RTP packet according to one or more aspects of this disclosure. In Figure 7A, packet 500 may be an RTP packet. Packet 500 may include: a payload 510, which may include a first time indicator T1 and, for example, video data; a header extension 520, which may include one or more header extension elements carrying information related to the first time indicator T1; and a header 530, which may be an RTP header.

[0104] Packet 502 may also be an RTP packet. Packet 502 may include: a payload 512, which in this example may include a first time indicator T1, a second time indicator T2, a third time indicator T3, and video data; a header extension 522, which may include one or more header extension elements carrying information related to the first time indicator T1, the second time indicator T2, and the third time indicator T3; and a header 532, which may be an RTP header. EAS 214 may determine the second time indicator T2 upon receiving packet 500, and may determine the third time indicator T3 after processing packet 500 and before sending packet 502. In some examples, the difference between the second time indicator T2 and the third time indicator T3 may represent the processing delay associated with EAS 214 processing packet 500.

[0105] AR glasses 202 can determine a fourth time indicator T4 upon receiving packet 502. When determining a data packet delay, AR glasses 202 can use a first time indicator T1, a second time indicator T2, a third time indicator T3, and / or a time indicator T4.

[0106] Figure 7B illustrates example formats for timestamps in payloads 510 and 512. For example, payload 510 includes T1, while payload 512 includes T1, T2, and T3.

[0107] Figure 8 is a flowchart illustrating an example delay measurement technique according to one or more aspects of this disclosure. Figure 8 is described relative to AR glasses 202, but the technique of Figure 8 can be implemented by any device capable of performing such techniques.

[0108] AR glasses 202 can send or receive SDP messages that include binding information associating a first RTP header extension and a second RTP header extension of an RTP session. This binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time (600) at which a first RTP packet including the first RTP header extension was sent. For example, AR glasses 202 can send or receive one or more SDP messages 120 from EAS 214. The SDP messages can associate a first RTP header extension (e.g., RTP header extension 330 with a timestamp T1) and a second RTP header extension (e.g., RTP header extension 332 with timestamps T1, T2, and T3) for the RTP session between AR glasses 202 and EAS 214. The binding information can indicate T1 in the first RTP header extension and T1 in the second RTP header extension, both of which indicate the time when the AR glasses 202 sends a first RTP packet including the first RTP header extension.

[0109] AR glasses 202 can send this first RTP packet (602). For example, AR glasses 202 can send RTP packet 320 to EAS 214.

[0110] AR glasses 202 can receive a second RTP packet, which includes the second RTP header extension. The second RTP header extension includes the first timestamp, a second timestamp indicating the time the second device received the first RTP packet, and a third timestamp (604) indicating the time the second device sent the second RTP packet. For example, AR glasses 202 can receive RTP packet 322 from EAS 214. RTP packet 322 may include a second type of RTP header extension 332, which includes timestamps T1, T2, and T3. Timestamp T1 may be the same as the timestamp T1 in the first type of RTP header extension 330 in RTP packet 320. Timestamp T2 may indicate the time EAS 214 received RTP packet 320. Timestamp T3 may indicate the time EAS 214 transmitted RTP packet 322.

[0111] AR glasses 202 can determine the delay (606) based on at least one of the first timestamp, the second timestamp, or the third timestamp. For example, AR glasses 202 can determine the delay from the first device to the second device by subtracting the value of the first timestamp (T1) from the value of the second timestamp (T2), where the delay = T2 - T1. For example, AR glasses 202 can determine the delay from the second device to the first device by subtracting the value of the third timestamp (T3) from the value (T4) indicating the time when the first device receives the second RTP packet, where the delay = T4 - T3. For example, AR glasses 202 can determine the RTT by subtracting the difference between the value of the third timestamp (T3) and the value of the second timestamp (T2) from the difference between the value (T4) indicating the time when the first device receives the second RTP packet and the value of the first timestamp (T1), where the RTT = (T4 - T1) – (T3 - T2). For example, AR glasses 202 can determine the processing delay by subtracting the value of the second timestamp (T2) from the value of the third timestamp (T3), where the processing delay = T3 - T2.

[0112] In some examples, AR glasses 202 may negotiate with EAS 214 the binding of the first RTP header extension and the second RTP header extension, wherein negotiating the binding includes sending or receiving the SDP message. In some examples, determining the latency is performed by at least one of the first device (e.g., AR glasses 202) or another device (e.g., QoS device 116 of FIG. 2). In some examples, the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

[0113] Figure 9 is a flowchart illustrating example other delay measurement techniques according to one or more aspects of this disclosure. Figure 9 is described relative to EAS 214, but the techniques of Figure 9 can be performed by any device capable of performing such techniques.

[0114] EAS 214 can send or receive SDP messages that include binding information associating a first RTP header extension and a second RTP header extension for an RTP session. This binding information indicates a first timestamp in the first RTP header extension and the first timestamp in the second RTP header extension, both indicating the time (700) at which a first RTP packet including the first RTP header extension was sent. For example, EAS 214 can send or receive one or more SDP messages 120 from AR glasses 202. The SDP message can associate a first RTP header extension (e.g., RTP header extension 330 with a timestamp T1) and a second RTP header extension (e.g., RTP header extension 332 with timestamps T1, T2, and T3) for the RTP session between EAS 214 and AR glasses 202 via the binding information. The binding information indicates T1 in the first RTP header extension and T1 in the second RTP header extension, both of which indicate the time when the AR glasses 202 sends a first RTP packet including the first RTP header extension.

[0115] EAS 214 can receive this first RTP packet (702). For example, EAS 214 can receive RTP packet 320 from AR glasses 202.

[0116] EAS 214 may send a second RTP packet including the second RTP header extension, which includes the first timestamp, a second timestamp indicating the time when the second device received the first RTP packet, and a third timestamp (704) indicating the time when the second device sent the second RTP packet. For example, EAS 214 may send RTP packet 322 to AR glasses 202. RTP packet 322 may include a second type of RTP header extension 332, which includes timestamps T1, T2, and T3. Timestamp T1 may be the same as the timestamp T1 in the first type of RTP header extension 330 in RTP packet 320. Timestamp T2 may indicate the time when EAS 214 received RTP packet 320. Timestamp T3 may indicate the time when EAS 214 transmitted RTP packet 322.

[0117] EAS 214 may determine the delay (706) based on at least one of the first timestamp, the second timestamp, or the third timestamp. For example, EAS 214 may determine the delay from the first device to the second device by subtracting the value of the first timestamp (T1) from the value of the second timestamp (T2), where the delay = T2 - T1. For example, EAS 214 may determine the processing delay by subtracting the value of the second timestamp (T2) from the value of the third timestamp (T3), where the processing delay = T3 - T2.

[0118] In some examples, EAS 214 may negotiate with AR glasses 202 the binding of the first RTP header extension and the second RTP header extension, wherein negotiating the binding includes sending or receiving the SDP message. In some examples, determining the latency is performed by at least one of the second device (e.g., EAS 214) or another device (e.g., QoS device 116 of FIG. 2). In some examples, the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

[0119] Various examples of the techniques disclosed herein are summarized in the following provisions:

[0120] Clause 1A. A method comprising: a first computing device communicating with a second computing device via a protocol relating to at least one field of a first type of Real-time Transport Protocol (RTP) header extension; the first computing device binding the at least one field of the first type of RTP header extension to at least one field of a second type of RTP header extension based on the communication; and determining a delay based on the binding.

[0121] Clause 2A. The method according to Clause 1A, wherein the protocol includes a session description protocol.

[0122] Clause 3A. The method according to Clause 1A or Clause 2A, wherein the at least one field of the RTP header extension of the first type includes a timestamp field.

[0123] Clause 4A. The method according to any one of Clauses 1A to 3A, wherein the first type of RTP header extension includes a first identifier, and the second type of RTP header extension includes a second identifier, wherein the first identifier is different from the second identifier.

[0124] Clause 5A. The method according to any one of Clauses 1A to 4A, wherein the binding includes mapping a first field in the RTP header extension of the first type to a first field in the RTP header extension of the second type.

[0125] Clause 6A. The method according to Clause 5A, wherein the first field includes a first timestamp indicating the time when the first computing device or the second computing device sent a first RTP packet including the RTP header extension of the first type.

[0126] Clause 7A. The method according to any one of Clauses 1A to 6A, wherein the binding includes mapping a second field in the RTP header extension of the second type based on one or more events associated with the RTP header extension of the first type.

[0127] Clause 8A. The method according to Clause 7A, wherein the second field in the second type of RTP header extension indicates the time when the first computing device or the second computing device receives an RTP packet including the first type of RTP header extension.

[0128] Clause 9A. The method according to any one of Clauses 1A to 8A, wherein the second type of RTP header extension includes an indicator indicating an association between a first RTP packet including the first type of RTP header extension and a second RTP packet including the second type of RTP header.

[0129] Clause 10A. A method comprising: determining, by a first device, a first time indicator, a second time indicator, and a third time indicator of an RTP packet including an RTP header extension; and determining, by the first device, a delay based on at least one of the first time indicator, the second time indicator, and the third time indicator.

[0130] Clause 11A. The method according to Clause 10A, wherein the first time indicator, the second time indicator and the third time indicator include a corresponding timestamp.

[0131] Clause 12A. The method according to Clause 10A, wherein the first time indicator and the second time indicator include corresponding timestamps, and wherein the third time indicator indicates the time difference between the time when the RTP packet was sent and the time indicated by the first time indicator.

[0132] Clause 13A. The method according to any one of Clauses 10A to 12A, wherein the RTP packet is a first RTP packet, and wherein the first time indicator indicates the time at which the second device sends the second RTP packet to the first device.

[0133] Clause 14A. The method according to any one of Clauses 10A to 13A, wherein the RTP packet is a first RTP packet, and wherein the second time indicator indicates the time at which the first device receives the second RTP packet from the second device.

[0134] Clause 15A. The method according to any one of Clauses 10A to 14A, wherein the RTP third time indicator indicates the time when the RTP packet is sent from the first device to the second device.

[0135] Clause 16A. The method according to Clause 10A, wherein the RTP packet is a first RTP packet, the method further comprising: determining by the first device that a second RTP packet has not yet been received from the second device; and determining, based on the fact that the second RTP packet has not been received from the second device, that the first time indicator and the second time indicator are equal to predetermined values.

[0136] Clause 17A. The method according to any one of Clauses 10A to 16A, wherein the RTP packet is a first RTP packet, the method further comprising associating the first RTP packet with a second RTP packet.

[0137] Clause 18A. A computing device comprising: one or more memories configured to store RTP packets; and one or more processors coupled to the memories, the one or more processors configured to perform any one of the methods described in Clauses 1A to 17A.

[0138] Clause 19A. The computing device as described in Clause 18A, wherein the computing device includes a mobile device or an application server.

[0139] Clause 20A. A computing device comprising at least one component for performing any one of the methods described in accordance with Clauses 1A to 17A.

[0140] Clause 21A. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any one of the methods described in Clauses 1A to 17A.

[0141] Clause 1B. A method for determining a delay, the method comprising: sending or receiving a Session Description Protocol (SDP) message by a first device, the SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; sending the first RTP packet by the first device; receiving a second RTP packet by the first device, the second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time of receipt of the first RTP packet by a second device, and a third timestamp indicating the time of transmission of the second RTP packet by the second device; and determining a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0142] Clause 2B. The method according to Clause 1B, the method further comprising the first device and the second device negotiating a binding between the first RTP header extension and the second RTP header extension, wherein negotiating the binding includes sending or receiving the SDP message.

[0143] Clause 3B. The method according to Clause 1B or Clause 2B, wherein determining the delay is performed by at least one of the first device or the other device.

[0144] Clause 4B. The method according to any one of Clauses 1B to 3B, wherein determining the delay comprises determining the delay from the first device to the second device by subtracting the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

[0145] Clause 5B. The method according to any one of Clauses 1B to 3B, wherein determining the delay comprises determining the delay from the second device to the first device by subtracting the value of the third timestamp (T3) from a value (T4) indicating the time when the first device receives the second RTP packet, wherein the delay = T4 - T3.

[0146] Clause 6B. The method according to any one of Clauses 1B to 3B, wherein determining the delay comprises determining the round-trip time (RTT) by subtracting the difference between the value of the third timestamp (T3) and the value of the second timestamp (T2) from the difference between the value of the first timestamp (T4) indicating the time when the first device received the second RTP packet and the value of the first timestamp (T1), wherein RTT = (T4 - T1) – (T3 - T2).

[0147] Clause 7B. The method according to any one of Clauses 1B to 3B, wherein determining the delay comprises determining the processing delay by subtracting the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

[0148] Clause 8B. The method according to any one of Clauses 1B to 7B, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

[0149] Clause 9B. A first device for processing media data, the first device comprising: one or more memories for storing the media data; and one or more processors communicatively coupled to the one or more memories, the one or more processors configured to: send or receive a Session Description Protocol (SDP) message, the SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; transmit the first RTP packet; receive a second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time of receipt of the first RTP packet by a second device, and a third timestamp indicating the time of transmission of the second RTP packet by the second device; and determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0150] Clause 10B. The first device as described in Clause 9B, wherein the one or more processors are configured to negotiate with the second device the binding of the first RTP header extension and the second RTP header extension, wherein, as part of negotiating the binding, the one or more processors are configured to send or receive the SDP message.

[0151] Clause 11B. The first device as described in Clause 9B or Clause 10B, wherein the delay includes a delay from the first device to the second device, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

[0152] Clause 12B. A first device pursuant to any one of Clauses 9B to 11B, wherein the delay includes a delay from the second device to the first device, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the third timestamp (T3) from a value (T4) indicating the time when the first device receives the second RTP packet, wherein the delay = T4 - T3.

[0153] Clause 13B. A first device pursuant to any one of Clauses 9B to 11B, wherein the delay includes round-trip time (RTT), and wherein, as part of determining the delay, the one or more processors are configured to subtract from the difference between the value of the third timestamp (T3) and the value of the second timestamp (T2) from the difference between a value (T4) indicating when the first device receives the second RTP packet and the value of the first timestamp (T1), wherein RTT = (T4 - T1) – (T3 - T2).

[0154] Clause 14B. The first device according to any one of Clauses 9B to 11B, wherein the delay includes determining a processing delay, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

[0155] Clause 15B. The first device pursuant to any one of Clauses 9B to 14B, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

[0156] Clause 16B. The first device pursuant to any one of Clauses 9B to 15B, wherein the first device includes a mobile device, an extended reality device, or an application server.

[0157] Clause 17B. A method for determining a delay, the method comprising: sending or receiving a Session Description Protocol (SDP) message by a second device, the SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; receiving the first RTP packet by the second device; transmitting a second RTP packet by the second device, the second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time the second device received the first RTP packet, and a third timestamp indicating the time the second device transmitted the second RTP packet; and determining a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0158] Clause 18B. The method according to Clause 17B, the method further comprising the second device negotiating a binding between the first RTP header extension and the second RTP header extension with the first device, wherein negotiating the binding includes sending or receiving the SDP message.

[0159] Clause 19B. The method described in accordance with Clause 17B or Clause 18B, wherein determining the delay is performed by at least one of the second device or the other device.

[0160] Clause 20B. The method according to any one of Clauses 17B to 19B, wherein determining the delay comprises determining the delay from the first device to the second device by subtracting the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

[0161] Clause 21B. The method according to any one of Clauses 17B to 19B, wherein determining the delay comprises determining the processing delay by subtracting the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

[0162] Clause 22B. The method according to any one of Clauses 17B to 21B, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

[0163] Clause 23B. A second device for processing media data, the second device comprising: one or more memories for storing the media data; and one or more processors communicatively coupled to the one or more memories, the one or more processors configured to: send or receive a Session Description Protocol (SDP) message, the SDP message including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; receive the first RTP packet; transmit a second RTP packet including the second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time the second device received the first RTP packet, and a third timestamp indicating the time the second device transmitted the second RTP packet; and determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

[0164] Clause 24B. The second device as described in Clause 23B, wherein the one or more processors are configured to negotiate with the first device the binding of the first RTP header extension to the second RTP header extension, wherein, as part of negotiating the binding, the one or more processors are configured to send or receive the SDP message.

[0165] Clause 25B. The second device as described in Clause 23B or Clause 24B, wherein the delay includes a delay from the first device to the second device, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

[0166] Clause 26B. The second device pursuant to Clause 23B or Clause 24B, wherein the delay includes a processing delay, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

[0167] Clause 27B. The second device according to any one of Clauses 23B to 26B, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

[0168] Clause 28B. A second device pursuant to any one of Clauses 23B to 27B, wherein the second device includes a mobile device, an extended reality device, or an application server.

[0169] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to extract instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0170] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include: compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combinations described above should also be included within the scope of computer-readable media.

[0171] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0172] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Specifically, as described above, various units may be combined in a codec hardware unit, or various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0173] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for determining a delay, the method comprising: The first device sends or receives a Session Description Protocol (SDP) message, which includes binding information that associates a first RTP header extension and a second RTP header extension of an RTP session. The binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both of which indicate the time when a first RTP packet including the first RTP header extension was sent. The first device sends the first RTP packet; the first device receives a second RTP packet, the second RTP packet including a second RTP header extension, the second RTP header extension including a first timestamp, a second timestamp indicating the time when the second device received the first RTP packet, and a third timestamp indicating the time when the second device sent the second RTP packet; and a delay is determined based on at least one of the first timestamp, the second timestamp, or the third timestamp.

2. The method according to claim 1, the method further comprising negotiating a binding between the first RTP header extension and the second RTP header extension by the first device and the second device, wherein negotiating the binding includes sending or receiving the SDP message.

3. The method of claim 1, wherein determining the delay is performed by at least one of the first device or the other device.

4. The method of claim 1, wherein determining the delay comprises determining the delay from the first device to the second device by subtracting the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

5. The method of claim 1, wherein determining the delay comprises determining the delay from the second device to the first device by subtracting the value of the third timestamp (T3) from a value (T4) indicating the time when the first device receives the second RTP packet, wherein the delay = T4 - T3.

6. The method of claim 1, wherein determining the delay comprises determining the round-trip time (RTT) by subtracting the difference between the value of the third timestamp (T3) and the value of the second timestamp (T2) from the difference between the value of the time indicating that the first device received the second RTP packet (T4) and the value of the first timestamp (T1), wherein RTT = (T4 - T1) – (T3 - T2).

7. The method of claim 1, wherein determining the delay comprises determining the processing delay by subtracting the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

8. The method of claim 1, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

9. A first apparatus for processing media data, the first apparatus comprising: One or more memories, the one or more memories being used to store the media data; and one or more processors, the one or more processors being communicatively coupled to the one or more memories, the one or more processors being configured to: send or receive Session Description Protocol (SDP) messages, the SDP messages including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; Send the first RTP packet; receive a second RTP packet, the second RTP packet including a second RTP header extension, the second RTP header extension including a first timestamp, a second timestamp indicating the time when the second device received the first RTP packet, and a third timestamp indicating the time when the second device sent the second RTP packet; and determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

10. The first device of claim 9, wherein the one or more processors are configured to negotiate with the second device a binding between the first RTP header extension and the second RTP header extension, wherein, as part of negotiating the binding, the one or more processors are configured to send or receive the SDP message.

11. The first device of claim 9, wherein the delay includes a delay from the first device to the second device, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

12. The first device of claim 9, wherein the delay includes a delay from the second device to the first device, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the third timestamp (T3) from a value (T4) indicating the time when the first device receives the second RTP packet, wherein the delay = T4 - T3.

13. The first device of claim 9, wherein the delay includes round-trip time (RTT), and wherein, as part of determining the delay, the one or more processors are configured to subtract the difference between the value of the third timestamp (T3) and the value of the second timestamp (T2) from the difference between a value (T4) indicating when the first device receives the second RTP packet and the value of the first timestamp (T1), wherein RTT = (T4 - T1) – (T3 - T2).

14. The first device of claim 9, wherein the delay includes determining a processing delay, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

15. The first device of claim 9, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

16. The first device according to claim 9, wherein the first device includes a mobile device, an extended reality device, or an application server.

17. A method for determining a delay, the method comprising: The second device sends or receives a Session Description Protocol (SDP) message, which includes binding information that associates a first RTP header extension and a second RTP header extension of an RTP session. The binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both of which indicate the time when a first RTP packet including the first RTP header extension was sent. The second device receives the first RTP packet; the second device sends a second RTP packet, the second RTP packet including a second RTP header extension, the second RTP header extension including a first timestamp, a second timestamp indicating the time when the second device received the first RTP packet, and a third timestamp indicating the time when the second device sent the second RTP packet; and a delay is determined based on at least one of the first timestamp, the second timestamp, or the third timestamp.

18. The method of claim 17, further comprising the second device negotiating a binding between the first RTP header extension and the second RTP header extension with the first device, wherein negotiating the binding includes sending or receiving the SDP message.

19. The method of claim 17, wherein determining the delay is performed by at least one of the second device or the other device.

20. The method of claim 17, wherein determining the delay comprises determining the delay from the first device to the second device by subtracting the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

21. The method of claim 17, wherein determining the delay comprises determining the processing delay by subtracting the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

22. The method of claim 17, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

23. A second apparatus for processing media data, the second apparatus comprising: One or more memories, the one or more memories being used to store the media data; and one or more processors, the one or more processors being communicatively coupled to the one or more memories, the one or more processors being configured to: send or receive Session Description Protocol (SDP) messages, the SDP messages including binding information associating a first RTP header extension and a second RTP header extension of an RTP session, wherein the binding information indicates a first timestamp in the first RTP header extension and a first timestamp in the second RTP header extension, both indicating the time of transmission of a first RTP packet including the first RTP header extension; Receive the first RTP packet; send a second RTP packet, the second RTP packet including a second RTP header extension, the second RTP header extension including the first timestamp, a second timestamp indicating the time when the second device received the first RTP packet, and a third timestamp indicating the time when the second device sent the second RTP packet; and determine a delay based on at least one of the first timestamp, the second timestamp, or the third timestamp.

24. The second device of claim 23, wherein the one or more processors are configured to negotiate with the first device a binding of the first RTP header extension and the second RTP header extension, wherein, as part of negotiating the binding, the one or more processors are configured to send or receive the SDP message.

25. The second device of claim 23, wherein the delay includes a delay from the first device to the second device, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the first timestamp (T1) from the value of the second timestamp (T2), wherein the delay = T2 - T1.

26. The second device of claim 23, wherein the delay includes a processing delay, and wherein, as part of determining the delay, the one or more processors are configured to subtract the value of the second timestamp (T2) from the value of the third timestamp (T3), wherein the processing delay = T3 - T2.

27. The second device of claim 23, wherein the SDP message further includes an indication that at least one of the first RTP header extension or the second RTP header extension is formatted as short or long.

28. The second device of claim 23, wherein the second device includes a mobile device, an extended reality device, or an application server.

Images