Handling timestamp inaccuracies for streaming network protocols
By calculating the corrected timestamp range, the accuracy of streaming data when losing network packets is solved, the reliability and efficiency of real-time communication services are improved, and the latency and privacy risks are reduced.
Patent Information
- Application Number
- CN201980082546.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-11
- Filing Date
- 2019-12-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-12-04
AI Technical Summary
In the prior art, when processing streaming data, due to the use of fixed timestamp increments, the amount of lost streaming data cannot be accurately calculated when network packets are lost, affecting the reliability and delay management of real-time communication services.
By receiving the timestamp values and frame attributes in the network packet, the corrected timestamp range is calculated, and the jitter buffer is adjusted to ensure accurate playback and delay control of streaming data.
Improves the reliability and efficiency of streaming data transmission, reduces latency and privacy issues, and provides more accurate timestamp information to manage latency and packet loss.
Smart Images

Figure CN113169904B_ABST
Abstract
Description
Background Art
[0001] Real-time communication services are used to transmit audio, video, and other types of streaming data via computer networks such as the Internet. Real-time communication services rely on reliable, timely delivery of network packets in order to provide a positive user experience. For example, if a real-time audio communication service used for an audio call encounters network problems (e.g., lost or delayed network packets), the quality of the communication will be affected.
[0002] Timing information is also important for real-time communication services. For example, timing information is needed to manage various aspects of network packet reception and playback. Without accurate timing information, reliable and accurate playback of streaming media will be affected (e.g., latency issues, playback synchronization issues, etc.).
[0003] Thus, there are substantial opportunities for improvement in techniques related to handling timestamp inaccuracies for streaming network protocols. Summary of the invention
[0004] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
[0005] Techniques for determining corrected timestamps for streaming data are described, the streaming data being encoded using frames having a variable frame size. The streaming data is encoded into frames and transmitted in network packets, wherein the network packets or frames are associated with timestamps that increment in fixed steps. When a network packet is received after a lost network packet, a corrected timestamp range can be calculated for the received network packet based at least in part on the received timestamp value and attributes of the received network packet and buffering characteristics.
[0006] For example, a computing device may perform an operation for determining a corrected timestamp range for streaming data, wherein the streaming data is encoded using frames having a variable frame size. The operation may include receiving a network packet, the network packet including a received timestamp value and a frame of streaming data, wherein the received timestamp value has a fixed timestamp increment. The operation may further include determining that a previous network packet including a previous frame of streaming data has been lost. The operation may also include calculating a possible buffer size range for the received network packet, wherein the possible buffer size range is for an encoding buffer for encoding the streaming data. The operation may further include calculating a corrected timestamp range for the received network packet based at least in part on the received timestamp value and the possible buffer size range. The operation may further include adjusting a jitter buffer based on the corrected timestamp range.
[0007] As described herein, various other features and advantages may be incorporated into the techniques as desired. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is a diagram depicting an example environment for determining a corrected timestamp range for streaming data using variable-sized frames.
[0009] Figure 2 is a diagram depicting an example environment for determining a corrected timestamp range for streaming audio using variable-sized frames.
[0010] Figure 3 is a diagram depicting example frames of streaming data along with real timestamps and fixed timestamps.
[0011] Figure 4 is a diagram depicting the use of past and future frames to calculate the corrected timestamp range.
[0012] Figure 5 is a flow chart of an example method for determining a corrected timestamp range for streaming data.
[0013] Figure 6 is a flow chart of an example method for determining a corrected timestamp range for streaming audio.
[0014] Figure 7 is a diagram of an example computing system in which some described embodiments may be implemented.
[0015] Figure 8 is an example mobile device that can be used in conjunction with the described techniques.
[0016] Fig. 9is an example cloud support environment that can be used in conjunction with the described techniques. DETAILED DESCRIPTION
[0017] Overview
[0018] As described herein, various techniques and solutions can be applied to determine a corrected timestamp (e.g., a corrected timestamp range and / or a specific corrected timestamp value) for streaming data, which is encoded using a frame with a variable frame size. The streaming data described herein can be any type of streaming data, such as streaming audio data, streaming video data, or streaming media data (e.g., including audio, video, graphics, etc.). Using a network protocol, the streaming data is encoded into frames and sent from a sending computing device to a receiving computing device in the form of a network packet. The frames of the streaming data are associated with a timestamp that increases in a fixed step size. For example, each frame of the streaming data can increase the timestamp value by a fixed step size (also referred to as a fixed timestamp increment), such as 20 milliseconds (ms). However, since the streaming data is encoded using a frame with a variable frame size, the amount of streaming data in a given frame may be different from the fixed timestamp increment. For example, the first frame of the streaming data can have 16ms of data (e.g., 16ms of audio samples), and the second frame of the streaming data can have 28ms of data (e.g., 28ms of audio samples). However, if the fixed timestamp increment is 20ms, then reported timestamp values (eg, timestamp values reported in a network packet header) will be in 20ms increments (eg, 20ms for the first frame and 40ms for the second frame).
[0019] The difference between the actual amount of streaming data in a frame and the fixed increment timestamp value may cause problems during streaming media playback. For example, this is important for error concealment and for other reasons for a receiving device (e.g., a decoder or playback component of a receiving device) that has fairly accurate timestamp information to manage delays (e.g., by controlling a jitter buffer). Because the frames of streaming data used in the techniques described herein have variable frame sizes, if a network packet is lost, the receiving device will not know how much streaming data has been lost. Although the receiving device may know that a network packet has been lost (e.g., via a sequence value in a network packet header of a subsequently received network packet), the receiving device will not know exactly how much streaming data is in the lost network packet. This is because the timestamp value received by the receiving device has a fixed timestamp increment and cannot accurately reflect the correct timestamp value, which depends on the actual amount of streaming data in the lost packet.
[0020] In the case of lost network packets and where the streaming data uses a fixed timestamp increment, the techniques described herein can be applied to determine a corrected timestamp range (a timestamp range that more accurately reflects the true timestamp). The corrected timestamp range can be calculated based on the characteristics of how buffering is performed for the streaming data. The buffering characteristics can include the amount of streaming data encoded by the sending device for a given frame, and how much streaming data is left in the buffer to be encoded in the next frame. For example, a network packet can be received, wherein the network packet includes a received timestamp value (using a fixed timestamp increment) and a frame of streaming data. It can then be determined that the previous network packet has been lost (e.g., based on the sequence number of the received network packet). In response to determining that the previous network packet has been lost, a possible buffer size range can be calculated with respect to the received network packet. Then, the corrected timestamp range can be calculated for the received network packet based at least in part on the received timestamp value and the possible buffer size range. The corrected timestamp range (e.g., the range itself and / or a specific corrected timestamp value selected from the range) can be used to control aspects of streaming, decoding, and / or playback of the streamed data (e.g., managing latency, packet loss concealment, etc.).
[0021] In some embodiments, streaming data is encoded and sent from a sending device to a receiving device according to a real-time transport protocol (RTP). RTP is a network protocol for streaming audio and / or video data using an Internet Protocol (IP) network. RTP contains a timestamp field in its network packet header, which indicates the timestamp of the start of the streaming data carried in the RTP packet. When using RTP, the technology herein uses a fixed timestamp increment, which is 20ms in some embodiments. Therefore, in such an embodiment, for each frame in the packet, the RTP timestamp value will be incremented by a fixed timestamp increment. For example, if each packet contains a frame of streaming data, and the fixed timestamp increment is 20ms, the first RTP packet will have an RTP timestamp value of 0ms (the start of the streaming data), the second RTP packet will have an RTP timestamp value of 20ms, the third RTP packet will have an RTP timestamp value of 40ms, and so on. As another example, if the fixed timestamp increment is 20ms, the RTP network packet with three frames of audio data will increment the timestamp value by 60ms.
[0022] In other solutions, a variable timestamp is sent in the network packet (e.g., an actual timestamp value reflecting the amount of streaming data in the packet). However, such a solution has many problems. For example, the network streaming protocol may specify the use of a fixed timestamp increment, and changing its operation may require custom coding and interfere with interoperability. In addition, the use of the actual timestamp value may raise privacy issues (e.g., it may enable others to gain insight into the content or activity of the streaming data (e.g., audio content)). Furthermore, including the actual timestamp in the network packet or frame may use additional bits and therefore increase bandwidth.
[0023] The techniques described herein also provide benefits in terms of reliability and efficiency (e.g., of real-time network communication services). For example, not including the actual timestamp in the network packet or frame can save bits and network bandwidth, which can also reduce latency. In addition, not including the actual timestamp in the network packet or frame improves the security of communications (e.g., reduces privacy issues). Finally, in the event of a packet loss that loses timing continuity (such as may be a possible event), the techniques described herein can be used to efficiently calculate a corrected timestamp range (and the corrected timestamp value).
[0024] buffer
[0025] In the techniques described herein, streaming data is buffered for encoding (e.g., buffered before being encoded by an audio and / or video codec). Since the streaming data is encoded using a variable frame size (and therefore a variable amount of streaming data per frame), there may often be data left in the buffer. For example, a typical streaming environment generates a fixed amount of data per unit time (e.g., a fixed number of audio samples per second, depending on the sampling rate used), which is buffered for encoding. In some embodiments, audio data is captured and buffered in increments of 20 ms, where 20 ms of audio samples are placed in the buffer every 20 ms. For example, if the audio codec is encoding 14 ms of audio samples in a frame from the buffer using 20 ms of data, 6 ms of audio samples will remain in the buffer.
[0026] Depending on the encoding technique used (e.g., the specific codec), a maximum amount of data may be retained in the buffer after the current frame is encoded. For example, if the codec supports encoding streaming data in 6ms subframes and captures 20ms of data per cycle, the buffer will always retain 2ms of data after encoding the first frame of 18ms. For the next cycle, 20ms of data will be added to the buffer, resulting in 22ms of data being encoded. The codec will encode the second frame of 18ms, retaining 4ms in the buffer. For the next time period, another 20ms of data will be added to the buffer, resulting in 24ms of data to be encoded. The codec will then encode the third frame of 24ms, retaining 0ms in the buffer. Therefore, for this relatively simple example, the minimum amount of data retained in the buffer will be 0ms (after encoding the frame), and the maximum amount of data retained in the buffer will be 4ms (after encoding the frame), and the maximum buffer size range will be [0, 4]ms. When the codec can encode different sized subframes of data (e.g., audio samples of any amount up to 6ms), it can also determine the minimum and maximum amount of data to remain in the buffer (called the maximum buffer size range).
[0027] In some embodiments, audio data is buffered according to an audio codec that uses a variable frame size and supports speech coding according to two modes: a voiced mode (for speech with a well-defined periodicity) and an unvoiced mode (for sounds without periodicity). For the voiced mode, a frame contains an integer number of pitch periods (also called pitch lags), with a maximum pitch period of 18.8 ms. For unvoiced frames, the number of buffered samples is fixed to a multiple of 4 ms. The samples remaining in the buffer will be encoded in the next frame. In addition, the mode can change from frame to frame. Because of the buffering, a frame starts earlier, but many samples are buffered in the previous frame. As a result, the real timestamp differs from the fixed timestamp increment (e.g., 20 ms) by a value in the following range:
[0028] -18.8ms to 0ms if the previous packet ends with voiced sound
[0029] -4ms to 0ms if the previous packet ended with an unvoiced sound
[0030] Sequential timestamp range tracking
[0031] In the techniques described herein, the maximum amount of streaming data that can be retained in the buffer after encoding the current frame can be used to determine the corrected timestamp range. The equations and operations described in this section are based on an example audio codec that uses variable-size frames and encodes audio data in voiced mode (encoding samples in pitch period increments or subframes, where the maximum pitch period is 18.8ms) and encodes audio data in unvoiced mode (encoding samples in increments or subframes of 4ms), and uses RTP. However, these equations and operations are generally applicable to other audio codecs (and are generally applicable to the encoding of streaming data). For example, the encoding of other types of streaming data may not use different types of streaming data, or may use more than two types of streaming data. In addition, fixed timestamp increments may be different for different codecs or encoding techniques. The minimum and / or maximum buffer sizes may also be different for different codecs or encoding techniques.
[0032] As mentioned before, the maximum amount buffered by the encoder for unvoiced frames is less than the maximum amount buffered by the encoder for unvoiced frames. The frame duration also tells us something about the amount buffered. For example, if a 35ms frame is received, then we know that at most 3.8ms can be buffered at the end of the frame, because otherwise 18.8ms more would need to be buffered for the previous frame. This logic can be combined to track the uncertainty in the buffer size across frames, reducing the margin for uncertainty in the true timestamp compared to single frame analysis.
[0033] The equations below operate on frames, which are the units of processing in the encoder and decoder. Each frame has a nominal length of 20ms. Each network packet contains one or more frames and has an RTP timestamp value associated with the first sample of the first frame in the packet. The RTP timestamp sent as part of the RTP header increments in multiples of 20 milliseconds and points to the sample that is a multiple of 20 milliseconds before the end of the available audio data. For frame n, the correct (real) timestamp is related to this fixed increment RTP timestamp, the frame duration, and the encoder buffer size (after processing frame n) by the following relationship.
[0034] Real timestamp (n)
[0035] =Rounded Timestamp(n)+20-Frame Duration(n)-Buffer Size(n)(ms)
[0036] (Equation 1)
[0037] Figure 3 It shows how the frames, buffers, and timestamps of Equation 1 are related. Figure 3Two frames, frame n-1 and frame n, are depicted at 300. After encoding frame n-1, some data remains in a buffer, depicted as buffer n-1. The correct (real) timestamp 310 for frame n-1 reflects the actual amount of streaming data included in frame n-1 (e.g., 12ms of data). The fixed delta timestamp reflects the timestamp value signaled in the network packet header (e.g., RTP header), which in some embodiments is a multiple of 20ms. As depicted at 300, buffer n-1 is encoded in the next frame, frame n. After encoding frame n, there is streaming data indicated by buffer n in the buffer (which may be a different amount of streaming data than the remaining streaming data in buffer n-1). Uncertainty 330 reflects the size of buffer n-1, which in some embodiments is 0 ms to 18.8 ms if the previous frame (frame n-2, not shown) is voiced, or 0 ms to 4 ms if the previous frame is unvoiced.
[0038] The buffer size for frame n depends on the buffer size for frame n-1 according to the following relationship.
[0039] Buffer size (n) = buffer size (n-1) + 20 - frame duration (n) (ms)
[0040] (Equation 2)
[0041] Using the knowledge that the maximum buffer size is 4ms for unvoiced frames and 18.8ms for voiced frames, the uncertainty range for the timestamps can be recursively expressed as follows. First, if frame n-1 is lost, the maximum possible buffer size range is initialized to:
[0042] Buffer size range (n-1) = [0, 18.8] (ms)
[0043] Then, for subsequent frames, the intersection of the range propagated from the previous frame (also called the propagated buffer size range) and the range according to the current speech type is determined as follows: If frame (n) is unvoiced, the following equation is used.
[0044] Buffer size range (n)
[0045] = buffer size range (n-1) + 20 - frame duration (n) ∩ [0, 4] (ms)
[0046] (Equation 3)
[0047] If frame (n) is voiced, the following equation is used.
[0048] Buffer size range (n)
[0049] = Buffer size range (n-1) + 20-frame duration (n) ∩ [0, 18.8] (ms)
[0050] (Equation 4)
[0051] In general, the techniques shown in Equations 3 and 4 are used to calculate a possible buffer size range. For example, calculating the possible buffer size range may include calculating a propagated buffer size range based at least in part on the buffer size range of the lost frame and the duration of the received frame (corresponding to the "buffer size range (n-1) + 20-frame duration (n)" portion of Equations 3 and 4), and calculating the intersection of the propagated buffer size range and the maximum buffer size range for the received frame (corresponding to [0, 4] or [0, 18.8] in Equations 3 and 4, and depending on whether the streaming data has different types of data, the different types of data have different maximum buffer size ranges).
[0052] Next, use the following equation to determine the corrected RTP timestamp range.
[0053] Real timestamp range (n)
[0054] = timestamp(n)+20-frame duration(n) -Buffer size range (n) (ms)
[0055] (Equation 5)
[0056] In general, the technique shown in Equation 5 is used to calculate a corrected timestamp range for the current streaming protocol in use. For example, calculating the corrected timestamp range may include adjusting the received timestamp value (corresponding to timestamp (n) in Equation 5) by a fixed timestamp increment (corresponding to 20 in Equation 5), a duration of the received frame (corresponding to frame duration (n) in Equation 5), and a possible buffer size range (corresponding to buffer size range (n) in Equation 5).
[0057] As shown in the equation above, the corrected timestamp range becomes narrower (or remains the same) with each new frame received. This occurs until there is a lost network packet and the range is reset.
[0058] Equations 3, 4, and 5 can also be applied when future frames are available. Figure 4This is shown in . The approach is to first calculate the buffer size range to the end of the last frame, and then convert it back to the RTP timestamp range at an earlier frame by taking the RTP timestamp of that frame, adding the nominal (fixed timestamp increment) length of all frames starting from that frame, subtracting the actual lengths of those frames, and then subtracting the final buffer size range. Figure 4 These operations are shown for three frames, frame n-1 420, frame n 422, and frame n+1 424. However, the method may be applied to any number of past and / or future frames.
[0059] The following examples illustrate the calculation of the corrected timestamp range when the previous network packet and the corresponding previous frame of streaming data are lost. In the first example, it is assumed that packet n-1 has been lost and packet n is received as a single unvoiced frame of duration 18ms. The buffer size range will then be calculated according to Equation 3 as shown below.
[0060] Buffer size range (n) = [0, 18.8] + 20-18 ∩ [0, 4]
[0061] =[2, 20.8]∩[0, 4]
[0062] =[2, 4](ms)
[0063] Next, the corrected RTP timestamp range is calculated according to Equation 5 as shown below.
[0064] True timestamp range (n) = timestamp (n) + 20 - frame duration (n) - buffer size range (n) (ms)
[0065] = timestamp(n)+20-18-[2,4]
[0066] =Timestamp(n)+[-2,0](ms)
[0067] Therefore, according to the above calculation, the corrected timestamp range (i.e., the real timestamp range) for the current frame n will be the received timestamp in the range of -2ms to 0ms (e.g., the RTP timestamp reported in the RTP header, which is a fixed incremental timestamp). For example, if the received timestamp value is 400ms, the corrected timestamp range will be 398ms to 400ms.
[0068] In the second example, assume that packet n-1 is lost and packet n is received as a single voiced frame of duration 35ms. Then, the buffer size range will be calculated according to Equation 4 as follows.
[0069] Buffer size range (n) = [0, 18.8] + 20-35 ∩ [0, 18.8]
[0070] =[-15, 3.8]∩[0, 18.8]
[0071] =[0, 3.8](ms)
[0072] Next, the corrected RTP timestamp range is calculated according to Equation 5 as shown below.
[0073] True timestamp range (n) = timestamp (n) + 20 - frame duration (n) - buffer size range (n) (ms)
[0074] = timestamp(n)+20-35-[0,3.8]
[0075] =Timestamp(n)+[-15,-11.2](ms)
[0076] Therefore, according to the above calculation, the corrected timestamp range (i.e., the real timestamp range) for the current frame n will be a received timestamp (e.g., the RTP timestamp reported in the RTP header, which is a fixed incremental timestamp) in the range of -15ms to -11.2ms. For example, if the received timestamp value is 400ms, the corrected timestamp range will be 385ms to 388.8ms.
[0077] Multi-frame grouping
[0078] In the techniques described herein, the described techniques for calculating the corrected timestamp range can be applied to network packets containing multiple frames. For a multi-frame packet, equations 3 and 4 can be iterated for all frames in the packet, which will provide a buffer size range at the end of the packet. Then, a slightly rewritten version of equation 5 can be used to trace the buffer size range at the end of the packet back to the RTP timestamp range at the beginning of the packet.
[0079] Real timestamp range
[0080] = timestamp + 20 * number of frames in the packet - packet duration - buffer size range (ms)
[0081] (Equation 6)
[0082] For packets with forward error correction (FEC) data, the FEC data may be used to reduce the buffer size range if the FEC data describes data that immediately precedes the main data.
[0083] Air Delay Report
[0084] In some embodiments, the decoder reports the playout delay to a controller (e.g., a controller that manages the playback of streaming data at a receiving device). If the RTP timestamp is correct, this delay will be equal to the RTP timestamp of the current packet minus the playout timestamp. However, since the technology described herein uses variable-sized frames and fixed timestamp increments, there is a certain range of uncertainty in the RTP timestamp of the received packet. Therefore, some values within the RTP timestamp range can be selected. In some embodiments, the most conservative delay is reported, which is the lowest possible delay value within the uncertainty range (e.g., within the corrected timestamp range). In other embodiments, another value is selected within the uncertainty range.
[0085] Due to uncertainty in the RTP timestamps, there will be some fluctuation in the reported latency. However, this reported latency will naturally fluctuate anyway, as it is sensitive to the temporal order of incoming packets relative to the request for generating output samples. A packet arriving a fraction of a millisecond early or a fraction of a millisecond late can change the reported latency by 20ms. The controller can be expected to handle this fluctuation.
[0086] Out of order grouping
[0087] In some implementations, a component of the receiving device (e.g., a decoder, a controller, or some other component) calculates a new corrected timestamp range for each new packet that arrives. As a result, out-of-order packets will cause a reset in the range: if packet n arrives before packet n-1, it appears that packet n-1 is missing when calculating the range for packet n.
[0088] In some embodiments, after receiving packet n-1, the corrected timestamp range calculation is restored based on the history up to packet n-2, and the range is updated using packets n-1 and n. However, in some embodiments, out-of-order packets are very rare, so this situation can be ignored and only the range is reset with little impact on the perceived quality.
[0089] Packet Loss Concealment
[0090] In some embodiments, the broadcast timestamp points to the next sample to be broadcast. The timestamp is incremented by the number of samples generated. When a packet arrives after a packet is lost, its corrected RTP timestamp range is compared with the broadcast timestamp. If the broadcast timestamp is earlier than the middle of the corrected RTP timestamp range, a certain number of hidden subframes (e.g., partial frames of audio samples) are generated (but many are needed) to make it as close as possible to the middle of the corrected RTP timestamp range of the received packet. Afterwards, playback is switched to the decoded subframe.
[0091] If after concealment the playout timestamp is already close to or outside the middle of the corrected RTP timestamp range for the first received packet, then stop generating the hidden subframes and switch to the decoded subframes. In this case, the reported playout delay will drop to zero or a negative value. Negative delays are treated as a special case. First, the delay is reported to the controller with a low action cost, and the controller is expected to request an increase in the delay up to the size of the negative delay. The decoder then increases the delay by simply lowering the playout timestamp by the requested delay increase. If the delay is still negative after this delay increase, the decoder will discard subframes until the playout timestamp matches the lower end of the corrected RTP timestamp range for the first subframe to be played out.
[0092] There are two types of concealment: extrapolation (when no future frames are available) and interpolation (between past and future frames). When interpolating, concealment needs to determine how many subframes to insert. If the past and future frames are unvoiced, it is known that each lost 20ms frame contains 4 subframes. In addition, for unvoiced signals, the corrected RTP timestamp range is narrow (at most 4ms), so the difference can be taken between some point in the old and new RTP timestamp range. In order to interpolate between voiced frames, it is necessary to find a sequence of pitch lags (each pitch lag defines a subframe length) so that (1) the interpolated pitch lags evolve smoothly and (2) they fill the estimated RTP timestamp interval. In some embodiments, it is reasonable to use 70% of the corrected RTP timestamp range as an estimate of the true RTP timestamps for voiced and unvoiced frames.
[0093] In some cases, two-sided extrapolation is used instead of interpolation. This can happen if the voicing pattern changes from a past frame to a future frame, or if there is a large jump in pitch lag between a past frame and a future frame. In this case, an extrapolation is performed starting from the past, and also working backwards from the future. At some point in time in between, the two extrapolations meet. This point is not necessarily halfway between a past frame and a future frame, but can be chosen in a signal-dependent manner.
[0094] Environment used to determine the corrected timestamp range
[0095] Figure 1 1 is a diagram depicting an example environment 100 for determining a corrected timestamp range for streaming data using variable-sized frames. Example environment 100 depicts a sending device 110. Sending device 110 can be any type of computing device (e.g., a server computer, a virtual machine, a desktop computer, a laptop computer, a tablet computer, a phone, or another type of computing device) that encodes and sends streaming data (e.g., streaming audio and / or video).
[0096] As shown at 112, a variable amount of streaming data is obtained from the buffer. For example, the buffer can capture streaming data (e.g., audio and / or video data) as the streaming data is generated (e.g., via a microphone or a video capture device). In one example, a certain amount of streaming data is periodically placed in the buffer (e.g., every 20 ms, 20 ms worth of audio samples captured from the microphone are placed in the buffer). Then, as shown at 112, a variable amount of streaming data is selected from the buffer for encoding and transmission. For example, if there is 20 ms of data in the buffer, a maximum of 20 ms of data (e.g., 4 ms, 12.4 ms, 20 ms, or some other amount) can be obtained. The variable amount of streaming data obtained from the buffer can be determined based on a variety of criteria. For example, if the streaming data is audio, the amount can vary based on the type of audio data (e.g., if the audio data is periodic, such as voiced data, the amount of audio data can be determined based on the number of pitch periods).
[0097] The variable amount of streaming data is encoded in frames having a corresponding variable frame size, as depicted at 114. For example, if 12 ms of audio data is obtained from the buffer, the 12 ms of audio data is encoded in a frame having a 12 ms frame size (e.g., having 12 ms worth of audio samples).
[0098] As depicted at 116, the variable size frame is sent to the receiving device 120 in a network packet. The network packet is associated with a timestamp value having a fixed timestamp increment. For example, the frame can be transmitted in a network packet having a timestamp header value having a fixed increment of 20 ms since the previous frame. The network packet can be sent to the receiving device 120 via a streaming network protocol such as RTP or another streaming protocol.
[0099] The sending device 110 sends the network packets via a network 130 (e.g., the Internet and / or another type of public or private data network) to the receiving device 120. The receiving device 120 can be any type of computing device (e.g., a server computer, a virtual machine, a desktop computer, a laptop computer, a tablet computer, a phone, or another type of computing device) that receives and decodes streaming data (e.g., streaming audio and / or video).
[0100] As depicted at 122, a network packet is received from a sending device 110 via a network 130. The received network packet includes a received timestamp value and a variable size frame. The received timestamp value is a fixed increment timestamp value associated with a network packet or frame. For example, if the network protocol is RTP, the received timestamp value is the RTP header timestamp value (a multiple of 20ms or other fixed increment).
[0101] As depicted at 124, it is determined that a previous network packet has been lost (a previous network packet is the network packet that immediately precedes the received network packet). The determination that the previous network packet has been lost may be based on a packet sequence number (e.g., having received a current network packet with a given sequence number and determining that a previous network packet with a previous sequence number has not been received). It may be determined whether the lost network packet has not been received within a given time period.
[0102] As depicted at 126, a corrected timestamp range is calculated for the received network packet. For example, the corrected timestamp range can be calculated based at least in part on the received timestamp value and the possible buffer size range. In some embodiments, the corrected timestamp range is calculated using a technique similar to Equations 3, 4, and 5. The corrected timestamp range can be used (e.g., by a component of the receiving device 120, such as a decoder, a controller, a playback component, etc.) to control decoding and / or playback of the streaming data (e.g., to control jitter buffer properties to manage latency).
[0103] Figure 2 is a diagram depicting an example environment 200 for determining a corrected timestamp range for streaming audio using variable-sized frames. In some implementations, the example environment 200 can be used to determine a corrected timestamp range for streaming video and / or other types of streaming media in addition to or in lieu of streaming audio. The example environment 200 depicts a sending device 210. The sending device 210 can be any type of computing device (e.g., a server computer, a virtual machine, a desktop computer, a laptop computer, a tablet computer, a phone, or another type of computing device) that encodes and transmits streaming data (e.g., streaming audio and / or video).
[0104] The audio capture component 240 (e.g., a microphone) of the transmitting device 210 captures audio and periodically saves it in a buffer 245. For example, a 20 ms audio sample may be obtained from the audio capture component 240 and saved in the buffer 245 every 20 ms. In some embodiments, the transmitting device 210 may include a video capture device and a corresponding video buffer.
[0105] As depicted at 212, a variable amount of streaming audio is obtained from the buffer 245. For encoding and transmission. For example, if there is 20 milliseconds of data in the buffer, a maximum of 20 milliseconds of data may be obtained (e.g., 4 milliseconds, 12.4 milliseconds, 20 milliseconds, or some other amount). The variable amount of streaming audio obtained from the buffer may be determined based on various criteria. For example, the amount may vary based on the type of audio data (e.g., if the audio data is periodic, such as voiced data, the amount of audio data may be determined based on the number of pitch periods).
[0106] A variable amount of streaming audio is encoded in frames having a corresponding variable frame size, as depicted at 214. For example, if 12 ms of audio data is obtained from the buffer, the 12 ms of audio data is encoded in a frame having a 12 ms frame size (e.g., having 12 ms worth of audio samples).
[0107] As depicted at 216, the variable size frame is sent to the receiving device 220 in a network packet. The network packet is associated with a timestamp value having a fixed timestamp increment. For example, the frame can be transmitted in a network packet having a timestamp header value having a fixed increment of 20 ms since the previous frame. The network packet can be sent to the receiving device 220 via a streaming network protocol such as RTP or another streaming protocol.
[0108] The sending device 210 sends the network packets via the network 130 (e.g., the Internet and / or another type of public or private data network) to the receiving device 220. The receiving device 220 can be any type of computing device (e.g., a server computer, a virtual machine, a desktop computer, a laptop computer, a tablet computer, a phone, or another type of computing device) that receives and decodes streaming data (e.g., streaming audio and / or video).
[0109] As depicted at 222, a network packet is received from a sending device 210 via the network 130. The received network packet includes a received timestamp value and a variable size frame. The received timestamp value is a fixed increment timestamp value associated with the network packet or frame. For example, if the network protocol is RTP, the received timestamp value is the RTP header timestamp value (a multiple of 20ms or other fixed increment).
[0110] As depicted at 224, it is determined that a previous network packet has been lost (a previous network packet is the network packet that immediately precedes the received network packet). The determination that the previous network packet has been lost may be based on a packet sequence number (e.g., having received a current network packet with a given sequence number and determining that a previous network packet with a previous sequence number has not been received). It may be determined whether the lost network packet has not been received within a given time period.
[0111] As depicted at 226, a corrected timestamp range is calculated for the received network packet. For example, the corrected timestamp range can be calculated based at least in part on the received timestamp value and the possible buffer size range. In some embodiments, the corrected timestamp range is calculated using a technique similar to Equations 3, 4, and 5. The corrected timestamp range can be used (e.g., by a component of the receiving device 220, such as a decoder, a controller, a playback component, etc.) to control decoding and / or playback of the streaming audio (e.g., to control jitter buffer properties to manage latency).
[0112] In some embodiments, example environments 100 and 200 are online real-time communication environments, where audio and / or video information is captured at a sending device 110 or 210 and sent to a receiving device 120 or 220 for presentation (e.g., as part of a real-time audio and / or video call, conference, party, etc.). For example, the online communication environment may include a Voice over Internet Protocol (VoIP) service. As part of the communication, the receiving device 120 or 220 may also send audio and / or video back to the sending device 110 or 210 for presentation (e.g., as part of a two-way audio and / or video call). Additionally, many other computing devices may participate in the communication (e.g., in a conference environment).
[0113] Method for determining corrected timestamp range
[0114] In any of the examples herein, a method for determining a corrected timestamp range for streaming data can be provided.
[0115] Figure 5 is a flow chart of an example method 500 for determining a corrected timestamp range for streaming data. The streaming data may be sent, for example, via a streaming network protocol such as RTP. For example, the example method 500 may be performed by a computing device such as receiving device 120 or receiving device 220.
[0116] A network packet including a received timestamp value and a frame of streaming data is received at 510. The received timestamp value has a fixed timestamp increment (eg, an increment of 20 ms or some other fixed value).
[0117] At 520, it is determined that a previous network packet has been lost. For example, a time period for receiving the previous network packet has expired.
[0118] At 530, a possible buffer size range is calculated with respect to the received network packet. The possible buffer size range is for an encoding buffer used to encode the streaming data. In some embodiments, the possible buffer size range is calculated by calculating a propagated buffer size range based at least in part on a buffer size range for lost frames and a duration of received frames, and then calculating the intersection of the propagated buffer size range and a maximum buffer size range for received frames. In some embodiments, the maximum buffer size range is selected based on the type of streaming data. For example, different types of streaming data can have different maximum buffer size ranges (e.g., for audio, a first maximum buffer size range for voiced audio and a second maximum buffer size range for unvoiced audio).
[0119] At 540, a corrected timestamp range is calculated for the received network packet based at least in part on the received timestamp value and the possible buffer size range. In some embodiments, calculating the corrected timestamp range includes: adjusting the possible buffer size range based on the duration of the received frame; and calculating the corrected timestamp range based on the adjusted possible buffer size range. A particular corrected timestamp value (e.g., a minimum value in the range, a maximum value in the range, or some value therebetween) may then be selected from the corrected timestamp range. The corrected timestamp range and / or value may be used, for example, to adjust a jitter buffer.
[0120] Figure 6 6 is a flow chart of an example method 600 for determining a corrected timestamp range for streaming audio. The streaming audio may be sent, for example, via a streaming network protocol such as RTP. For example, the example method 600 may be performed by a computing device such as receiving device 120 or receiving device 220.
[0121] At 610, a network packet is received, the network packet including a received timestamp value and a frame of streaming audio. The received timestamp value has a fixed timestamp increment (eg, an increment of 20 ms or some other fixed value).
[0122] At 620, it is determined that a previous network packet has been lost. For example, a time period for receiving the previous network packet has expired.
[0123] At 630, a possible buffer size range is calculated with respect to the received network packet. The possible buffer size range is for an encoding buffer used to encode the streaming audio. In some embodiments, the possible buffer size range is calculated by calculating a propagated buffer size range based at least in part on a buffer size range for lost frames and a duration of received frames, and then calculating the intersection of the propagated buffer size range and a maximum buffer size range for received frames. In some embodiments, the maximum buffer size range is selected based on the type of streaming audio. For example, different types of streaming audio can have different maximum buffer size ranges (e.g., a first maximum buffer size range for voiced audio and a second maximum buffer size range for unvoiced audio).
[0124] At 640, a corrected timestamp range is calculated for the received network packet based at least in part on the received timestamp value and the possible buffer size range. In some embodiments, calculating the corrected timestamp range includes: adjusting the possible buffer size range based on the duration of the received frame; and calculating the corrected timestamp range based on the adjusted possible buffer size range. A specific corrected timestamp value (e.g., a minimum value in the range, a maximum value in the range, or some value therebetween) can then be selected from the corrected timestamp range. The corrected timestamp range and / or value can be used, for example, to adjust a jitter buffer.
[0125] Computing System
[0126] Figure 7 A general example of a suitable computing system 700 is depicted in which the described techniques may be implemented. Because the techniques may be implemented in a variety of general-purpose or special-purpose computing systems, computing system 700 is not intended to suggest any limitation as to scope of use or functionality.
[0127] refer to Figure 7 , the computing system 700 includes one or more processing units 710, 715 and memories 720, 725. Figure 7 , the basic configuration 730 is included within the dashed line. Processing units 710, 715 execute computer executable instructions. The processing unit may be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. The processing unit may also include multiple processors. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Figure 7A central processing unit 710 and a graphics processing unit or co-processing unit 715 are shown. Tangible memory 720, 725 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two accessible by the processing unit. The memory 720, 725 stores software 780 implementing one or more techniques described herein in the form of computer-executable instructions suitable for execution by the processing unit.
[0128] The computing system may have additional features. For example, the computing system 700 includes storage 740, one or more input devices 750, one or more output devices 760, and one or more communication connections 770. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of the computing system 700. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 700 and coordinates the activities of the components of the computing system 700.
[0129] Tangible storage 740 may be removable or non-removable and include disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information and that can be accessed within computing system 700. Storage 740 stores instructions for software 780 that implements one or more of the techniques described herein.
[0130] Input device(s) 750 may be a touch input device such as a keyboard, mouse, pen or trackball, voice input device, scanning device, or another device that provides input to computing system 700. For video encoding, input device 750 may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into computing device 700. Output device 760 may be a display, printer, speaker, CD burner, or other device that provides output from computing system 700.
[0131] Communication connection 770 enables communication with another computing entity via a communication medium. The communication medium transmits information such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has information encoded in the signal in a manner that sets or changes one or more of its characteristics. By way of example and not limitation, the communication medium may use electrical, optical, RF or other carriers.
[0132] The technology may be described in the general context of computer executable instructions, such as computer executable instructions contained in program modules, which are executed in a computing system on a target real or virtual processor. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between various program modules as needed. Computer executable instructions for program modules may be executed within a local or distributed computing system.
[0133] The terms "system" and "device" are used interchangeably. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and / or general-purpose hardware and software that implements the functionality described herein.
[0134] For ease of explanation, terms such as "determine" and "use" are used in the detailed description to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.
[0135] mobile device
[0136] Figure 8 800 includes various optional hardware and software components shown generally at 802. Any component 802 in the mobile device can communicate with any other component, although not all connections are shown for ease of illustration. The mobile device can be any of a variety of computing devices (e.g., a cell phone, a smart phone, a handheld computer, a personal digital assistant (PDA), etc.), and can enable wireless two-way communication with one or more mobile communication networks 804 (e.g., a cellular network, a satellite network, or other networks).
[0137] The illustrated mobile device 800 may include a controller or processor 810 (e.g., a signal processor, microprocessor, ASIC, or other control and processing logic circuit) for performing tasks such as signal encoding, data processing, input / output processing, power control, and / or other functions. An operating system 812 may control the allocation and use of components 802 and support one or more applications 814. The applications may include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications), or any other computing applications. A function 813 for accessing an application store may also be used to obtain and update applications 814.
[0138] The mobile device 800 shown may include a memory 820. The memory 820 may include a non-removable memory 822 and / or a removable memory 824. The non-removable memory 822 may include RAM, ROM, flash memory, a hard disk, or other well-known memory storage technologies. The removable memory 824 may include a flash memory or a subscriber identity module (SIM) card well-known in the GSM communication system, or include other well-known memory storage technologies such as "smart cards". The memory 820 may be used to store data and / or code for running an operating system 812 and applications 814. Example data may include web pages, text, images, sound files, video data, or other data sets to be sent to one or more network servers or other devices and / or received from one or more network servers or other devices via one or more wired or wireless networks. The memory 820 may be used to store subscriber identifiers, such as an international mobile subscriber identity (IMSI), and device identifiers, such as an international mobile equipment identifier (IMEI). Such identifiers may be sent to a network server to identify users and devices.
[0139] The mobile device 800 may support one or more input devices 830, such as a touch screen 832, a microphone 834, a camera 836, a physical keyboard 838, and / or a trackball 840, and one or more output devices 850, such as a speaker 852 and a display 854. Other possible output devices (not shown) may include piezoelectric or other tactile output devices. Some devices may serve more than one input / output function. For example, a touch screen 832 and a display 854 may be combined in a single input / output device.
[0140] The input device 830 may include a natural user interface (NUI). A NUI is any interface technology that enables a user to interact with a device in a "natural" way, without artificial constraints imposed by input devices such as a mouse, keyboard, remote control, etc. Examples of NUI methods include those that rely on voice recognition, touch and stylus recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence. Other examples of NUIs include motion gesture detection using accelerometers / gyroscopes, facial recognition, 3D displays, head, eye and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide a more natural interface, and technology for sensing brain activity using electric field sensing electrodes (EEG and related methods). Therefore, in a specific example, the operating system 812 or application 814 may include voice recognition software as part of a voice user interface that allows a user to operate the device 800 via voice commands. In addition, the device 800 may include input devices and software that allow user interaction via a user's spatial gestures, such as detecting and interpreting gestures to provide input to a gaming application.
[0141] The wireless modem 860 may be coupled to an antenna (not shown) and may support bidirectional communications between the processor 810 and external devices, as is well known in the art. The modem 860 is shown generically and may include a cellular modem for communicating with the mobile communication network 804 and / or other radio-based modems (e.g., Bluetooth 864 or Wi-Fi 862). The wireless modem 860 is typically configured to communicate with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between a mobile device and a public switched telephone network (PSTN).
[0142] The mobile device may further include at least one input / output port 880, a power source 882, a satellite navigation system receiver 884 such as a global positioning system (GPS) receiver, an accelerometer 886, and / or a physical connector 890, which may be a USB port, an IEEE 1394 (Firewire) port, and / or an RS-232 port. The components 802 shown need not be or are not all included, as any components may be deleted and other components may be added.
[0143] Cloud Support Environment
[0144] Fig. 9A general example of a suitable cloud support environment 900 in which the described embodiments, methods, and techniques may be implemented is shown. In the example environment 900, a cloud 910 provides various types of services (e.g., computing services). For example, the cloud 910 may include a collection of computing devices, which may be centrally located or distributed, that provide cloud-based services to various types of users and devices connected via an internet (e.g., the Internet). The implementation environment 900 may be used in different ways to complete computing tasks. For example, some tasks (e.g., processing user input and presenting a user interface) may be performed on a local computing device (e.g., connected devices 930, 940, 950), while other tasks (e.g., storing data to be used in subsequent processing) may be performed in the cloud 910.
[0145] In the example environment 900, the cloud 910 provides services for connected devices 930, 940, 950 with various screen capabilities. The connected device 930 represents a device with a computer screen 935 (e.g., a medium-sized screen). For example, the connected device 930 can be a personal computer, such as a desktop computer, a laptop computer, a notebook computer, a netbook, etc. The connected device 940 represents a device with a mobile device screen 945 (e.g., a small-sized screen). For example, the connected device 940 can be a mobile phone, a smart phone, a personal digital assistant, a tablet computer, etc. The connected device 950 represents a device with a large screen 955. For example, the connected device 950 can be a TV screen (e.g., a smart TV) or another device connected to the TV (e.g., a set-top box or a game console) or a similar device. One or more of the connected devices 930, 940, 950 may include touch screen functionality. The touch screen can accept input in different ways. For example, a capacitive touch screen detects touch input when an object (e.g., a fingertip or a stylus) distorts or interrupts the current running on the surface. As another example, a touch screen can use an optical sensor to detect touch input when a light beam from the optical sensor is interrupted. For some touch screen detected inputs, physical contact with the screen surface is not required. Devices without screen capabilities can also be used in example environment 900. For example, cloud 910 can provide services for one or more computers (e.g., server computers) without displays.
[0146] Services may be provided by cloud 910 through service provider 920 or through other providers of online services (not shown). For example, cloud services may be customized for the screen size, display capabilities, and / or touch screen capabilities of a particular connected device (e.g., connected devices 930, 940, 950).
[0147] In example environment 900, cloud 910 provides the techniques and solutions described herein to various connected devices 930, 940, 950, at least in part using service provider 920. For example, service provider 920 can provide a centralized solution for various cloud-based services. Service provider 920 can manage service subscriptions for users and / or devices (e.g., for connected devices 930, 940, 950 and / or their respective users).
[0148] Example Implementations
[0149] Although the operations of some disclosed methods are described in a specific, sequential order for ease of presentation, it should be understood that such description includes rearrangement unless the specific language set forth below requires a specific order. For example, in some cases, the operations described in sequence may be rearranged or performed simultaneously. In addition, for simplicity, the accompanying drawings may not show the various ways in which the disclosed methods can be used in conjunction with other methods.
[0150] Any disclosed method may be implemented as computer executable instructions or computer program products stored on one or more computer readable storage media and executed on a computing device (i.e., any available computing device, including a smartphone or other mobile device containing computing hardware). A computer readable storage medium is a tangible medium (one or more optical media, such as a DVD or CD, a volatile memory (such as DRAM or SRAM), or a non-volatile memory (such as flash memory or a hard drive)) that can be accessed in a computing environment. By way of example and with reference to Figure 7 , the computer readable storage medium includes memories 720 and 725 and storage device 740. By way of example and with reference to Figure 8 , computer readable storage media include memory and storage devices 820, 822, and 824. The term computer readable storage medium does not include signals and carrier waves. In addition, the term computer readable storage medium does not include communication connections, such as 770, 860, 862, and 864.
[0151] Any computer executable instructions for implementing the disclosed techniques and any data created and used during the implementation of the disclosed embodiments can be stored on one or more computer readable storage media. The computer executable instructions can be, for example, part of a dedicated software application or a software application (e.g., a remote computing application) accessed or downloaded via a web browser or other software application. Such software can be executed using one or more network computers, for example, on a single local computer (e.g., any suitable commercial computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (e.g., a cloud computing network), or other such networks).
[0152] For the sake of clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C++, Java, Perl, or any other suitable programming language. Likewise, the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well known and need not be elaborated in this disclosure.
[0153] In addition, any software-based embodiment (e.g., including computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via appropriate communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including optical cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.
[0154] The disclosed methods, apparatus, and systems should not be limited in any way. Rather, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and subcombinations with each other. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the embodiments of the present disclosure require the presence of any one or more specific advantages or problems to be solved.
[0155] The technology from any example can be used in combination with the technology described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology.
Claims
1. A computing device, comprising: processor; Internet connection; as well as Memory; The computing device is configured to perform, via computer-executable instructions, operations for determining a corrected timestamp range for streaming data, wherein the streaming data is encoded using frames having a variable frame size, the operations comprising: receiving, via the network connection, a network packet comprising a received timestamp value and a frame of streaming data, wherein the received timestamp value has a fixed timestamp increment; determining that a previous network packet including a previous frame of streaming data has been lost; Calculating a possible buffer size range for the received network packet by: calculating a propagated buffer size range based at least in part on a buffer size range for lost frames and a duration of the received frames, and calculating an intersection of the propagated buffer size range and a maximum buffer size range for the received frames, wherein the possible buffer size range is for an encoding buffer used to encode the streaming data; and A corrected timestamp range for the received network packet is calculated based at least in part on the received timestamp value and the possible buffer size range.
2. The computing device of claim 1, the operations further comprising: A jitter buffer is adjusted based on the corrected timestamp range, wherein the jitter buffer is used to buffer network packets including frames of the streaming data, the network packets being received from a sending computing device via the network connection.
3. The computing device of claim 1, wherein: The streaming data is received via a Real-time Transport Protocol (RTP), and wherein the received timestamp value is an RTP timestamp value.
4. The computing device of claim 1, wherein: Calculating the possible buffer size range includes: determining a type of streaming data for the received frame based on a plurality of supported types of streaming data, wherein each of the plurality of supported types of streaming data has a different maximum buffer size range; Based on the determined type of streaming data, selecting a corresponding maximum buffer size range; calculating a propagated buffer size range based at least in part on the buffer size range for lost frames and a duration of received frames; and An intersection of the propagated buffer size range and the selected maximum buffer size range for the received frame is calculated.
5. The computing device of claim 1, wherein: The streaming data includes streaming audio data, and wherein the streaming audio data is encoded at least in part using pitch periods, wherein each of one or more frames contains an integer number of pitch periods.
6. The computing device of claim 1, wherein: Calculating the corrected timestamp range includes: The received timestamp value is adjusted by the fixed timestamp increment, the duration of the received frame, and the range of possible buffer sizes.
7. The computing device of claim 1, wherein: Calculation of the corrected timestamp range is performed according to the following equation: Corrected timestamp range = received timestamp value + fixed timestamp increment - received frame duration - possible buffer size range.
8. The computing device of claim 1, wherein: Calculation of the possible buffer size range is performed according to the following equation: Possible buffer size range = buffer size range for lost frames + fixed timestamp increment - duration of received frames ∩ maximum buffer size range for received frames.
9. A method implemented by a computing device for determining a corrected timestamp range for streaming audio, wherein: The streaming audio is encoded using frames having a variable frame size, and the method comprises: receiving a network packet including a received timestamp value and a frame of streaming audio, wherein the received timestamp value has a fixed timestamp increment; determining that a previous network packet including a previous frame of streaming audio has been lost; Calculating a possible buffer size range for the received network packet by: calculating a propagated buffer size range based at least in part on a buffer size range for lost frames and a duration of the received frames, and calculating an intersection of the propagated buffer size range and a maximum buffer size range for the received frames, wherein the possible buffer size range is for an encoding buffer used to encode the streaming audio; and A corrected timestamp range for the received network packet is calculated based at least in part on the received timestamp value and the possible buffer size range.
10. The method according to claim 9, further comprising: A jitter buffer is adjusted based on the corrected timestamp range, wherein the jitter buffer is used to buffer network packets including frames of the streaming audio, the network packets being received from a sending computing device via a network connection.
11. The method according to claim 9, wherein: The streaming audio is received via a Real-time Transport Protocol (RTP), and wherein the received timestamp value is an RTP timestamp value.
12. The method according to claim 9, wherein: Calculating the possible buffer size range includes: Determining a type of streaming audio for a received frame according to multiple supported types of streaming audio, wherein the multiple types of streaming audio include: a voiced speech type having a first maximum buffer size range; and an unvoiced type having a second maximum buffer size range different from the first maximum buffer size range; Based on the determined type of streaming audio, selecting a corresponding maximum buffer size range; calculating a propagated buffer size range based at least in part on the buffer size range for lost frames and a duration of received frames; and An intersection of the propagated buffer size range and the selected maximum buffer size range for the received frame is calculated.
13. A method implemented by a computing device for determining a corrected timestamp range for streaming audio, wherein: The streaming audio is encoded using frames having a variable frame size, and the method comprises: receiving a network packet including a received timestamp value and a frame of streaming audio, wherein the received timestamp value has a fixed timestamp increment; determining that a previous network packet including a previous frame of streaming audio has been lost; Calculating a possible buffer size range for the received network packet by: calculating a propagated buffer size range based at least in part on a buffer size range for lost frames and a duration of the received frames, and calculating an intersection of the propagated buffer size range and a maximum buffer size range for the received frames, wherein the possible buffer size range is for an encoding buffer used to encode the streaming audio; calculating a corrected timestamp range for the received network packet based at least in part on the received timestamp value and the possible buffer size range; and A jitter buffer for storing received network packets of the streaming audio is adjusted based on the corrected timestamp range.
