Method, apparatus, and device for deadline-sensitive streaming

Abstract

The application provides a deadline-sensitive streaming method, device and equipment, and solves the technical problem of inaccurate real priority calculation of data blocks in DTP. The application obtains real packet loss conditions by distinguishing between congestion-type packet loss and non-congestion-type packet loss when performing link congestion control, thereby avoiding the influence of link inherent packet loss / random packet loss on deadline-sensitive streaming. When calculating the real priority of data blocks, the influence factors of link delay fluctuation on priority are reduced on the basis of the original DTP protocol, so that the calculation of the real priority is more accurate. In the case where there is a dependency relationship between data blocks, the influence factor of the number of dependent data blocks on priority is introduced, the influence degree of the dependency relationship on the priority of the current data block is adjusted through an adjustment coefficient, so that the calculation of the priority is more accurate and more applicable to application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data communication, and particularly relates to a deadline-sensitive flow transmission method, device and equipment. BACKGROUND

[0002] Streaming is a method of data transmission used by people when watching videos on the Internet. It is a way of delivering a small amount of video file at a time, usually from a remote storage location. By transmitting a few seconds of file over the Internet at a time, the client device does not have to download the entire video before starting to play. Live streaming refers to streaming media video that is sent over the Internet in real time, without being recorded and stored first. Today, television broadcasts, video game streams and social media videos can all be streamed in real time. The difference between regular streaming and live streaming can be compared to the difference between an actor reciting a monologue and improvising a speech. In the former, the content is created beforehand, stored and relayed to the audience. In the latter, the audience receives the content at the same time as the actor creates it, just as in live streaming.

[0003] With the popularity of a large number of emerging applications such as virtual reality (VR), augmented reality (AR), autonomous driving, ultra-low latency games, and large-scale Internet of Things systems, compared with traditional network-intensive applications (such as multimedia streaming), the above-mentioned emerging applications show several unique differences in terms of network traffic workload and quality of service (QoE) requirements, the most obvious of which is that the applications have certain requirements for priority and deadlines. For example, when a VR server estimates a more accurate head pose position message, or when an autonomous driving car updates an object's movement trajectory, distance or speed message, a frame in a video stream and a message in an online game, these messages are transmitted by the application in the form of data blocks. For block transmission, the order in which data arrives within a block is not important, as long as all the data of a certain block reaches the receiver before the deadline of the block. Blocks that miss the deadline will greatly affect the user experience, and when new data arrives, the old block becomes useless due to timeliness, even if it arrives later.

[0004] Deadline-aware Transport Protocol (DTP) is a protocol developed and researched based on QUIC (Quick UDP Internet Connections) transport protocol, which is used in real-time video streaming scenarios and belongs to the extension of QUIC protocol. Its use scenario is to meet the demand of completing block transmission before the deadline. Taking watching live broadcast as an example, the deadline and the block refer to: the delay that actually needs to be completed within a short time (such as 1 second) for video picture transmission is the deadline, and if this time is exceeded, even if the picture is transmitted, the effect on live broadcast is also great; and the block is equivalent to a video frame, control signaling, which is a single, controllable and independently usable data component.

[0005] When the data block and the corresponding metadata (deadline, frame priority) are transmitted from the application layer to the transport layer, they are placed in the buffer, and the scheduler selects the data block to be sent according to the preset scheduling algorithm. Then the selected data block is cut into QUIC format packets for transmission. Among them, the congestion control module is responsible for sending QUIC packets, collecting ACK, and detecting packet loss. If packet loss occurs, it will put the lost QUIC packet back into the retransmission queue RTX, and notify the scheduler of the detected network state such as bandwidth and RTT, so that it can make decisions in the next packet transmission process.

[0006] The most critical part of the DTP protocol is the decision-making process of the scheduler on which data block to send: DTP selects to make decisions from the three angles of data block original priority, block deadline, and time required to complete the remaining block transmission. Because if only the original priority of the data block is considered for decision-making, the scheduler selects a block with high priority at a certain moment, but at this time, the data block with low priority is very close to the deadline, and sending a block with high priority will cause the low-priority block to timeout. At this time, if the low-priority block is sent first and then the high-priority block, it may be able to ensure that the former does not timeout. In addition, when the low-priority block is about to be transmitted, another high-priority block is suddenly selected for transmission, and the former may timeout and have to trigger retransmission, resulting in the waste of bandwidth previously used to transmit the low-priority block.

[0007] Based on the above analysis, the scheduler of DTP uses the following strategy: assuming that the original priority of a block is priority (the higher the priority, the smaller the value of priority), the deadline is deadline, the remaining block size is remaining_size, the link bandwidth is bandwidth, and the link delay is RTT (Round Trip Time), DTP calculates the real priority of the data block by the following formula:

[0008]

[0009] Real Priority represents the final calculated data block real priority. In the formula, priority represents the original priority of the data block, RTT / 2 represents the inherent one-way delay of the current link, (remaining_size) / bandwidth represents the time required to complete the transmission of the remaining data block at the current link bandwidth, and deadline represents the deadline for the data block to be transmitted within the given deadline. The difference (absolute value) between the deadline and the sum of RTT / 2 and (remaining_size) / bandwidth represents the gap between the time required to complete the transmission of the remaining data block and the given deadline. In the case where the original priorities of two data blocks are the same, if the gap corresponding to a certain data block is smaller, it means that the calculated real priority value will also be smaller, and the real priority will be higher, indicating that the data block needs to be transmitted as soon as possible; if the calculated gap of the two data blocks is the same, then the real priority is determined by the original priority of the data block. Therefore, multiplying the original priority of the data block by the gap between the time required to complete the transmission of the remaining data block and the given deadline can more comprehensively consider the real priority of the data block.

[0010] Although DTP uses QUIC as the transport layer protocol and considers the real priority of the data block from multiple aspects, the congestion control algorithm used does not consider factors such as the impact of packet loss type on the sending rate, resulting in a certain degree of distortion in the calculated sending priority of the data block, thereby affecting the transmission quality. SUMMARY

[0011] Therefore, the present application provides a deadline-sensitive streaming method, device and equipment, which solves the technical problem of inaccurate calculation of the real priority of the data block in DTP.

[0012] Based on an aspect of an embodiment of the present application, the present application provides a deadline-sensitive streaming method, which is applied to a streaming device using a deadline-sensitive protocol DTP, and the method comprises:

[0013] When performing path congestion control, distinguish between congestion packet loss and non-congestion packet loss, calculate the current path sending rate (sending rate) based on the real congestion packet loss rate and perform corresponding congestion control; the congestion packet loss refers to the packet loss caused by the fullness of the transport layer buffer;

[0014] The real priority of the data block to be sent is calculated according to the current path sending rate determined by the path congestion control module and the original priority of the data block to be sent, and whether the data block to be sent is sent is decided based on the real priority.

[0015] Further, the method for distinguishing the congestion packet loss from the non-congestion packet loss is:

[0016] In the case that the link round-trip delay is measured for three times continuously, if packet loss occurs in the process of measuring the link round-trip delay for the third time, the packet loss is determined to be the congestion packet loss when the following two conditions are met:

[0017] Condition one, the round-trip delay measured for the second time is within the preset fluctuation range of the round-trip delay measured for the first time, and the round-trip delay measured for the second time is greater than the known minimum round-trip delay of the link; and

[0018] Condition two, the round-trip delay measured for the third time is within the preset fluctuation range of the round-trip delay measured for the second time, and the round-trip delay measured for the third time is greater than the known minimum round-trip delay of the link.

[0019] The preset fluctuation range of the round-trip delay is determined by adding and subtracting a preset fluctuation coefficient to the round-trip delay measured for the last time.

[0020] Further, the method for calculating the real priority of the data block to be sent is:

[0021] The deadline of the data block after being sent is subtracted by the ratio of the remaining block size and the current sending rate, and then subtracted by the one-way delay (RTT / 2) and the delay fluctuation influence value to obtain the remaining transmission time of the data block to be sent;

[0022] The product of the original priority and the remaining transmission time is taken as the real priority.

[0023] Further, the method for calculating the real priority of the data block to be sent is:

[0024] After the product of the original priority and the remaining transmission time is calculated, it is multiplied by the priority adjustment coefficient α, and then the product of the block dependent weight adjustment coefficient β and the reciprocal of the dependent block number is added to obtain the final real priority.

[0025] Based on another aspect of the embodiments of the present application, the present application further provides a deadline-sensitive flow transmission method, which is applied to a flow sending device using a deadline-sensitive protocol DTP, and the method comprises the following steps:

[0026] The deadline of the data block to be sent is reduced by the ratio of the remaining size and the current sending rate, and then is further reduced by the one-way delay (RTT / 2) and the delay fluctuation influence value, so as to obtain the remaining transmission time of the data block to be sent.

[0027] The product of the original priority and the remaining transmission time is taken as the real priority, and whether to send the data block to be sent is decided based on the real priority.

[0028] Further, the method for calculating the real priority of the data block to be sent comprises the following steps:

[0029] After the product of the original priority and the remaining transmission time is obtained, the product is multiplied by the priority adjustment coefficient α, and then the product of the block dependent weight adjustment coefficient β and the reciprocal of the dependent block number is added, so as to obtain the final real priority.

[0030] Based on another aspect of the embodiments of the present application, the present application further provides a deadline-sensitive flow transmission method, which is applied to a flow sending device using a deadline-sensitive protocol DTP, and the method comprises the following steps:

[0031] The product of the original priority and the remaining transmission time is multiplied by the priority adjustment coefficient α, and then the product of the block dependent weight adjustment coefficient β and the reciprocal of the dependent block number is added, so as to obtain the real priority.

[0032] Whether to send the data block to be sent is decided based on the real priority.

[0033] Further, the method for calculating the real priority of the data block to be sent comprises the following steps:

[0034] The deadline of the data block to be sent is reduced by the ratio of the remaining size and the current sending rate, and then is further reduced by the one-way delay (RTT / 2) and the delay fluctuation influence value The remaining transmission time of the data block to be sent is obtained.

[0035] Based on another aspect of the embodiment of the present application, the present application further provides a deadline-sensitive flow transmission device, which is applied to a flow sending device using a deadline-sensitive protocol DTP, and the device comprises:

[0036] a congestion control module, which is used for distinguishing congestion packet loss and non-congestion packet loss when performing path congestion control, calculating a current path sending rate based on a real congestion packet loss rate, and performing corresponding congestion control; the congestion packet loss refers to packet loss caused by fullness of a transmission layer buffer;

[0037] a decision module, which is used for obtaining a remaining transmission time of a data block to be sent by subtracting a ratio of a remaining size and a current sending rate from a deadline of the data block to be sent, and then subtracting a one-way delay (RTT / 2) and a delay fluctuation influence value obtaining a final real priority by multiplying a product of an original priority and the remaining transmission time by a priority adjustment coefficient α, and then adding a product of a block dependent weight adjustment coefficient β and an inverse of a dependent block number.

[0038] The present application obtains real packet loss by distinguishing congestion type packet loss and non-congestion type packet loss when performing link congestion control, thereby avoiding influence of link inherent packet loss / random packet loss on deadline-sensitive flow transmission. When calculating a real priority of a data block, the influence of link delay fluctuation on the priority is reduced on the basis of the original DTP protocol, so that the calculation of the real priority is more accurate. In the case that there is a dependent relationship between data blocks, an influence factor of a dependent data block number on the priority is introduced, the influence degree of the dependent relationship on the priority of the current data block is adjusted through an adjustment coefficient, so that the calculation of the priority is more accurate and more applicable to application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0040] Figure 1 A step flowchart of a deadline-sensitive flow transmission method provided by an embodiment of the present application;

[0041] Figure 2 A diagram of the dependency relationship between a group of pictures and various types of frames in the H.265 protocol;

[0042] Figure 3 A structural schematic diagram of an electronic device for implementing the deadline-sensitive streaming method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0043] The terms used in the embodiments of the present application are merely for the purpose of describing particular embodiments and are not intended to limit the embodiments of the present application. The singular forms "a," "an," and "the" used in the embodiments of the present application are intended to include the plural forms, unless the context clearly indicates otherwise. Although the embodiments of the present application can employ a manner such as first, second, third to describe, such a description manner is only for distinguishing similar information, entities or steps, and is not intended to describe a particular order or sequence. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the embodiments of the present application. For another example, in some scenarios, the first information can refer to one information, or refer to multiple information of the same type. In addition, the word "if" used in the embodiments of the present application can be interpreted as "when" or "when" or "in response to determining". The "and / or" in the present application is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present application, "multiple" means two or more than two, unless otherwise specified. "At least one of the following" or the like means any combination of the items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0044] The inventors found through research and analysis that the existing DTP protocol DTP uses QUIC as a transport layer protocol and considers the real priority of data blocks from multiple aspects, but the congestion control algorithm used has some technical defects as follows:

[0045] DTP uses the bottleneck bandwidth and round-trip propagation time (BBR) congestion control algorithm by default, which does not distinguish between inherent packet loss / random packet loss (the reasons for such packet loss often come from below the transport layer, which may be a routing problem, or the channel quality itself may be problematic) and real packet loss (caused by unreasonable packet sending of the transport layer itself), which will affect the sending rate of the data block in DTP, and in turn affect the real-time transmission effect (user viewing experience).

[0046] DTP uses half of the RTT (RTT / 2) as the one-way transmission delay to calculate the priority, and does not consider the influence of the delay fluctuation that may occur in the network on the block transmission completion time (for example, the link quality used when accessing a mobile terminal to play an online video is often unstable), causing a certain degree of distortion in the calculated real priority.

[0047] DTP does not consider the coupling between application layer data blocks, for example, in real-time video streams encoded in the H.265 protocol, there is a coding dependency relationship between different video frames (I frames / P frames / B frames), which will also affect the real priority of the data block.

[0048] To solve the above technical problems, the present application provides a deadline-sensitive video stream transmission scheme, and the specific implementation process of the present application is described in detail below in combination with the drawings and specific embodiments. It should be noted that the steps shown in the flowchart and the embodiment can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Figure 1 The step flowchart of the deadline-sensitive stream transmission method provided by an embodiment of the present application. The deadline-sensitive stream transmission method provided by the present application is still implemented based on the overall architecture of the DTP protocol, and the deadline-sensitive stream transmission process is optimized at multiple levels from the perspectives of detecting real packet loss, evaluating delay fluctuation, and evaluating inter-frame coding dependency. The following embodiments of the present application exemplarily describe time-sensitive video streams, but based on the same basic idea, the technical solutions of the present application are also applicable to any deadline-sensitive data stream with similar stream characteristics.

[0050] Step 101. When performing path congestion control, distinguish between congestion packet loss and non-congestion packet loss, calculate the current path sending rate based on the real congestion packet loss rate, and perform corresponding congestion control; the congestion packet loss refers to the packet loss caused by fullness of the transport layer buffer.

[0051] DTP uses BBR as a transport layer congestion control algorithm for path congestion control. BBR algorithm uses bottleneck bandwidth and round-trip time (RTT) to optimize congestion control. The bottleneck bandwidth refers to the part of the network path with the lowest transmission bandwidth. RTT is the round-trip delay time of data packets in the network. The BBR congestion control algorithm adjusts the congestion window according to the equivalent bandwidth of the bottleneck bandwidth and the round-trip time of the network to achieve more efficient data transmission. It estimates the bottleneck bandwidth and RTT of the network path, and dynamically adjusts the sending rate to make effective use of network bandwidth and avoid congestion.

[0052] Since the BBR congestion control module calculates the bottleneck bandwidth and RTT of the network path according to the packet loss of the network and reports it to the scheduler of the DTP protocol, the scheduler schedules and sends the data blocks in the buffer according to the preset scheduling strategy based on the current path bandwidth and RTT values reported by the BBR congestion control module, therefore, in the process of real-time video stream transmission sensitive to deadline, it is very important and has a great influence to accurately distinguish between congestion packet loss caused by full transmission layer buffer and non-congestion packet loss for the BBR congestion control module to accurately estimate the current path bandwidth and adjust the sending rate. The non-congestion packet loss refers to the packet loss caused by non-transmission layer reasons (such as link inherent packet loss and / or random packet loss caused by routing problems, channel quality problems, etc.).

[0053] Since the packet loss based on BBR algorithm often occurs in the buffer limited stage, that is, the excessive increase of in-flight packets causes the intermediate device buffer to be completely occupied and then packet loss occurs. The characteristics of this stage are that the bandwidth and real-time round-trip time RTT are in a stable state (even if there is fluctuation, it is very small). In the bandwidth limited stage when the buffer is occupied but not completely occupied, although the bandwidth is also in a stable state (the bandwidth is the bottleneck bandwidth as in the buffer limited stage), the RTT will change with the degree of buffer occupation.

[0054] Based on the above analysis, and considering that BBR algorithm almost always works in the above two stages, in order to obtain the congestion packet loss rate in the link that can truly reflect the congestion condition of the link, the application sets a fluctuation coefficient γ for the BBR congestion control module, and distinguishes the type of packet loss by judging the conditional relationship between the three times of measured packet round-trip time RTT i , RTT i+1 and RTT i+2 , so as to more accurately calculate the real congestion packet loss rate.

[0055] The judgment method used in this embodiment is as follows:

[0056]

[0057] In the above formula, RTT i For the first measured round-trip time, RTT i+1 For the second measured round-trip time, RTT i+2 This is the third round-trip time (RTT) measurement. i -γ and RTT i +γ is used to limit the RTT measured in the second measurement. i+1 The lower and upper limits of the fluctuation range, RTT i+1 -γ and RTT i+1 +γ represent the RTT measured in the third measurement. i+2 The lower and upper limits of the fluctuation range. stminRTT <RTT i+1 This indicates that the round-trip time measured the second time is greater than the known minimum round-trip time of the link detected by the BBR. Similarly, stminRTT <RTT i+2 This indicates that the round-trip delay measured for the third time is greater than the known minimum link delay detected by the BBR. The value of γ needs to be determined based on the actual link conditions. If the link's delay fluctuates significantly, a larger γ value should be selected; conversely, a smaller γ value should be selected. The meaning of the second line of the above formula is similar to that of the first line; please refer to the explanation of the meaning of the first line.

[0058] The BBR congestion control algorithm detects and records the inherent latency of the link during the latency probing phase. If the previously detected minimum RTT does not update within a preset time (e.g., 10 seconds), BBR will forcibly switch to the latency probing phase, regardless of the previous phase. Once in the latency probing phase, BBR limits the sending window size CWND to a very small value (e.g., sending only 4 packets) and maintains this for a period of time (e.g., at least 200 milliseconds) to ensure that this flow does not cause packet backlog on devices in the link, thereby probing the inherent round-trip latency of the link.

[0059] Based on Formulas 1 and 2 above, the method of this invention for distinguishing between congestion packet loss and non-congestion packet loss is as follows: If packet loss occurs during the third measurement of link round-trip time (RTT) after three consecutive measurements, the method determines whether the packet loss is congestion packet loss by judging whether the following two conditions are met:

[0060] 1) When the round-trip delay measured for the second time is within the preset fluctuation range of the round-trip delay measured for the first time, and the round-trip delay measured for the second time is greater than the known minimum round-trip delay of the link;

[0061] 2) when the third measured round-trip delay is within the preset fluctuation range of the second measured round-trip delay, and the third measured round-trip delay is greater than the known link minimum round-trip delay.

[0062] The preset fluctuation range of the round-trip delay is a range determined by adding and subtracting a fluctuation coefficient γ to the last measured round-trip delay.

[0063] The BBR congestion control module calculates the real congestion packet loss rate of the current path based on the measured real congestion packet loss quantity only when the above two conditions are met. The BBR congestion control module can more accurately calculate the bandwidth of the current path according to the real packet loss rate, thereby being able to more effectively perform congestion control, such as adjusting the sending window size and the sending rate, etc.

[0064] Step 102. Calculate the real priority (Real Priority) of the to-be-sent data block according to the current path sending rate (sending rate) determined by the path congestion control module and the original priority (Priority) of the to-be-sent data block, and decide whether to send the to-be-sent data block based on the real priority.

[0065] After the DTP protocol obtains the current path sending rate (sending rate) reported by the congestion control module, the real priority (Real Priority) of the to-be-sent data block can be calculated based on the way in which the DTP protocol calculates the real priority of the data block, and the calculation method is as follows:

[0066]

[0067] In the above formula 3, deadline is the deadline of the to-be-sent data block, remaining_size is the remaining block size of the to-be-sent data block, RTT is the link round-trip delay, and priority is the original priority of the to-be-sent data block.

[0068]

Optimization of remaining transmission time calculation

[0069] In another embodiment of the present application, the calculation process of the DTP protocol for calculating the remaining transmission time (remaining time) of the to-be-sent data block is also optimized. On the basis of the original DTP calculation of the remaining transmission time of the video frame, the influence value of the fluctuation of the link delay is further reduced, thereby making the remaining transmission time of the data block calculated by the DTP protocol more accurate.

[0070] The network in the process of real-time video streaming is likely to be unstable (especially with the popularity of mobile Internet, many users start to watch videos with mobile phones, and the WIFI or cellular network used often has poor link quality), so in the process of transmitting deadline-sensitive video stream data, considering the influence of link delay fluctuation can more accurately calculate the remaining transmission time.

[0071] Based on the above analysis, in the process of calculating the remaining transmission time of a certain video frame, in addition to considering the current detected link one-way delay (RTT / 2), the influence of delay fluctuation also needs to be considered, so the calculation of one-way delay standard deviation is added when calculating the remaining transmission time of the data block, so that the calculation of the remaining transmission time (remaining time) of a certain video frame / data block is more accurate and safe:

[0072]

[0073] Wherein, remaining time is the remaining transmission time of a certain video frame / data block, deadline is the deadline for sending the current data block, remaining size is the remaining block size, sending rate is the current sending rate. RTT is the link round-trip time, RTT / 2 is the link one-way delay. stddev is the standard deviation of the round-trip time RTT, stddev / 2 is the standard deviation of the one-way delay. stddev reflects the delay fluctuation of the link, k is the standard deviation adjustment coefficient in [0, 1], which determines the extent to which the one-way delay standard deviation should affect the calculation of the remaining transmission time. On the link with large delay fluctuation, k should take a larger value within the limited range, and vice versa.

[0074] After calculating the remaining transmission time (remaining time) of the data block to be sent, the DTP protocol can calculate the real priority of the data block to be sent based on the following formula 5, that is, the real priority of the data block to be sent is the product of the original priority and the remaining transmission time, so as to decide whether to send the data block to be sent according to the calculated real priority.

[0075] Real Priority=priority×remaining time (Formula 5)

[0076] In this embodiment, the DTP considers the influence of the delay fluctuation in the network on the block transmission completion time when calculating the real priority, so that the calculated real priority is more close to the real value.

[0077]

Optimization of real priority based on coupling relationship between data blocks

[0078] This embodiment takes the video coding standard H.265 as an example to illustrate the optimization method of real priority based on the coupling relationship between data blocks. Figure 2 The figure shows the dependence relationship between the image group and each type of frame in H.265 protocol. H.265 protocol defines three types of frames: I frame, P frame and B frame. I frame is also called intra-frame coding frame, which completely preserves the current picture, and only this frame is needed for decoding; P frame, also called forward prediction coding frame, represents the difference between this frame and a previous key frame (or P frame), and the final picture is generated based on the previously cached picture and the difference defined by this frame; B frame, also called bidirectional prediction interpolation coding frame, simply records the difference between this frame and the previous and next frames.

[0079] That is, to decode the B frame, not only the previous cached picture is needed, but also the picture after it, and the final picture is obtained by superimposing the data of the previous and next pictures with the data of the current frame. The basic unit of video stream coding and decoding is the image group (GOP), which consists of an I frame and a number of P frames and B frames. Without I frame, P frame and B frame cannot be decoded; without P frame with a lower sequence number, P frame with a higher sequence number cannot be decoded.

[0080] Based on the above analysis, in addition to the priority of I frame, P frame and B frame (set to 0, 1 and 2), this embodiment also evaluates the priority of a missing frame in a GOP from the impact of the missing frame on the decoding of other frames, and uses dependent blocks to indicate the number of frames / data blocks that have a dependence on the missing frame and cannot be decoded if the missing frame is lost. On this basis, the BLOCK_INFO frame used in the original DTP to represent frame data is modified, and DependentBlocks is added to represent the number of other frames that depend on the frame.

[0081] In summary, the final priority of a video frame is obtained by the following formula (6) in this embodiment:

[0082]

[0083] Wherein, Real Priority is the real priority of the data block to be sent after optimization, the original priority priority is the original priority of the frame originally, the dependent block number dependent blocks is used to indicate the number of frames that cannot be decoded due to the dependence on the current data block to be sent if the frame is lost, that is, the number of data blocks dependent on the current data block to be sent. The remaining time is the remaining transmission time of the data block to be sent, which can be the remaining transmission time calculated by the original DTP protocol, or the remaining transmission time optimized by considering the path delay fluctuation. Alpha is the priority adjustment coefficient, and beta is the block dependence weight adjustment coefficient. The values of the two coefficients need to be dependent on the specific scene, for example, for the application that pays more attention to video clarity and video smoothness, the value of beta should be appropriately increased to improve the sending priority of the frame with higher dependence. If more attention is paid to guaranteeing the priority sending of data blocks with high priority or closer to the deadline, the value of alpha should be appropriately increased. The above calculation process simultaneously considers the priority of the video frame itself, the remaining transmission time, and the dependence of other frames in the GOP on the current frame, and the scheduler selects the frame with the smallest Real Priority, that is, the highest priority, to send each time.

[0084] Figure 3 The electronic device structure schematic diagram for implementing the deadline-sensitive streaming method provided by the embodiment of the application is provided. The device 300 includes a processor 310 such as a central processing unit (CPU), a communication bus 320, a communication interface 340, and a memory 330. The processor 310 and the memory 330 can communicate with each other through the communication bus 320. The memory 330 stores a computer program, and when the computer program is executed by the processor 310, the function of one or more steps in the deadline-sensitive streaming method provided by the application can be realized.

[0085] The memory refers to a device for storing computer programs and / or data based on a certain storage medium, which can be a volatile memory (VM, commonly known as memory) or a non-volatile memory (NVM). The memory refers to an internal memory that exchanges data directly with the processor, which can read and write data at any time and has very fast speed as a temporary data storage medium for the operating system and other running programs. The memory can be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), etc. The non-volatile memory refers to a memory using a persistent storage medium, which has the characteristics of large capacity and persistent data storage, and can be a storage class memory (SCM), a solid state disk (SSD), a NAND flash memory, a disk, etc. The SCM is a general term for a new storage medium between the memory and the flash memory in the industry, which is a composite storage technology combining the persistent storage characteristics and the memory characteristics, and has a slower access speed than the DRAM and a faster access speed than the SSD hard disk.

[0086] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0087] It should be appreciated that embodiments of the present application can be realized by either computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory (or referred to as non-persistent) storage medium. The methods can be implemented in a computer program using standard programming techniques, including non-transitory storage media configured with a computer program, which when loaded into a computer results in an apparatus for practicing the application. Each program can be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language can be a compiled or interpreted language. Furthermore, the programs can be stored on a non-transitory storage medium, which can be a magnetic medium, optical medium, or solid state medium. Moreover, the programs can be executed as part of a virtual computer, a cloud computing environment, or other such environment. Furthermore, the programs can be executed on a specially configured integrated circuit, or other such hardware. In addition, the operations of the processes described herein can be performed in an arbitrary sequence unless otherwise indicated, or otherwise inherently obvious in context. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. The computer programs include program instructions that are executable by one or more processors.

[0088] Further, the methods can be implemented in any suitable type of computing platform operably connected to, including but not limited to, a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine readable code stored on a non-transitory storage medium or device, whether removable or integrated to a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it is readable by a programmable computer and, when the storage medium or device is read by a computer, is useful in configuring and operating the computer to perform the processes described herein. Furthermore, the machine readable code, or portions thereof, can be transmitted over wired or wireless networks. The present application includes these and other different types of non-transitory computer readable storage media when such media include instructions or programs implementing the steps described above in conjunction with a microprocessor or other data processor. The present application also includes the computer itself when programmed according to the methods and techniques described herein.

[0089] The above description is merely illustrative of the embodiments of the present application and is not in any way limiting. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the present application. Any and all modifications, variations or equivalents that fall under the spirit and scope of the present application are intended to be within the scope of the present application.

Claims

1. A deadline-sensitive streaming method, characterized in that, This method is applied to streaming devices using the Time-Sensitive Protocol (DTP), and includes: When performing path congestion control, congestion packet loss is distinguished from non-congestion packet loss. The current path transmission rate is calculated based on the actual congestion packet loss rate, and corresponding congestion control is performed. The congestion packet loss refers to packet loss caused by the transport layer buffer being full. The actual priority of the data block to be sent is calculated based on the current path transmission rate determined by the path congestion control module and the original priority of the data block to be sent, and a decision is made on whether to send the data block to be sent based on the actual priority. The method for distinguishing between congestion-induced packet loss and non-congestion-induced packet loss is as follows: If packet loss occurs during the third link round-trip time measurement after three consecutive link round-trip time measurements, the packet loss is determined to be congestion packet loss if the following two conditions are met: Condition 1: The round-trip time measured a second time is within the preset fluctuation range of the round-trip time measured a first time, and the round-trip time measured a second time is greater than the known minimum round-trip time of the link; and, Condition 2: The round-trip delay measured for the third time is within the preset fluctuation range of the round-trip delay measured for the second time, and the round-trip delay measured for the third time is greater than the known minimum round-trip delay of the link; The preset fluctuation range of the round-trip delay is determined by adding or subtracting a preset fluctuation coefficient from the previously measured round-trip delay.

2. The method according to claim 1, characterized in that, The method for calculating the true priority of the data block to be sent is as follows: The remaining transmission time of the data block to be sent is obtained by subtracting the ratio of the remaining block size to the current path transmission rate from the deadline for sending the data block, then subtracting the one-way delay, and then subtracting the impact of delay fluctuation. The product of the original priority and the remaining transmission time is used as the true priority.

3. The method according to claim 1 or 2, characterized in that, The method for calculating the true priority of the data block to be sent is as follows: After calculating the product of the original priority and the remaining transmission time, it is then combined with the priority adjustment factor. Multiply, then add the block dependency weight adjustment factor. The product of the product of the number of dependent blocks and the inverse of the product is used as the final true priority.

4. A time-sensitive streaming device, characterized in that, This device is used in streaming devices employing the Time-Sensitive Protocol (DTP). The device includes: The congestion control module is used to distinguish between congestion-induced packet loss and non-congestion-induced packet loss when performing path congestion control, calculate the current path transmission rate based on the actual congestion-induced packet loss rate, and perform corresponding congestion control; the congestion-induced packet loss refers to packet loss caused by the transport layer buffer being full. The decision module is used to calculate the remaining transmission time of the data block to be sent by subtracting the ratio of the remaining block size to the current transmission rate from the deadline for data block transmission, then subtracting the one-way delay, and then subtracting the impact of delay fluctuation. The product of the original priority and the remaining transmission time is then multiplied by the priority adjustment coefficient. Multiply, then add the block dependency weight adjustment factor. The final true priority is obtained by multiplying the value of the data block to be sent by the inverse of the number of dependent blocks. The decision on whether to send the data block to be sent is based on the true priority. Specifically, when distinguishing between congestion-induced packet loss and non-congestion-induced packet loss, the congestion control module is used for: If packet loss occurs during the third link round-trip time measurement after three consecutive link round-trip time measurements, the packet loss is determined to be congestion packet loss if the following two conditions are met: Condition 1: The round-trip time measured a second time is within the preset fluctuation range of the round-trip time measured a first time, and the round-trip time measured a second time is greater than the known minimum round-trip time of the link; and, Condition 2: The round-trip delay measured for the third time is within the preset fluctuation range of the round-trip delay measured for the second time, and the round-trip delay measured for the third time is greater than the known minimum round-trip delay of the link; The preset fluctuation range of the round-trip delay is determined by adding or subtracting a preset fluctuation coefficient from the previously measured round-trip delay.

5. An electronic device, characterized in that, It includes a processor, a communication interface, a storage medium, and a communication bus, wherein the processor, the communication interface, and the storage medium communicate with each other through the communication bus; Storage medium used to store computer programs; A processor, when executing a computer program stored on a storage medium, implements the method of any one of claims 1-3.

6. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1 to 3.

Images