Method and device for controlling real-time video streaming and storage medium

Abstract

The invention discloses a method and device for controlling real-time video streaming and a storage medium. The present disclosure relates to described techniques for optimizing real-time video streaming that dynamically manage data transmission and bandwidth estimation by using padding data as a buffer for early detection of network congestion.

Description

TECHNICAL FIELD

[0001] The present invention relates to techniques for managing network congestion, and in particular, to techniques for controlling the bit rate of encoded video data during transmission over a network using padding data added to the encoded video data. BACKGROUND

[0002] Streaming video data over a network of limited capacity presents significant challenges. The core problem is to ensure that the amount of data transmitted does not exceed what the network can handle. If the data rate exceeds the network's capacity, then network congestion increases and, once the level of network congestion exceeds a certain threshold, data packets are lost. To reduce this risk, the network bandwidth can be estimated, a process known as Bandwidth Estimation (BWE). The estimated BWE value can then be used to adjust the bit rate during video encoding by a bit rate controller.

[0003] Various methods and protocols have been developed for streaming, each employing a different method for calculating BWE. Despite these advances, the risk of data loss remains, particularly if the bit rate controller is slow to adapt to changing bandwidth or if the bandwidth decreases rapidly. For example, when multiple streaming devices share the same bandwidth, BWE can become challenging, leading to increased contention and fluctuating available capacity.

[0004] One well-known solution to this problem involves the use of the Transmission Control Protocol (TCP), which ensures reliability by retransmitting potentially lost data packets. However, the reliability of TCP comes at the cost of reduced control and potentially lower transmission rates.

[0005] On the other hand, datagram-based protocols such as the User Datagram Protocol (UDP) are not affected by these drawbacks. UDP allows for more efficient transmission rates and greater control over data flow. However, due to its lack of retransmission mechanism, it inherently carries the original problem of elevated risk of data loss.

[0006] Therefore, there is a need for improvement in this regard. SUMMARY

[0007] In view of the above, it would be advantageous to address or at least alleviate one or more of the above-identified deficiencies.

[0008] According to a first aspect of the present application, there is provided a method for controlling real-time video streaming, the method comprising: providing a data set structure for transmission of video data, wherein a data set generated according to the data set structure in a first configuration comprises padding data and encoded video data having a first encoding bit rate; generating a first data set according to the data set structure in the first configuration and transmitting the first data set as a data stream via a communication channel; during transmission of the first data set, receiving a first measure indicative of a network congestion level of the communication channel and determining, according to the received first measure, that the network congestion level has risen above a threshold; adjusting the data set structure to a second configuration, wherein a data set generated according to the adjusted data set structure comprises padding data and encoded video data having a second encoding bit rate, wherein the second encoding bit rate is lower than the first encoding bit rate; and generating a second data set according to the data set structure in the second configuration and transmitting the second data set as a data stream via the communication channel.

[0009] Advantageously, the present disclosure proposes techniques to maintain a high bit rate while managing network congestion. These techniques include using padding as a buffer to detect congestion early. Initially, the data sets transmitted comprise encoded video data at a high bit rate supplemented with padding data. During transmission, if a rise in network congestion level above a set threshold is detected, the data set structure used to transmit the encoded video data is adjusted to reduce the encoding bit rate of the video data while still including padding data. Typically, a bit rate controller in the encoder is requested to reduce the bit rate. This adjustment thus helps to reduce the bit rate, thereby alleviating congestion. The modified data sets are then transmitted with the lower bit rate. By incorporating padding data, the system can identify early signs of congestion, allowing timely adjustment of the bit rate and ensuring smoother streaming and reduced data loss. This approach can further allow the bit rate to be maintained high under varying network conditions. By dynamically adjusting the bit rate in response to detected congestion, the approach advantageously allows for improved video quality while reducing interruptions and packet loss in fluctuating network environments.

[0010] As used herein, a "data set" refers to a structured unit of data for transmission over a network. In particular, it is a data packet or collection of data packets (e.g., Internet Protocol (IP) packets) containing encoded video data and / or additional data. A "data set structure" defines how video data and additional information such as padding data are organized within each data packet or collection of data packets.

[0011] As used herein, a "communication channel" refers to the medium through which data is transmitted between a sender and a recipient in a network. The channel can utilize various transmission technologies (Wi-Fi, 4G, fiber, etc.) and protocols to manage and facilitate the flow of data. For example, Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) are protocols used in communication channels for video streaming.

[0012] "Padding data" refers to additional, non-essential data added to video data packets being transmitted over a network. It acts as a buffer for early signs of congestion in the communication channel. Padding data can be created by appending extra bytes to the encoded video data within each packet or data set. These bytes do not carry any meaningful information and are used only to monitor the capacity of the network. Rather than simply adding zeros, padding data can be more complex, such as noise or other insignificant data that can mimic characteristics of actual video data. This can make the padding more representative of real video data traffic, ensuring more accurate detection of congestion.

[0013] As used herein, a "network congestion level" (the level can be replaced by degree, amount, range, etc.) refers to the degree of traffic load and performance degradation experienced by a communication channel. The level is quantified by evaluating specific metrics that indicate the efficiency and reliability of data transmission. These metrics (as further explained below, numbered first, second, third, etc. herein) can include, for example: packet loss rate, measuring the percentage of packets that fail to reach their destination; jitter, evaluating the variability of packet arrival times; transmission buffer occupancy, representing the fill level of a buffer used to temporarily store data prior to transmission; and latency, measuring the delay of data packet transfer over the network. An increase in any of these metrics that exceeds a predefined metric-specific threshold indicates a higher network congestion level, prompting necessary adjustments as described herein to maintain video quality while reducing interruptions and packet loss in fluctuating network environments. Thresholds can be application-specific. Examples of thresholds can include 2% packet loss, 30 milliseconds (ms) jitter, 100 ms latency, and 80% transmission buffer occupancy. Any other thresholds can be applied, such as 4% packet loss, 25 ms jitter, 150 ms latency, and 75% transmission buffer occupancy.

[0014] In addition, Explicit Congestion Notification (ECN) can be another indicator of network congestion. ECN is a mechanism used in IP networks to indicate network congestion without dropping packets. Routers experiencing congestion mark packets with ECN, and this information is passed back to the sender by the receiver. The sender can then reduce its transmission rate accordingly as described herein. The level of ECN can be measured based on the number or frequency of received ECN marked packets. When a network router experiences congestion, it marks packets with ECN bits instead of dropping them. The receiver of these packets then sends a congestion notification to the sender. The sender can track the frequency or count of these ECN marked packets to assess the level of congestion. The threshold value for the application can be received as an ECV (Explicit Congestion Notification) feedback message.

[0015] In the context of the present disclosure, the terms "first", "second", "third" and the like do not necessarily indicate a sequential or hierarchical order. Rather, these terms are merely used to identify and differentiate different features, elements or steps within the description. Unless explicitly stated otherwise, the terms are intended to provide clarity and should not be interpreted as implying any particular order or hierarchy.

[0016] In some examples, the method further comprises, in response to determining that the network congestion level has risen above the threshold: adjusting the data set structure to a third configuration, wherein data sets generated according to the data set structure in the third configuration comprise encoded video data having a second encoding bit rate and do not comprise padding data; generating third data sets according to the data set structure in the third configuration and transmitting the third data sets as a data stream via the communication channel; during transmission of the third data sets, receiving a second measure indicative of the network congestion level of the communication channel and determining from the received second measure that the network congestion level has fallen below the threshold; and adjusting the data set structure to the second configuration.

[0017] In the present example, in response to determining that the network congestion level has risen above the threshold, the system first adjusts the data set structure to a third configuration. In this configuration, each data set comprises encoded video data having a reduced encoding bit rate and does not comprise padding data. This adjustment effectively reduces the overall data rate being transmitted, thereby helping to alleviate network congestion. The system then generates these third data sets and transmits them via the communication channel. During transmission of the third data sets, if it is determined (via the second measure) that the network congestion level has fallen below the threshold, the data set structure can be adjusted to the second configuration, in which the data sets comprise padding data and encoded video data having a reduced bit rate. Advantageously, increased flexibility can be achieved which allows the system to respond quickly to improved network conditions by reintroducing padding data, which helps to maintain a smoother stream as described herein.

[0018] In some examples, the method further comprises: in response to determining that the network congestion level has risen above the threshold, instructing the video encoder to begin encoding the video data at the second encoding bit rate; adjusting the data set structure to a fourth configuration, wherein a data set generated according to the data set structure in the fourth configuration comprises encoded video data having the first encoding bit rate and does not comprise padding data; and in response to receiving the instruction that the video encoder is encoding the video data at the second encoding bit rate, adjusting the data set structure to the second configuration.

[0019] Advantageously, the latency of the response of the bit rate controller of the encoder can be taken into account. As the bit rate controller can take some time to reach a lower bit rate, the removal of padding data initially reduces the overall data load on the network immediately, providing fast congestion mitigation. Then, when the bit rate is lowered by the encoder, padding is reintroduced to maintain the ability to monitor and manage further network fluctuations.

[0020] In some examples, the method further comprises: during transmission of the second data set, receiving a third measure indicative of a network congestion level of the communication channel, and determining from the received third measure that the network congestion level has decreased below the threshold; adjusting the data set structure to the first configuration; and generating a fourth data set according to the data set structure in the first configuration and transmitting the fourth data set as a data stream via the communication channel. Advantageously, a process for adapting to improved network conditions can be implemented. In particular, it allows the system to take advantage of improved network conditions by restoring a higher bit rate and keeping padding data.

[0021] In some examples, the data set structure defines that encoded video data is prioritized over padding data in a data set generated according to the data set structure during transmission of the data set. Advantageously, when network congestion occurs, the lower priority padding data can be dropped first during transmission, thereby preserving the integrity and quality of the encoded video data.

[0022] In some examples, the data set structure defines that encoded video data is positioned before padding data in a data set generated according to the data set structure. This positioning ensures that padding data is transmitted last, such that any dropped packets due to network congestion will be more likely to be padding data. This approach further helps to preserve the quality and integrity of the video stream by reducing the likelihood of encoded video data being lost.

[0023] In some examples, the method further comprises determining the size of the padding data using at least one of: a transmission technology of the communication channel; a size of the encoded video data; a measure indicative of a change in available bandwidth of the communication channel; a measured round trip time, RTT, of a signal transmitted on the communication channel; and a user input indicative of an importance of the video data. Advantageously, by tailoring the size of the padding data based on one or more of these specific factors, network resources can be managed more accurately and dynamically. For example, if Wi-Fi is used as the transmission technology, the available bandwidth can vary more than if a cellular network such as 4G or 5G is used, which can result in a larger size of padding data when using Wi-Fi than when using 5G. The RTT can be measured by transmitting a ping signal on the communication channel or by extracting the RTT from the transmission protocol of the communication channel such as TCP or RTCP. A larger RTT can advantageously result in a larger padding data.

[0024] In some examples, determining the measure indicative of a change in available bandwidth of the communication channel comprises using historical data identifying a frequency of changes in configuration of the data set structure. Such historical data provides insight into the stability and fluctuations of the network bandwidth. Advantageously, past behaviour can be utilised to predict future network conditions, allowing for more accurate and proactive adjustment of the padding size and therefore allowing for improved live streaming as described herein.

[0025] In some examples, the data set is transmitted over the communication channel using at least one of a TCP protocol and a UDP protocol. In some examples, both the padding data and the video data are transmitted using the same protocol. In other examples, the encoded video data can be transmitted using the TCP protocol and the padding data is transmitted using the UDP protocol. Thus, the encoded video data can be transmitted using TCP for its reliability, facilitating error correction and in-order delivery of the video data, while the padding data is transmitted using UDP despite its lack of delivery guarantees, to take advantage of its speed and efficiency.

[0026] According to a second aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to the first aspect when executed on an apparatus having processing capability.

[0027] According to a third aspect of the present application, the above mentioned object is achieved by an apparatus for controlling real-time video streaming, the apparatus being configured for: providing a data set structure for transmission of video data, wherein a data set generated according to the data set structure in a first configuration comprises padding data and encoded video data having a first encoding bitrate; generating a first data set according to the data set structure in the first configuration and transmitting the first data set as a data stream via a communication channel; during transmission of the first data set, receiving a first measure indicative of a network congestion level of the communication channel and determining, according to the received first measure, that the network congestion level has risen above a threshold; adjusting the data set structure to a second configuration, wherein a data set generated according to the adjusted data set structure comprises padding data and encoded video data having a second encoding bitrate, wherein the second encoding bitrate is lower than the first encoding bitrate; and generating a second data set according to the data set structure in the second configuration and transmitting the second data set as a data stream via the communication channel.

[0028] In some examples, the apparatus is implemented in a camera that captures the video data. In other examples, the apparatus is connected to a camera that captures the video data.

[0029] The second and third aspects can generally have the same features and advantages as the first aspect. It is further noted that the present disclosure relates to all possible combinations of features unless explicitly stated otherwise. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above mentioned and other objects, features and advantages of the present application will be better understood from the following illustrative and non-limiting detailed description of embodiments thereof, taken in conjunction with the accompanying drawings, wherein the same reference notations are used to refer to similar elements, and in which:

[0031] Figure 1 A system for real-time streaming of video data according to an embodiment is shown;

[0032] Figure 2 A buffer established during a reduced bandwidth of a communication channel for real-time streaming according to an embodiment is shown;

[0033] Figures 3 to 6 A method for controlling real-time video streaming according to an embodiment is shown;

[0034] Figure 7 A prioritization of data in a data structure for generating a data set transmitted in real-time streaming according to an embodiment is shown;

[0035] Figures 8 to 10 A flowchart of a method for controlling real-time video streaming according to an embodiment is shown. DETAILED DESCRIPTION

[0036] In the following, the application will be described more fully and in connection with preferred embodiments by referring to the drawings, in which embodiments of the application are shown. The systems and apparatuses disclosed herein will be described during operation.

[0037] The techniques described herein optimize live video streaming by dynamically managing data transmission and bandwidth estimation (BWE) to handle network congestion. Conventional approaches strive to maintain high video quality while minimizing or reducing network congestion metrics such as latency, jitter, packet loss rate, and transmission buffer occupancy under varying network conditions. Key approaches described herein include using padding data as a buffer to detect congestion early and adjusting the data set structure of the streaming data in response to elevated network congestion levels to quickly relieve congestion. These techniques allow for responsive adjustments to bitrate and padding to maintain streaming quality even under fluctuating network conditions. Advantages can include enhanced video quality, reduced data loss, efficient bandwidth utilization, and / or reduced latency, facilitating a smoother and more reliable streaming experience.

[0038] Figure 1 A system 100 for live streaming of encoded video data 106 is shown by way of example. The system 100 includes a camera 104 that captures a video stream depicting a scene 102. The camera (or a separate encoding entity) encodes the video stream into encoded video data 106. The encoded video data 106 is organized into data sets 108 configured according to a data set structure. The data sets 108 are structured units of data for transmission over a network, including encoded video data 106 that has been organized into packets or frames suitable for efficient transmission via a communication channel of the network 112 and decoded at a receiver side 114. The transmission technology of the communication channel 112 can be Wi-Fi, 5G, 4G, fiber, Ethernet, etc. The receiver can be, for example, a display 114 configured to decode and display the received encoded video data. The data set structure refers to the specific format and organization of these data sets, which define how the encoded video data and any additional information such as padding data or control signals are arranged within each data set. The data set structure also specifies the bitrate of the encoded video data. The specific details and arrangement within the data set structure are referred to as the configuration of the data set structure.

[0039] In a live video streaming system, BWE is a technique used to optimize the quality and stability of the transmitted encoded video data 106. BWE includes accurately estimating the available network bandwidth to adapt the bitrate of the encoded video data 106, ensuring that the transmitted bitrate does not exceed the capacity of the network 112, otherwise this can lead to elevated network congestion levels. However, if the bitrate adaptation is too slow or if the bandwidth decreases rapidly, there is still a risk of dropped data.

[0040] As mentioned above, various metrics can be used to determine or measure the level of network congestion. In the following examples, generally, latency is used to illustrate the techniques described herein, with an increase (or decrease) in latency relative to some threshold acting as the key indicator. However, it should be noted that other suitable methods for determining the level of network congestion, such as jitter, packet loss rate, level of Explicit Congestion Notification (ECN) (e.g., the number or frequency of packets marked with ECN bits by a router), or transmission buffer occupancy, can also be applied. Figure 2 Illustratively, how such latency and packet loss occurs in these cases is shown.

[0041] To not introduce latency or data loss, depending on the frames per second (FPS) of the video being streamed, each frame of data will have a time slot 202a-202e for transmission. For example, if a video is being streamed with 30 FPS, each image frame has a time slot 202a-202e of 33 ms for transmission. When the bandwidth 204 decreases, the transmission of the data set 108 that would have been for one time slot 202a-202e overflows to the next time slot, delaying the data set 108 of the subsequent image frame. Thus, the buffer in the transmission channel (e.g., the buffer in the router of the transmission channel) begins to increase incrementally. If the buffer reaches its capacity, data will be dropped. In Figure 2 In the middle, the data sets 108 for each time slot 202a-202e are marked with different patterns, such as the diagonal striped pattern of the data set 108 that would have been for time slot 202a and the square pattern of the data set 108 that would have been for time slot 202b. When the bandwidth decreases during the third time slot 202c, the data set 108 that would have been for that time slot (marked with vertical stripes) is delayed to the next time slot 202d, and so on. This problem gets worse as the bandwidth decreases further in time slots 202d-202e. Thus, the latency increases, and the data sets 108 can be dropped.

[0042] As mentioned above, the inventors have recognized that adding padding data to the data sets of a live streaming video can advantageously act as a buffer for early detection of congestion. Using this technique, the streaming quality can be improved, and data loss can be avoided. Figures 3 to 6 Embodiments are shown of how padding data can be used to dynamically manage data transmission and BWE. These embodiments will now be described in relation to Figures 8 to 10 the flowchart of FIG. 8.

[0043] Figure 8 A method 800 for controlling real-time video streaming with the use of padding data is described. A data set structure S802 is provided for transmission of video data. As Figure 1In some examples, using this data set structure, a data set can be generated S804 and transmitted as a data stream via a communication channel. A first data set is generated S804 according to the data set structure in the first configuration. This is illustrated in the left half of Figure 1 The first data set 302 generated S804 according to the data set structure in the first configuration includes padding data 304 and encoded video data 306 having a first (high) encoding bit rate.

[0044] The padding data 304 is generally non-sensory data that is generated to be similar in composition to the video data. Any suitable algorithm can be used to generate the padding data 304, such as random pixel values, synthetic patterns, or even scrambled video frames or portions of video frames.

[0045] The size of the padding data 304 can depend on various rules, configurations, or properties of the video streaming system. For example, the size of the padding data 304 can depend on the transmission technology of the communication channel, where each transmission technology can be mapped to a particular size. More reliable transmission technologies, such as 4G / 5G, can require less padding data, while less reliable transmission technologies, such as Wi-Fi, can require more padding data. In some examples, the size of the padding data 304 can be a relative size that depends on the size of the encoded video data 306, where the size of the padding data 304 generally grows with the size of the encoded video data 306. In some examples, the size of the padding data 304 can depend on a measure that indicates a variation in the available bandwidth of the communication channel, where a larger variation generally results in larger padding data. This variation can be determined using historical data that identifies a frequency of variation of the configuration of the data set structure. This variation can be further determined using historical data related to a measured round trip time (RTT) of the communication signal. This variation can be determined at periodic intervals and used until the next determination of the variation. In some examples, the size of the padding data 304 can depend on an RTT of a signal transmitted over the communication channel, such as an RTT of a ping signal transmitted over the communication channel or an RTT received via a protocol used to transmit the data set over the communication channel. A larger RTT generally results in larger padding data 304. In some examples, the size of the padding data 304 can depend on a user input that indicates an importance of the video data, where more important video data generally results in larger padding data. Such methods of configuring the padding data size can provide a flexible and adaptive method for enhancing the robustness of real-time video streaming. By adjusting the size of the padding data based on various factors, including the transmission technology, the video data size, the bandwidth variation, the RTT, and the user input, the system can effectively manage network congestion and maintain high video quality.

[0046] Now returning to Figure 8During transmission of the first data set 302, the system (e.g., a camera or device collectively referred to as network equipment processing the streaming setup) can receive an indication that the network congestion level of the communication channel has risen above a threshold. The network congestion level can be determined S806 from a metric received that indicates the network congestion level. As described above, the metric can indicate that the RTT has risen above a certain level, that the transmit buffer used in the live streaming is filled, that the packet loss rate has risen, that the jitter is high, and / or that a long queue length is observed in the network device. This means that network congestion above a certain threshold is detected. For example, the latency of the communication channel can have increased above a certain threshold. Accordingly, the data set structure is adjusted S808 to a second configuration. The second data set 308 generated S810 according to the adjusted data set structure (in the second configuration) includes padding data 310 and encoded video data 312 having a second encoding bit rate. Depending on the rules defined for determining the size, the padding data 310 of the second data set 308 can have a smaller size than the padding data 304 of the first data set 302, or the same size. The second encoding bit rate (i.e., the bit rate of the encoded video data 312 of the second data set 308 in the right half of the figure) is lower than the bit rate of the encoded video data 306 of the first data set 302. Accordingly, the bandwidth required for live transmission of the second data set 308 is smaller than the bandwidth required for live transmission of the first data set 302, which helps to manage the network congestion. If the network congestion is still indicated to be above the threshold despite the reduced encoding bit rate (e.g., because the latency is still above the threshold), steps S808 to S810 can be repeated to continue reducing the encoding bit rate of the video data until the high network congestion is no longer detected. Figure 3

[0047] In some cases, as shown in Figure 6 , during transmission of the second data set 308, a new metric indicating the network congestion level of the communication channel is received. According to the newly received metric, it can be determined S812 that the network congestion level has decreased below the threshold. In this case, the data set structure can be adjusted S814 back to the first configuration, and data sets 602 can be generated according to the data set structure in the first configuration such that the bit rate of the encoded video data 606 of these data sets 602 is the same as the original bit rate of the encoded video data 306 of the first data set 302.

[0048] In some examples, as described in Figure 3 , an intermediate step is implemented between the generation of the first data set 302 and the generation of the second data set 308. Such an intermediate step can be implemented to more quickly or more accurately alleviate the high network congestion that has been detected.

[0049] Figure 9 and Figure 4 ​One such example showing intermediate steps is illustrated. In this example, in response to determining S806 that the network congestion level has risen above the threshold, the data set structure is first adjusted S902 to a third configuration. The third data set 402 generated S904 according to the data set structure in the third configuration comprises encoded video data 404 with a lower second encoding bit rate and does not comprise padding data. Thus, the bandwidth required for real-time transmission of the third data set 402 is initially reduced more compared to directly using the second configuration. During transmission of the third data set 402, when a new measure indicative of the network congestion level of the communication channel is received, it is determined S906 from the new measure that the network congestion of the communication channel has decreased below the threshold (e.g. the latency has again decreased below the threshold), padding data can be reintroduced by adjusting S808 the data set structure to the second configuration and generating S810 the second data set 308 accordingly for transmission.

[0050] Figure 10 and Figure 5 Another example showing intermediate steps is illustrated. In this example, in response to determining S806 that the network congestion level has risen above the threshold, the video encoder is instructed S1002 to encode the video data using a lower second encoding bit rate. In this example, any latency in adapting the bit rate of the encoded video data is advantageously taken into account by first adjusting the data set structure to a fourth configuration, wherein a fourth data set 502 generated according to the data set structure in the fourth configuration comprises encoded video data 504 with the first encoding bit rate and does not comprise padding data. When the instruction S1008 is received that the video encoder is to encode the video data using the lower second encoding bit rate, the data set structure can be adjusted S808 to the second configuration (reducing the bit rate and again including padding data), and the second data set 308 can be generated S810 and transmitted as described above.

[0051] In some examples, the data set structure can define that encoded video data is prioritized over padding data during transmission of a data set generated according to the data set structure. Such examples are illustrated in Figure 7which portions of the data set can be dropped can be preferable, at least on a statistical level. For example, the data set structure can provide that the encoded video data is placed in the data set 302 generated according to the data set structure before padding data. In an example, the data set is transmitted in multiple packets. The first packets can have a higher probability of reaching the receiving device over the communication network 112 than subsequent packets. Thus, subsequent portions of the data set 302 can be dropped on the communication network 112. By ordering the data in each data set 302 such that the padding data 304 is transmitted after the encoded video data 306, this prioritization can be achieved. Thus, if any packets are dropped 702, the padding data 304 can be dropped 702 first. Any other suitable manner of prioritizing the encoded video data 306 over the padding data 304 can be implemented. For example, a transport protocol such as the DiffServ protocol can be used to prioritize different types of data packets by marking them with different priorities in the headers of the different types of data packets.

[0052] The above embodiments should be understood to be illustrative only. Further embodiments of the application are envisaged. For example, prioritization can be achieved by marking the data with quality of service flags or by implementing custom transport queues. It will be appreciated that any feature described in relation to any one embodiment can be used alone or in combination with other features described, and that the one or more features of any other embodiment or any combination of any other embodiments. Furthermore, equivalents and modifications not described above can also be employed without departing from the scope of the application, which is defined in the claims attached hereto.

Claims

1. A method for controlling real-time video streaming, the method comprising: Provide (S802) a dataset structure for the transmission of video data, wherein the dataset generated according to the dataset structure in the first configuration includes padding data (304) and encoded video data (306) having a first encoding bit rate; A first dataset (302) is generated (S804) according to the dataset structure in the first configuration, and the first dataset is transmitted as a data stream via the communication channel (112); During the transmission of the first dataset, a first metric indicating the network congestion level of the communication channel is received, and it is determined (S806) based on the received first metric that the network congestion level has risen above a threshold. The dataset structure is adjusted (S808) to a second configuration, wherein the dataset generated according to the adjusted dataset structure includes padding data (310) and encoded video data (312) with a second encoding bit rate, wherein the second encoding bit rate is lower than the first encoding bit rate; and A second dataset (308) is generated (S810) according to the dataset structure in the second configuration, and the second dataset is transmitted as a data stream via the communication channel.

2. The method according to claim 1, further comprising: In response to determining (S806) that the network congestion level has risen above the threshold: The dataset structure is adjusted (S902) to a third configuration, wherein the dataset generated according to the dataset structure in the third configuration includes encoded video data (404) with the second encoding bit rate and does not include padding data; A third dataset (402) is generated (S904) according to the dataset structure in the third configuration, and the third dataset is transmitted as a data stream via the communication channel; During the transmission of the third dataset, a second metric indicating the network congestion level of the communication channel is received, and based on the received second metric, it is determined (S906) that the network congestion level has decreased below the threshold; and Adjust the dataset structure (S808) to the second configuration.

3. The method according to claim 1, further comprising: In response to determining (S806) that the network congestion level has risen above the threshold, instructing (S1002) the video encoder to start encoding the video data at the second encoding bit rate; The dataset structure is adjusted (S1004) to a fourth configuration, wherein the dataset (502) generated according to the dataset structure in the fourth configuration (S1006) includes encoded video data (504) with the first encoding bit rate and does not include padding data; and In response to receiving (S1008) an instruction from the video encoder to encode video data at the second encoding bit rate, the dataset structure is adjusted (S808) to the second configuration.

4. The method according to claim 1, further comprising: During the transmission of the second dataset, a third metric indicating the network congestion level of the communication channel is received, and it is determined (S812) based on the received third metric that the network congestion level has been reduced to below the threshold. Adjust the dataset structure (S814) to the first configuration; as well as A fourth dataset (602) is generated (S816) according to the dataset structure in the first configuration, and the fourth dataset is transmitted as a data stream via the communication channel.

5. The method of claim 1, wherein, The dataset structure defines that during the transmission of the dataset generated according to the dataset structure, the encoded video data takes precedence over the padding data.

6. The method of claim 1, wherein, The dataset structure specifies that the encoded video data is placed before the padding data in the dataset generated according to the dataset structure.

7. The method of claim 1, further comprising: The size of the padding data is determined using at least one of the following: The transmission technology of the communication channel; The size of the encoded video data; A measurement indicating the change in the available bandwidth of the communication channel; The round-trip time (RTT) of the signal transmitted on the communication channel; as well as User input indicating the importance of the video data.

8. The method of claim 7, further comprising: Using historical data that identifies the frequency of changes in the configuration of the dataset structure, the measurement value indicating changes in the available bandwidth of the communication channel is determined.

9. The method of claim 7, wherein, A relatively large RTT results in a relatively large padding data size.

10. The method of claim 1, wherein, The dataset is transmitted through the communication channel using at least one of the TCP and UDP protocols.

11. The method of claim 10, wherein, The encoded video data is transmitted using the TCP protocol, and the padding data is transmitted using the UDP protocol.

12. The method of claim 1, wherein, The metric indicating the network congestion level of the communication channel is defined as one of the following: packet loss rate, jitter, transmission buffer occupancy, number or frequency of explicit congestion notifications, and latency.

13. A non-transitory computer-readable storage medium having instructions stored thereon for implementing the method of claim 1 when executed on a camera with processing capabilities.

14. An apparatus for controlling real-time video streaming, the apparatus being configured to: Provides (S802) a dataset structure for the transmission of video data, wherein, The dataset generated according to the dataset structure in the first configuration includes padding data (304) and encoded video data (306) with a first encoding bit rate; A first dataset (302) is generated (S804) according to the dataset structure in the first configuration, and the first dataset is transmitted as a data stream via the communication channel (112); During the transmission of the first dataset, a first metric indicating the network congestion level of the communication channel is received, and it is determined (S806) based on the received metric that the network congestion level has risen above a threshold. The dataset structure is adjusted (S808) to a second configuration, wherein the dataset generated according to the adjusted dataset structure includes padding data (310) and encoded video data (312) with a second encoding bit rate, wherein the second encoding bit rate is lower than the first encoding bit rate; and A second dataset (308) is generated (S810) according to the dataset structure in the second configuration, and the second dataset is transmitted as a data stream via the communication channel.

15. The apparatus of claim 14, implemented in a camera that captures the video data.

Images