Method and apparatus for transmitting media stream

Abstract

Provided is a method, performed by a transmission device, for transmitting a media stream. The method may comprise the steps of: identifying transmission data for transmission to a reception device, the transmission data including a plurality of packets for media data and at least one forward error correction (FEC) packet for the plurality of packets; generating additional transmission data for the transmission data; transmitting the transmission data and the additional transmission data to the reception device; receiving, from the reception device, feedback related to the transmission data and the additional transmission data; and adjusting attributes of the media data on the basis of the feedback.

Description

Method and device for transmitting a media stream

[0001] The present disclosure relates to a method and apparatus for transmitting a media stream. More specifically, the present disclosure relates to estimating network conditions at a transmitting side and adjusting parameters related to the media stream.

[0002] Video streaming over networks using UDP (User Datagram Protocol) sockets has both advantages and disadvantages. Since this protocol does not guarantee delivery, correct order, or non-duplicate datagrams, it is prone to errors. On the other hand, the protocol can verify the accuracy of these matters at the transport layer and avoid the overhead of processing packets. Due to these advantages, UDP is widely used in real-time systems where dropping packets is preferable to resending them. All error correction is performed at the application layer.

[0003] A characteristic of video streaming over UDP sockets is the use of Forward Error Correction (FEC) technology, which can recover lost packets. In this method, when a user transmits video data, the user must encode the video data using an error correction code (e.g., Reed Solomon code). Encoding the data means appending parity symbols to the end of the video data (as individual packets). This generates additional and redundant data transmitted to the receiver. If the receiver acquires the data and some of the datagrams are lost (due to the use of UDP), the receiver can recover the lost datagrams provided that the amount of data received is sufficient. This technology can save time for retransmitting dropped packets.

[0004] Unstable network connections that reduce network bandwidth can degrade the quality of video streams through issues such as lagging, framing, and freezing. For example, if network bandwidth drops below the required bandwidth for the media currently being streamed, the sender may be unable to deliver video data within a set time, and users may experience frame freezing and delays. Even if the network bandwidth falls below the required level for a short period, it can be sufficient to affect the stream. If it is anticipated that network throughput will deteriorate, video stream parameters can be modified in advance by reducing the bitrate, frame rate, resolution, or any other quality parameters. It may be possible to predict such situations through continuous monitoring of network conditions. This allows for the prediction that network bandwidth will soon fall below the required level, enabling parameters to be modified in advance.

[0005] Therefore, technologies are required to efficiently predict network status.

[0006] Embodiments of the present disclosure can provide a method and apparatus that can efficiently predict or estimate network conditions without affecting the current video stream and, accordingly, dynamically adjust parameters of media data being streamed.

[0007] According to one embodiment, a method for transmitting a media stream by a transmitting device is provided. The method may include the step of identifying transmission data comprising a plurality of packets for media data and at least one forward error correction (FEC) packet for said plurality of packets for transmission to a receiving device. The method may include the step of generating additional transmission data for said transmission data. The method may include the step of transmitting said transmission data and said additional transmission data to said receiving device. The method may include the step of receiving feedback from said receiving device regarding said transmission data and said additional transmission data. The method may include the step of adjusting the attributes of said media data based on said feedback.

[0008] The above additional transmission data may include additional packets.

[0009] The above additional packets may include at least one of packets duplicated from at least a portion of the plurality of packets, at least one packet duplicated from at least a portion of the at least one FEC packet, a number of FEC packets increased by increasing the FEC rate, or packets containing random data.

[0010] The step of adjusting the attributes of the media data may include, based on the feedback, determining that the network state satisfies a higher quality than the current quality of the media data, and adjusting the attributes of the media data to correspond to the higher quality.

[0011] The step of adjusting the attributes of the media data may include, based on the feedback, determining that the network state does not satisfy the current quality of the media data or reaches the boundary of the current quality, and adjusting the attributes of the media data to correspond to a quality lower than the current quality of the media data.

[0012] The attributes of the above media data may include at least one of resolution, FPS (frames per second), or bitrate.

[0013] The step of transmitting the transmission data and the additional transmission data to the receiving device may include maintaining the size of the sum of the transmission data and the additional transmission data corresponding to a predetermined number of adjacent frames at a constant level. The step of transmitting the transmission data and the additional transmission data to the receiving device may include gradually increasing the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames.

[0014] The step of maintaining the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames at the constant level may include the step of maintaining the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames at the maximum value of the size of the transmission data corresponding to the predetermined number of adjacent frames.

[0015] The step of gradually increasing the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames may include the step of linearly increasing the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames.

[0016] The step of transmitting the transmission data and the additional transmission data to the receiving device may include the step of transmitting the additional transmission data at a time interval. The time interval may be determined based on at least one of the feedback or the current quality of the media data.

[0017] The additional transmission data may include at least one of data duplicated from at least a portion of the content of at least some of the plurality of packets and added to a corresponding packet, data duplicated from at least a portion of the content of at least one of the at least one FEC packet and added to a corresponding FEC packet, data corresponding to an increased size of the at least one FEC packet by increasing the FEC rate, or random data added to the plurality of packets or the at least one FEC packet.

[0018] According to one embodiment, a transmitting device for transmitting a media stream is provided. The transmitting device may include a memory for storing instructions; and at least one processor. The instructions may cause the transmitting device to perform operations when executed collectively or individually by the at least one processor. The operations may include a step of identifying transmission data comprising a plurality of packets for media data and at least one forward error correction (FEC) packet for said plurality of packets for transmission to a receiving device. The operations may include a step of generating additional transmission data for said transmission data. The operations may include a step of transmitting said transmission data and said additional transmission data to the receiving device. The operations may include a step of receiving feedback from the receiving device regarding said transmission data and said additional transmission data. The operations may include a step of adjusting the attributes of said media data based on said feedback.

[0019] According to one embodiment, a non-transient computer-readable storage medium for storing instructions is provided. The instructions may cause the transmitting device to perform operations when executed collectively or individually by the at least one processor of the transmitting device. The operations may include the step of identifying transmission data comprising a plurality of packets for media data and at least one forward error correction (FEC) packet for said plurality of packets for transmission to a receiving device. The operations may include the step of generating additional transmission data for said transmission data. The operations may include the step of transmitting said transmission data and said additional transmission data to the receiving device. The operations may include the step of receiving feedback from the receiving device regarding said transmission data and said additional transmission data. The operations may include the step of adjusting the attributes of said media data based on said feedback.

[0020] Figure 1 shows an exemplary communication network for the transmission of media data.

[0021] FIG. 2 shows an exemplary communication network for transmitting media data according to one embodiment of the present disclosure.

[0022] FIG. 3 shows components of a transmitting device according to one embodiment of the present disclosure.

[0023] Figure 4 shows the sum sizes of a specific number of adjacent av packets and FEC packets.

[0024] FIG. 5 shows a FIFO queue for determining the size of the sum of a specific number of adjacent av packets and FEC packets according to one embodiment of the present disclosure.

[0025] Figure 6 shows the size of the sum of an exemplary number of adjacent av packets, FEC packets, and additional packets.

[0026] FIG. 7 shows the throughput, target value, and total throughput without additional packets when α=1.0 and β=1.0 according to one embodiment of the present disclosure.

[0027] FIG. 8 shows the throughput, target value, and total throughput without additional packets when α=1.1 and β=1.0 according to one embodiment of the present disclosure.

[0028] FIG. 9 shows the throughput, target value, and total throughput without additional packets when α=1.0 and β=100.0 according to one embodiment of the present disclosure.

[0029] FIG. 10 shows the throughput, target value, and total throughput without additional packets when α=1.1 and β=100.0 according to one embodiment of the present disclosure.

[0030] FIG. 11 shows the throughput without additional packets and the total throughput according to one embodiment of the present disclosure when w=50, k=40, and a=0.5.

[0031] FIG. 12 shows the throughput without additional packets and the total throughput according to one embodiment of the present disclosure when w=100, k=75, and a=0.5.

[0032] FIG. 13 shows the throughput without additional packets and the total throughput according to one embodiment of the present disclosure when w=100, k=100, and a=0.5.

[0033] FIG. 14 shows the throughput without additional packets and the total throughput according to one embodiment of the present disclosure when w=50, k=55, and a=0.5.

[0034] FIG. 15 shows an increase in the number of exemplary additional packets according to one embodiment of the present disclosure.

[0035] FIG. 16 shows an increase in the number of exemplary additional packets according to one embodiment of the present disclosure.

[0036] FIG. 17 illustrates sampling additional packets at exemplary constant time intervals according to one embodiment of the present disclosure.

[0037] FIG. 18 illustrates sampling additional packets at exemplary constant time intervals according to one embodiment of the present disclosure.

[0038] FIG. 19 illustrates sampling exemplary additional packets according to one embodiment of the present disclosure.

[0039] FIG. 20 illustrates a process for increasing media stream quality according to one embodiment of the present disclosure.

[0040] FIG. 21 illustrates a process for reducing media stream quality according to one embodiment of the present disclosure.

[0041] FIG. 22 illustrates a process for increasing media stream quality according to one embodiment of the present disclosure.

[0042] FIG. 23 illustrates a process for managing the quality of a media stream according to one embodiment of the present disclosure.

[0043] FIG. 24 is a flowchart illustrating a method for transmitting a media stream by a transmitting device according to one embodiment of the present disclosure.

[0044] FIG. 25 is a block diagram of a transmitting device according to one embodiment of the present disclosure.

[0045] The operating principles of preferred embodiments of the present disclosure are described in detail below with reference to the attached drawings. Identical components shown in the drawings are indicated by the same reference numerals whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure below, detailed descriptions of related known functions or configurations will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Additionally, terms used below are defined considering their functions in the present disclosure, and these may vary depending on the intentions or conventions of the user or operator. Therefore, their definitions should be based on the content throughout this specification.

[0046] The present disclosure is capable of various modifications and may have various embodiments, and specific embodiments are described in detail by way of illustration in the drawings. However, this is not intended to limit the present disclosure to specific embodiments and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure.

[0047] Additionally, unless explicitly otherwise indicated in this specification, singular expressions such as "as" and "above" will be understood to include plural expressions.

[0048] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0049] Furthermore, the terms used in this specification are used merely to describe specific embodiments and are not intended to limit the disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0050] Furthermore, in the embodiments of this disclosure, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which this disclosure pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of this disclosure.

[0051] Figure 1 shows an exemplary communication network for the transmission of media data.

[0052] In a communication network, a transmitting (Tx) device (110) can transmit media data to a receiving (RX) device (120). The media data may include video data. The media data may also include additional (extra) data (e.g., audio data).

[0053] Media data may include multiple AV packets. Each AV packet may include data corresponding to a single frame. If the media data has the attribute of 30 frames per second (30 fps: frames per second), the Rx device (120) needs to receive data corresponding to at least 30 AV packets per second. 30 fps is an exemplary value and may vary depending on the attribute of the media data or the network environment.

[0054] Each of the AV packets may be divided into multiple smaller packets (e.g., N packets) for transmission. The smaller packets may be referred to as payload packets or simply packets. The size of each of the multiple packets may be the same, but is not limited thereto. The size of an AV packet may be equal to the sum of the sizes of the multiple packets divided from it.

[0055] The Tx device (110) can generate at least one forward error correction (FEC) packet corresponding to multiple packets for each av packet. At least one FEC packet can be generated by encoding multiple packets or av packets.

[0056] The Tx device (110) can transmit multiple packets corresponding to multiple av packets and FEC packets to transmit media data to the Rx (120) device.

[0057] FIG. 2 shows an exemplary communication network for transmitting media data according to one embodiment of the present disclosure.

[0058] According to one embodiment, the Tx device (110) may generate and transmit additional data in addition to the plurality of packets and FEC packets described in FIG. 1 to estimate or predict the network state. The additional data may include additional packets. The additional packets may include at least one of the following.

[0059] - Packets duplicated from at least some of the payload packets

[0060] - Packets duplicated from at least some of the FEC packets

[0061] - The number of FEC packets increased by increasing the FEC rate

[0062] - Packets containing random data

[0063] According to one embodiment, additional data may include data that increases the size of an existing packet or an FEC packet. In this case, the additional data may include at least one of the following.

[0064] - Data duplicated from at least a portion of the content of at least some of the payload packets and added to the corresponding payload packets

[0065] - Data added to the corresponding FEC packet, duplicated from at least a portion of the content of at least some of the FEC packets

[0066] - Data corresponding to the increased size of FEC packets by increasing the FEC rate

[0067] - Random data added to payload packets or FEC packets

[0068] For each AV packet, a group of multiple packets, at least one FEC packet, and additional data (e.g., additional packets) may be formed. The following strategies may be considered regarding how much additional data should be transmitted for each AV packet, at what frequency, and in what environment.

[0069] - Continuously (e.g., linearly) increase the additional data transmitted in subsequent frames (e.g., the number of additional packets).

[0070] - When the total packet size (e.g., the sum of packets and FEC packets) is less than a threshold defined by the system, additional packets are added.

[0071] ○ The threshold value may be a value defined by the system as an arbitrary value.

[0072] ○ The threshold may be the average value of packet sizes over an unspecified time period.

[0073] ○ The threshold may be a value defined by an external algorithm that considers network parameters, network stability, packet loss rate, and other options.

[0074] The threshold can be defined by an AI (artificial intelligence) algorithm that makes decisions based on network parameters, network stability, packet loss rate, and other options.

[0075] - Add additional packets to the local minimum of the total size of AV packets calculated over a predefined time period.

[0076] - Add additional packets to exceed the previous maximum level achieved by the system (sum of payload packets + FEC packets + additional packets).

[0077] FIG. 3 shows components of a transmitting device according to one embodiment of the present disclosure.

[0078] The Tx device (110) may include a video source, an FEC encoder, a network traffic equalizer, and a network traffic sampler as components.

[0079] The video source can read AV packets containing compressed data such as video and / or audio.

[0080] The FEC encoder can encode AV packets to generate parity bytes that can be used to recover dropped packets.

[0081] A network traffic equalizer can generate equalized packets according to a selected strategy to balance network traffic at a given throughput.

[0082] A network traffic sampler can generate sampler packets (i.e., additional packets) according to a selected strategy to increase network traffic beyond a balanced level.

[0083] Figure 4 shows the sum sizes of a specific number of adjacent av packets and FEC packets.

[0084] An algorithm needs to be implemented to calculate the required amount of data to be transmitted for each of a specific (or predefined) number of adjacent AV packets. 410 in FIG. 4 represents the sum of the sizes of a specific number of adjacent AV packets in a media stream and the FEC packets generated in association with said AV packets. In the example of FIG. 4, during the media stream, the largest sum of the sizes is approximately 790,000 bytes, which is the highest bandwidth at which media data was successfully transmitted. The largest sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets can be indicated as 420 in FIG. 4 and may change over time.

[0085] An implementation of the algorithm according to one embodiment may generate additional data to fill the area between the largest sum (420) of the sizes of a specific number of adjacent av packets and the FEC packets generated in association with the av packets and the remaining individual points (i.e., the area between 420 and 410).

[0086] In the interval where the largest sum (420) of the sizes of a specific number of adjacent av packets and the FEC packets generated in association with said av packets does not change, it is necessary to maintain the amount of data transmitted at a similar level. That is, the Tx device (110) may attempt to maintain a constant amount of data transmitted to the Rx device (120) within a specific time interval by modifying additional data (e.g., decreasing or increasing the number of additional packets). To this end, the current difference between the total sum of the sizes of a specific number of adjacent av packets and the FEC packets generated in association with said av packets and the current maximum sum of said sizes needs to be calculated.

[0087] First, the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets must be calculated. For the next incoming AV packet, the sum of the sizes of the first AV packet and the FEC packets generated in association with said AV packets must be subtracted from the total sum, and the sum of the sizes of the incoming AV packet and the FEC packets generated in association with said AV packet must be added. This can be implemented through a FIFO (first in first out) queue (510) illustrated in FIG. 5. FIG. 5 illustrates a FIFO queue for calculating the size of the sum of a specific number of adjacent AV packets and FEC packets according to one embodiment of the present disclosure. Within the FIFO queue (510), there always exist a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets. For every new AV packet and the sum of the FEC packets generated in association with the new AV packet, the first element of the FIFO queue (510) is deleted and new elements are added.

[0088] If the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets is greater than the current maximum total sum, said maximum total sum can be updated. If the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets is not greater than the current maximum total sum, new additional packets can be generated and transmitted to the Rx device (120). The number of additional packets can be calculated according to the following formula (1).

[0089] (1): num of packets = (max total sum - current sum) / size of single packet

[0090] num of packets: number of additional packets

[0091] max total sum: The maximum total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets.

[0092] Current sum: The current total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets.

[0093] size of single packet: size of one additional packet

[0094] If the current total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets is greater than the maximum total sum of the sizes, there is no need to generate additional packets.

[0095] A network traffic equalizer responsible for calculating the current difference between the total sum of sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said specific number of adjacent AV packets, and the maximum total sum of sizes of FEC packets generated in association with said specific number of adjacent AV packets, in order to maintain network bandwidth at a given level, can perform the following operations.

[0096] First, calculate the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said specific number of adjacent AV packets, and the maximum total sum of the sizes of the FEC packets generated in association with said specific number of adjacent AV packets.

[0097] Next, for the incoming AV packet and the FEC packets associated with the AV packet, the sum of the sizes of the FEC packets associated with the first AV packet is subtracted from the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said specific number of adjacent AV packets, and the sum of the sizes of the incoming AV packet and the FEC packets associated with the AV packet is added.

[0098] - The above operation can be performed through a FIFO queue consisting of the sum of a specific number of adjacent AV packets and the FEC packets associated with those packets. For each new AV packet and the sum of its associated FEC packets, the first element of the queue is removed (meaning its size is deducted) and the incoming element is added (meaning its size is added).

[0099] - If the current total sum of a specific number of adjacent AV packets and the FEC packets associated with those packets is greater than the maximum total sum of the sizes of the FEC packets generated in association with the specific number of adjacent AV packets, the maximum total sum is updated.

[0100] - If the above maximum total sum is not updated in the previous step, additional packets are generated according to formula (1).

[0101] - If the above maximum total sum is updated, additional packets are not generated.

[0102] FIG. 6 shows the size of the sum of an exemplary number of adjacent AV packets, FEC packets, and additional packets. 610 represents the current total sum of the sizes of an adjacent number of AV packets and the FEC packets generated in association with said AV packets. 620 represents the maximum total sum of the sizes of an adjacent number of AV packets and the FEC packets generated in association with said AV packets. 630 represents the total sum of the sizes of a specific number of AV packets, the FEC packets generated in association with said AV packets, and additional packets.

[0103] Table 1 shows a pseudo code for maintaining the amount of data transmitted according to one embodiment at a similar level.

[0104] for each packet:stream_total_size -= stream_total_size_deq.front()stream_total_size_deq.pop_front()stream_total_size += packet.size()stream_total_size_deq.push_back(packet.size())if stream_total_size > max_total_size thenmax_total_size = stream_total_sizesampling_total_size -= sampling_total_size_deq.front()sampling_total_size_deq.pop_front()sampling_total_size += packet.size()sampling_total_size_deq.push_back(packet.size())diff_size = max_total_size - sampling_total_size;if diff_size > 0 thennum_of_extra_packets = diff_size / packet_sizesampling_total_size_deq.back() += diff_sizesampling_total_size += diff_size

[0105] - sampling_total_size: The total sum of the sizes of a specific number of adjacent AV packets, FEC packets generated in association with said AV packets, and additional packets - sampling_total_size_deq: A structure that allows adding / deleting elements from the start and end - max_total_size: The maximum total sum of the sizes of a specific number of adjacent AV packets and FEC packets generated in association with said AV packets

[0106] According to one embodiment, the number of additional packets can be calculated by minimizing the error value between the estimated values ​​and the target values.

[0107] In Equation (2), b(k) represents the amount of bytes transmitted from point k. b(k + p│k) is an estimate of the amount of bytes transmitted from point k + p, which is performed at point k.

[0108] (2):

[0109] u(k) represents the amount of additional bytes to be transmitted.

[0110] (3):

[0111] (4):

[0112] N is a configurable time window.

[0113] At each point k, the following function (Equation (5)) can be optimized.

[0114] (5):

[0115] e(x) is the error function, and t(k) is the target value to be reached. to minimize The values ​​of can be found (Equation (6)).

[0116] (6):

[0117] Here, the limitation of formula (7) can be given.

[0118] (7):

[0119] u(k│k) is the amount of additional bytes transmitted from point k. The operation algorithm is as follows in Equation (8).

[0120] (8):

[0121] A network traffic equalizer responsible for calculating the current difference between the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said specific number of adjacent AV packets, and the maximum total sum of the sizes of the FEC packets generated in association with said specific number of adjacent AV packets, in order to maintain network bandwidth at a given level, can perform the following operations at each point in time.

[0122] - A target function is used to calculate the desired sum of the sizes of N adjacent packets.

[0123] - An estimation function is used to estimate the future sum of the sizes of N adjacent packets.

[0124] - An error function is used to calculate the error between the target values ​​and the estimated values.

[0125] The sizes of additional packets are calculated by finding the values ​​that minimize the error function value.

[0126] - The first value of the calculated values ​​is taken, and the corresponding number of additional packets are added.

[0127] FIGS. 7, 8, 9, and 10 show that the above algorithm can be adjusted through different error functions, estimation functions, and other parameters. FIG. 7 shows the throughput (y(k)), target value (t(k)), and total throughput (z(k)) without exemplary additional packets when α=1.0 and β=1.0 according to one embodiment of the present disclosure. FIG. 8 shows the throughput (y(k)), target value (t(k)), and total throughput (z(k)) without exemplary additional packets when α=1.1 and β=1.0 according to one embodiment of the present disclosure. FIG. 9 shows the throughput (y(k)), target value (t(k)), and total throughput (z(k)) without exemplary additional packets when α=1.0 and β=100.0 according to one embodiment of the present disclosure. FIG. 10 shows the throughput (y(k)), target value (t(k)), and total throughput (z(k)) without additional packets when α=1.1 and β=100.0 according to one embodiment of the present disclosure. That is, the algorithm can be configured extensively as needed.

[0128] According to one embodiment, the number of additional packets can be calculated by calculating the difference between the quantile of the last N packets and the current total sum of the sizes of a specific number of adjacent av packets and the FEC packets generated in association with said av packets.

[0129] In the following formula (9), d(i) represents the number of data transmitted from a specific point i, and Indicates the number of equalized data, represents the number of FEC data and represents the number of equalized packets.

[0130] (9):

[0131] The helper sigma can be derived by formula (10).

[0132] (10):

[0133] Through formula (11), the k-th (k-th) smallest group size in a specific time window w can be derived for sigma.

[0134] (11):

[0135] sort represents a permutation of a given set that is an incremental sequence. Through Equation (11), the number of additional packets can be calculated for delta (δ).

[0136] (12):

[0137] a is the round coefficient.

[0138] A network traffic equalizer responsible for calculating the current difference between the total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said specific number of adjacent AV packets, and the maximum total sum of the sizes of the FEC packets generated in association with said specific number of adjacent AV packets, in order to maintain network bandwidth at a given level, can perform the following operation.

[0139] - The number of additional packets is the maximum number of registered packets minus the current number of packets.

[0140] FIGS. 11, 12, 13, and 14 illustrate how the corresponding algorithm operates for different parameters w, k, and a. In FIGS. 11, 12, 13, and 14, σ(d, f, i) represents the total throughput, and d^(d, f, i) represents the throughput without additional packets. FIG. 11 shows the exemplary throughput without additional packets and the total throughput for w=50, k=40, and a=0.5 according to one embodiment of the present disclosure. FIG. 12 shows the exemplary throughput without additional packets and the total throughput for w=100, k=75, and a=0.5 according to one embodiment of the present disclosure. FIG. 13 shows the exemplary throughput without additional packets and the total throughput for w=100, k=100, and a=0.5 according to one embodiment of the present disclosure. FIG. 14 shows the throughput without additional packets and the total throughput according to one embodiment of the present disclosure when w=50, k=55, and a=0.5.

[0141] The following describes algorithms for generating additional packets.

[0142] According to one embodiment, the Tx device (110) may generate a number of additional packets that increase linearly. Until a first specific number of av packets are generated, the Tx device (110) does not generate any additional packets to determine the largest total size of these packets. Subsequently, the Tx device (100) may begin to increase the number of additional packets according to a given interval and factor. This process may be repeated during media streaming until the total size of the packets reaches a higher quality boundary and it is decided to increase at least one media stream parameter. The frequency of generating additional packets may be determined by customized parameters (e.g., interval and factor). An algorithm may be used to maintain network bandwidth at a given level in relation to the additional packets generated here.

[0143] FIG. 15 illustrates an increase in the number of exemplary additional packets according to one embodiment of the present disclosure. 1510 represents throughput without additional packets (i.e., the sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets). 1520 represents maximum throughput without additional packets (i.e., the largest total sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with said AV packets). 1530 represents total throughput including additional packets (i.e., the sum of the sizes of a specific number of adjacent AV packets, the FEC packets generated in association with said AV packets, and the additional packets).

[0144] According to one embodiment, the Tx device (110) may generate additional packets based on a target value for total throughput. Here, the target function needs to be modified to increase in order to increase the target value. FIG. 16 illustrates an increase in the number of exemplary additional packets according to one embodiment of the present disclosure. FIG. 16 shows the throughput (y(k)), target value (t(k)), and total throughput (z(k)) without additional packets when α=1.1 and β=100.0.

[0145] According to one embodiment, the number of additional packets may be based on the previous response window of the Rx device (120). If reception at the Rx device (120) is stable for a specific time window, the number of additional packets may be increased by a specific factor value. If reception at the Rx device (120) is unstable for a specific time window, the number of additional packets may be decreased. If the number of additional packets reaches a specific network bandwidth boundary, the quality of the media data is upgraded. If the connection is not stable for a longer time window and the number of additional packets is 0, the quality is downgraded.

[0146] In the following equation (13), b(k) represents the number of packets transmitted at a specific time point k. represents the number of received packets corresponding to time point k. s(k) represents the ratio of transmitted packets to received packets at a specific time l=k.

[0147] (13):

[0148] In the equation (14) below, the quality function q(k) represents the quality of the connection at time point k. w represents the weight vector.

[0149] (14):

[0150] In the following formula (15), delta (δ) represents the following decision based on the current quality of the streaming.

[0151] (15):

[0152] If a specific limiting lambda (λ) condition is satisfied, the determinant may be increased or decreased depending on the current streaming performance. Otherwise, the determinant may be decreased. The determinant may be negative, but the sum of additional packets will be clamped to 0. Finally, the next iteration b can be calculated according to the following formula (16).

[0153] (16):

[0154] Sigma (σ) is a coefficient used with the current determinant to sample the network. The length of the weight vector is referred to as the time window of the decision process. If the determinant reaches a required value, the current stream quality may be promoted (or increased). If the determinant is sufficiently small (even if negative), the stream quality may be demoted (or decreased). With each promotion / demotement of the stream quality, the determinant is set to 0.

[0155] According to one embodiment, additional packets may be generated when the total size of the packets (av packets + FEC packets + additional packets) is less than a predefined threshold (e.g., local minimum). Additional packets are generated when the total size of the packets is less than a predefined threshold (e.g., local minimum).

[0156] - The number of additional packets generated at every stage increases, and the total size of the packets (av packets + FEC packets + additional packets) must exceed the previously achieved value within a given time period.

[0157] - The process of increasing the total size of packets can be bilinear, having a small increase in size, or aggressive, having a rapid increase.

[0158] - The process must be correlated with network flows and detected network states.

[0159] According to one embodiment, an increasing number of additional packets may be generated at unspecified time intervals. The number of additional packets increases over time by a specific factor value. FIG. 17 illustrates sampling additional packets at exemplary constant time intervals according to one embodiment of the present disclosure. 1710 represents the size of the av packets. 1720 represents the size of the av packets at the moment they are sampled into additional packets. Additional packets may be generated to test network bandwidth. In the example of FIG. 17, additional packets may be generated every 30th av packet. The interval for generating additional packets may be variable and may be irregular. The time at which additional packets are generated may be determined by the Tx device (110) based on statistical reports (i.e., feedback) received from the Rx device (120) (or Rx application) and the current stream quality.

[0160] According to one embodiment, the number of additional packets may increase according to the sum of the sizes of adjacent packets and generated additional packets. FIG. 18 illustrates sampling additional packets at exemplary constant time intervals according to one embodiment of the present disclosure. 1810 represents the sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with them. 1820 represents the maximum sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with them. 1830 represents the sum of the sizes of a specific number of adjacent AV packets, the FEC packets generated in association with them, and additional packets. Here, the number of additional packets may increase according to the sum of the sizes of adjacent packets and generated additional packets. Table 2 shows pseudo codes for this.

[0161] for each packet:if generate_extra_data() thenif max_sampling_total_size > max_total_size thenmax_sampling_total_size += factor * packet_sizeelsemax_sampling_total_size = max_total_size + factor * packet_sizenum_of_extra_packets = (max_sampling_total_size - max_total_size) / packet_sizeelsenum_of_extra_packets = 0

[0162] - max_total_size: The maximum sum of the sizes of a specific number of adjacent AV packets and the FEC packets generated in association with them. - max_sampling_total_size: The maximum size of max_total_size that includes additional generated packets. Below, the modification of media data quality based on the generation of additional packets is described.

[0163] FIG. 19 illustrates sampling exemplary additional packets according to one embodiment of the present disclosure. In FIG. 19, the average packet size may refer to the average size of groups of av packets. Each group of av packets may include an av packet (or payload packet), an FEC packet associated with the av packet, and (if present) additional packets. While media data is being streamed to have high definition (HD) quality, if the average packet size reaches FullHD due to the generation of increasing additional packets, the Tx device (110) may change the attributes of the media data being streamed from HD to FullHD. In this way, if the attributes of the network being tested through additional packets (e.g., bandwidth, throughput, average packet size) satisfy conditions for streaming of a higher quality than the current quality, the attributes of the media data may be changed to improve quality.

[0164] Table 3 shows exemplary resolutions and their corresponding average AV packet sizes.

[0165] Resolution Average AV Packet Size 426x240 1742 bytes 640x360 2832 bytes 960x540 8647 bytes 1280x720 14915 bytes 1920x1080 29605 bytes 2560x1440 71259 bytes 3840x2160 130578 bytes

[0166] Table 4 shows exemplary resolutions and the network bandwidth required for them.

[0167] Resolution Network Bandwidth 426x2400,40 Mbps 640x3600,65 Mbps 960x5401,98 Mbps 1280x7203,41 Mbps 1920x10806,78 Mbps 2560x144016,31 Mbps 3840x216029,89 Mbps

[0168] FIG. 20 illustrates a process for increasing media stream quality according to one embodiment of the present disclosure. A Tx device (110) and an Rx device (120) may be connected to each other using a given network protocol for bidirectional communication between them. During video (or media) streaming, the Tx device (110) may calculate / estimate / use predefined values ​​to determine the boundaries of higher or lower video streaming quality. Here, different video streaming parameters may be changed dynamically. During video streaming, the Rx device (120) may transmit reports (i.e., feedback) containing information about current network capabilities to the Tx device (110). Based on changes in network connectivity, network communication, or bandwidth utilization, the transmission rate for video streaming may be modified. Detection of such changes may be based on the reports.

[0169] Step 1: The Tx device (110) may begin transmitting video streaming AV packets and associated FEC packets (i.e., linked to payload data) to the Rx device (120) with a specific quality (e.g., resolution, bitrate, fps, etc.).

[0170] Step 2: The network traffic equalizer of the Tx device (110) can balance the network traffic to maintain network traffic at a similar level.

[0171] Step 3: After maintaining network traffic at a similar level, the network traffic sampler of the Tx device (110) can generate additional packets to gradually increase network traffic.

[0172] Step 4: After reaching the network bandwidth required to increase quality, the Tx device (110) can modify the video streaming parameters to increase the quality of the video streaming.

[0173] Step 5: The Tx device (110) can continue streaming with uninterrupted, smooth transitions. It can then return to Step 1.

[0174] FIG. 21 illustrates a process for reducing media stream quality according to one embodiment of the present disclosure.

[0175] Step 1: The Tx device (110) may begin transmitting video streaming AV packets and associated FEC packets (i.e., linked to payload data) to the Rx device (120) with a specific quality (e.g., resolution, bitrate, fps, etc.).

[0176] Step 2: The network traffic equalizer of the Tx device (110) can balance the network traffic to maintain network traffic at a similar level.

[0177] Step 3: After maintaining network traffic at a similar level, the network traffic sampler of the Tx device (110) can generate additional packets to gradually increase network traffic.

[0178] Step 4: The Rx device (120) can transmit a report to the Tx device (110) containing network statistics including information regarding network bandwidth degradation.

[0179] Step 5: The Tx device (110) may decide to reduce the quality of the video stream by modifying specific parameters to adapt to the current network bandwidth.

[0180] Step 6: The Tx device (110) can continue streaming. Then, it can return to Step 1.

[0181] FIG. 22 illustrates a process for increasing media stream quality according to one embodiment of the present disclosure.

[0182] Step 1: The Tx device (110) may begin transmitting video streaming AV packets and associated FEC packets (i.e., linked to payload data) to the Rx device (120) with a specific quality (e.g., resolution, bitrate, fps, etc.).

[0183] Step 2: The network traffic equalizer of the Tx device (110) can balance the network traffic to maintain network traffic at a similar level.

[0184] Step 3: After maintaining network traffic at a similar level, the network traffic sampler of the Tx device (110) can increase network traffic by transmitting higher quality packets.

[0185] Step 4: The Tx device (110) may decide to increase the quality of the video stream by replacing the payload data with additional packets.

[0186] If the Tx device (110) can successfully transmit current and higher quality video streaming packets, the Rx device (120) can make a decision to improve the quality of the displayed video. In this way, the Tx device (110) can continue streaming at a higher quality (by replacing the payload data with additional packets having higher video quality) and then start transmitting additional packets that are higher than the current quality.

[0187] Step 5: The Tx device (110) can continue streaming together. Then, it can return to Step 1.

[0188] FIG. 23 illustrates a process for managing the quality of a media stream according to one embodiment of the present disclosure.

[0189] The Tx device (110) can perform continuous streaming of media data (2210).

[0190] The Tx device (110) can determine the throughput required for a higher stream quality and / or the throughput required for a lower stream quality (2220).

[0191] The Tx device (110) can determine whether the network throughput is stable (2230). Whether the network throughput is stable can be determined based on reports (or feedback) from the Rx device (120).

[0192] The Tx device (110) may reduce quality when network throughput is unstable and reaches boundary throughput (2240). Here, boundary throughput may be the minimum throughput required for the quality of the current streaming.

[0193] The Tx device (110) can start transmitting additional packets when the network throughput is stable (2250).

[0194] The Tx device (110) can determine whether the required throughput has been reached (2260). The required throughput here may be the throughput required for higher quality streaming.

[0195] The Tx device (110) can increase the stream quality when the required throughput is reached (2280).

[0196] If the required spurt is not reached, the Tx device (110) continues to transmit additional packets (2270) and can determine whether the network throughput is stable (2230).

[0197] FIG. 24 is a flowchart illustrating a method for transmitting a media stream by a transmitting device according to one embodiment of the present disclosure.

[0198] The Tx device (110) can identify transmission data including a plurality of packets for media data to be transmitted to the Rx device (120) and at least one FEC packet for said plurality of packets (2410). Here, the plurality of packets may refer to av packets or payload packets.

[0199] The Tx device (110) can generate additional transmission data for the transmission data (2420). The additional transmission data may be at least one additional packet or an increased size of the existing packets.

[0200] The Tx device (110) can transmit the transmission data and the additional transmission data to the Rx device (120) (2430).

[0201] The Tx device (110) can receive feedback from the Rx device (120) regarding the transmission data and the additional transmission data (2440).

[0202] The Tx device (110) can adjust the attributes of the media data based on the feedback (2450).

[0203] FIG. 25 is a block diagram of a transmitting device according to one embodiment of the present disclosure.

[0204] The Tx device (110) may include a memory (2510), a transceiver (2520), and a controller (2530).

[0205] The memory (2510) may store at least one of information transmitted and / or received through the transceiver (2520) and information generated through the controller (2530). The memory (2510) may additionally store information / data / commands / instructions to be executed by the controller (2530) to enable the Tx device (110) to perform the operations described herein. The memory (2510) may also be referred to as a non-transient computer-readable storage medium.

[0206] The transceiver (2520) may be used to transmit signals from the controller (2530) to other entities or to receive signals from other entities. The transceiver (2520) may support communication via various wireless access technologies (RATs), including, for example, LTE (Long Term Evolution), Wi-Fi (Wireless Fidelity), OFDM (orthogonal frequency division multiplexing), CDMA (code divisional multiple access), and Bluetooth, but the RATs supported by the transceiver (2520) are not limited thereto. The transceiver (2520) may support wired communications as well as RATs.

[0207] The controller (2530) can control the operations of other elements of the Tx device (110). The operations of the Tx device (110) can be understood as substantially executed by the controller (2530). The controller (2530) can be implemented through at least one processor. The controller (2530) can execute instructions stored in memory (2510) to perform the operations of the Tx device (110) described herein.

[0208] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments but should be defined by the claims set forth below as well as equivalents thereof.

Claims

1. A method for transmitting a media stream by a transmitting device (110), A step (2410) of identifying transmission data including a plurality of packets for media data and at least one FEC (forward error correction) packet for the plurality of packets for transmission to a receiving device (120); Step (2420) of generating additional transmission data for the above transmission data; A step (2430) of transmitting the above transmission data and the above additional transmission data to the receiving device (120); A step (2440) of receiving feedback related to the transmission data and the additional transmission data from the receiving device (120); and A method comprising the step (2450) of adjusting the attributes of the media data based on the above feedback, method.

2. In Paragraph 1, The above additional transmission data includes additional packets, method.

3. In any one of paragraphs 1 to 2, The above additional packets are: Packets duplicated from at least some of the above plurality of packets; At least one packet duplicated from at least a portion of the above at least one FEC packet; The number of FEC packets increased by increasing the FEC rate; or including at least one of the packets containing random data, method.

4. In any one of paragraphs 1 to 3, The step of adjusting the attributes of the above media data is: Based on the above feedback, a step of determining that the network state satisfies a higher quality than the current quality of the media data; and A step comprising adjusting the attributes of the media data to correspond to the higher quality, method.

5. In any one of paragraphs 1 through 4, The step of adjusting the attributes of the above media data is: Based on the above feedback, a step of determining that the network state does not satisfy the current quality of the media data or reaches the boundary of the current quality; and A step comprising adjusting the attributes of the media data to correspond to a quality lower than the current quality of the media data, method.

6. In paragraphs 1 through 5, The attributes of the above media data include at least one of resolution, FPS (frames per second), or bitrate. method.

7. In any one of paragraphs 1 through 6, The step of transmitting the above transmission data and the additional transmission data to the receiving device (120) is: A step of maintaining the size of the sum of the transmission data and the additional transmission data corresponding to a predetermined number of adjacent frames at a constant level; and A step comprising gradually increasing the size of the sum of the transmission data and the additional transmission data corresponding to the aforementioned predetermined number of adjacent multiple frames. method.

8. In any one of paragraphs 1 through 7, The step of maintaining the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames at the constant level includes the step of maintaining the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames at the maximum value of the size of the transmission data corresponding to the predetermined number of adjacent frames. method.

9. In any one of paragraphs 1 through 8, The step of gradually increasing the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames includes the step of linearly increasing the size of the sum of the transmission data and the additional transmission data corresponding to the predetermined number of adjacent frames. method.

10. In any one of paragraphs 1 through 9, The step of transmitting the transmission data and the additional transmission data to the receiving device includes the step of transmitting the additional transmission data at a time interval, and The above time interval is determined based on at least one of the feedback or the current quality of the media data, method.

11. In any one of paragraphs 1 through 10, The above additional transmission data is: Data added to a corresponding packet by replicating at least a portion of the content of at least some of the above plurality of packets; Data added to a corresponding FEC packet by replicating at least a portion of the contents of at least one of the above-mentioned at least one FEC packet; Data corresponding to the increased size of the at least one FEC packet by increasing the FEC rate; or including at least one of the plurality of packets or at least one random data added to the at least one FEC packet, method.

12. As a transmitting device (110) for transmitting a media stream, Memory for storing commands (2510); and It includes at least one processor (2530), and when the instructions are executed collectively or individually by the at least one processor (2530), the transmitting device (110) performs operations, and the operations are: An operation of identifying transmission data comprising a plurality of packets for media data and at least one FEC (forward error correction) packet for said plurality of packets for transmission to a receiving device (120), The operation of generating additional transmission data for the above transmission data, The operation of transmitting the above transmission data and the above additional transmission data to the receiving device (120), The operation of receiving feedback related to the transmission data and the additional transmission data from the receiving device (120), and including an operation to adjust the attributes of the media data based on the above feedback, Transmitting device.

13. In Paragraph 12, The above operations include at least one operation according to the method of any one of claims 2 to 10, Transmitting device.

14. A non-transient computer-readable storage medium for storing instructions, wherein the instructions, when executed collectively or individually by the at least one processor (2530) of the transmitting device (110), cause the transmitting device (110) to execute operations, and the operations are: An operation of identifying transmission data comprising a plurality of packets for media data and at least one FEC (forward error correction) packet for said plurality of packets for transmission to a receiving device (120); An operation to generate additional transmission data for the above transmission data; The operation of transmitting the above transmission data and the above additional transmission data to the receiving device; An operation of receiving feedback related to the transmission data and the additional transmission data from the receiving device; and including an operation to adjust the attributes of the media data based on the above feedback, Non-transient computer-readable storage media.

15. In Paragraph 14, The above operations include at least one operation according to the method of any one of claims 2 to 10, Non-transient computer-readable storage media.

Images